Migration guide

Moving workloads to SparkHouse.

A practical guide for teams moving notebooks, jobs and tables from Databricks or Snowflake. It lists what SparkHouse supports today, where it has limits, what it does not do, and a suggested order of work.

From Databricks From Snowflake

This page describes SparkHouse v0.12.0 as documented in October 2026, and was last reviewed on 17 October 2026. It describes SparkHouse only. For any other company's product, the maker's own documentation governs, and we link to it. See the legal notices at the end of the page.

From Databricks

Notebooks, jobs and tables.

SparkHouse accepts Databricks notebook source files and implements a subset of the Databricks Jobs API, so many job clients need only a new host and token. Tables can be read through an Iceberg catalog connection first and converted to Iceberg at your own pace.

2 supported7 with limits1 not currently supported

What SparkHouse supports

AreaStatus in SparkHouseHow SparkHouse handles it
Notebook filesSupportedSparkHouse imports notebook source files with comment markers such as # Databricks notebook source, and Jupyter .ipynb files. Widget defaults declared with dbutils.widgets.text become notebook parameters. Scala and R cells are kept as comments and are not executed.
Notebook utilitiesSupported with limitsSparkHouse provides dbutils.widgets.get, dbutils.secrets.get, dbutils.notebook.exit and dbutils.jobs.taskValues. Other dbutils modules are not documented as supported, so check your code.
Jobs REST APISupported with limitsSparkHouse implements a subset of the publicly documented Databricks Jobs API, versions 2.1 and 2.2 (Databricks Jobs API reference): create, get, list, update, reset, delete and run-now for jobs, and get, list, cancel and repair for runs. Task graphs, retries, timeouts, schedules, queues and concurrency limits are supported. Methods outside the subset are listed in the last row.
API clients and orchestratorsSupported with limitsWe tested version 0.147.0 of the open source Databricks SDK for Python against this API. Airflow, Dagster and Prefect integrations call the same endpoints, and are documented to work with a different host and token. We have not tested each of them.
SQL compute and saved queriesSupportedSQL endpoints in sizes XS to XL with auto-resume and auto-stop. Saved queries can be referenced by id in a job's SQL task.
Reading tables from Unity CatalogSupported with limitsSparkHouse can connect to the Iceberg REST interface that Databricks documents for Unity Catalog (Databricks documentation: access tables from Apache Iceberg clients), using a service principal or a token. Nothing is copied. Which tables can be read, and which permissions and settings are required, are set by Databricks, so follow its documentation. This connection is built, and we have not yet recorded a run against a live Databricks account.
Delta Lake tablesSupported with limitsSparkHouse's native table format is Apache Iceberg. Apache Spark endpoints read Delta tables by path on S3, which we tested on AWS. Delta path tables are not available on data planes that use vended storage. Plan to convert Delta tables to Iceberg.
dbtSupported with limitsA dbt task runs dbt-spark on the Apache Spark engine over Spark Connect. It builds into one catalog. Python models, run-operation and freshness are not offered.
PipelinesSupported with limitsA pipeline task runs .py and .sql files with refresh options. SparkHouse has no stored pipeline definitions.
Features SparkHouse does not currently provideNot currently supportedIn the version described here: the runs/submit method; Python wheel, JAR and spark-submit task types; notification destinations such as Microsoft Teams and PagerDuty; row filters, column masks and lineage.

A suggested order of work.

  1. Take stock

    List notebooks, jobs, tables, libraries and permissions. Mark Scala and R code, JAR and wheel tasks, runs/submit callers and Delta-only features.

  2. Check your agreements

    Read the terms that cover notice periods, minimum commitments, data export and fees before you move data or stop services.

  3. Pick where it runs

    Choose the cloud and region, or your own Kubernetes cluster, and set the residency policy before any data moves.

  4. Connect data without copying

    Add the Unity Catalog connection, test it, and read one table.

  5. Move notebooks

    Import the source files. Check the parameters that came from widgets, and add secrets to the workspace.

  6. Move jobs

    Point your client at the SparkHouse host with an API token, create the jobs, and run them beside the originals.

  7. Compare, then cut over

    Match row counts and results on both sides for several cycles. Switch the schedules, and convert Delta tables to Iceberg when you are ready.

shctl
# add a Unity Catalog connection (nothing is copied)
UNITY_SECRET=... shctl catalog add lakehouse --provider unity \
  --workspace-url https://dbc-1234.cloud.databricks.com \
  --remote-catalog prod --client-id <sp-id> \
  --client-secret-env UNITY_SECRET --location s3://acme-lake/prod
shctl catalog test lakehouse

# call SparkHouse's Jobs API with the open source Databricks SDK
from databricks.sdk import WorkspaceClient
w = WorkspaceClient(host="https://sparkhouse.example.com", token="<SparkHouse API token>")
run = w.jobs.run_now(job_id=123, job_parameters={"day": "2026-10-01"}).result()

From Snowflake

Tables first, then the work.

The most direct route runs through Apache Iceberg. Where your tables are available as Iceberg tables through the vendor's catalog interface, SparkHouse can read them without an export step. SQL, procedures and Python code need review and testing.

3 supported8 with limits1 not currently supported

What SparkHouse supports

AreaStatus in SparkHouseHow SparkHouse handles it
ComputeSupportedSQL endpoints in sizes XS to XL with auto-resume and auto-stop. Sizes are SparkHouse's own and do not correspond one to one to any other product's sizes, so re-estimate capacity and cost. SparkHouse meters uptime per endpoint, user, engine and placement.
Iceberg tables through Horizon CatalogSupported with limitsSparkHouse can connect to the Iceberg REST interface that Snowflake documents for Horizon Catalog (Snowflake documentation: access Iceberg tables with an external engine through Horizon Catalog), using a programmatic access token and a role. Which tables are reachable, with which permissions and at what cost, are set by Snowflake, so follow its documentation. This connection is built, and we have not yet recorded a run against a live Snowflake account.
Open CatalogSupported with limitsSparkHouse can connect to a Snowflake Open Catalog (Snowflake documentation: Open Catalog overview) with a service connection. Credentials for storage are vended per table, and the connection is read-only unless you switch on writes. Check Snowflake's documentation for current availability. This connection is built, and we have not yet recorded a run against a live account.
Data in other table formatsSupported with limitsSparkHouse reads Apache Iceberg tables. Data held in other formats has to be converted or copied into Iceberg first. Check your agreement and the vendor's documentation for export options and any charges.
SQLSupported with limitsQueries run on Spark SQL (Apache Spark and Sail) and on DataFusion. Syntax and functions differ between SQL dialects, so plan to review, translate and test each query and procedure.
Scheduled workSupportedJobs with SQL, notebook and Python tasks, cron schedules and time zones, retries, timeouts, task graphs and repair.
Ingestion and streamingSupported with limitsManaged Apache Kafka clusters, and a stream task that reads Kafka into Iceberg on Apache Spark. SparkHouse has no file-triggered ingestion service.
Python codeSupported with limitsNotebooks run Python cells. On an Apache Spark endpoint, spark is a Spark Connect session, and Ray runs distributed Python. Code written against another vendor's Python libraries needs to be adapted.
SQL editorSupportedWorksheets, a schema browser, parameters, explain, formatting, run history and result charts.
Drivers and BI toolsSupported with limitsSparkHouse offers an Arrow Flight SQL gateway and a SQL HTTP API. JDBC driver 19.0.0 passed 10 of 10 checks and ADBC 3 of 3, through a port-forward. ODBC and BI tools were not tried. The gateway is read-only. Applications use these interfaces in place of another vendor's drivers.
Roles and access controlSupported with limitsWorkspaces, groups, built-in roles and route authorization. Row access and masking policies are not currently supported.
Features SparkHouse does not currently provideNot currently supportedIn the version described here: data sharing between accounts, incrementally refreshed (dynamic) tables, managed AI services, an application framework, and a hosted Python app builder.

A suggested order of work.

  1. Take stock

    List databases, warehouses, tasks, streams, shares, stored procedures, functions and Python code. Mark anything in the not-currently-supported row.

  2. Check your agreements

    Read the terms that cover notice periods, minimum commitments, data export and fees before you move data or stop services.

  3. Make tables readable

    Use Iceberg tables that other engines can read, through the vendor's catalog interface, or copy data into Iceberg in your own storage.

  4. Connect the catalog

    Add the Horizon Catalog or Open Catalog connection, test it, and query one table from a SparkHouse endpoint.

  5. Translate and test SQL

    Rewrite queries and procedures that use dialect-specific syntax. Run each on SparkHouse and compare the results with the original.

  6. Rebuild compute and schedules

    Create endpoints, and recreate scheduled work as jobs with the same schedules.

  7. Repoint readers

    Move readers to the Flight SQL gateway or the SQL HTTP API. Writers need a SparkHouse job or a Spark connection.

  8. Compare, then cut over

    Run both in parallel, reconcile, then retire the old objects.

shctl
# add a Horizon Catalog connection
SF_PAT=... shctl catalog add sf --provider snowflake_horizon \
  --account-url myorg-myaccount --remote-catalog ANALYTICS \
  --user SVC_SPARKHOUSE --role ANALYST \
  --client-secret-env SF_PAT --location s3://acme-sf-iceberg
shctl catalog test sf

# then query it from any SparkHouse endpoint
SELECT * FROM sf.sales.orders LIMIT 10;

Before you move

Check these first.

General points for any migration. They are not specific to one vendor, and they are not legal advice.

  • Contracts. Your agreements may set notice periods, minimum commitments, data export terms and fees. SparkHouse cannot advise on them. Ask your legal and procurement teams.
  • Licences. Notebooks, libraries, functions and sample code can carry their own licences. Check them before you copy code to a new environment.
  • Personal and regulated data. Moving data can change where it is stored and who processes it. Review data protection obligations, processor agreements and residency rules for both environments.
  • Security review. Treat the new environment as a new system. Review access, secrets, network paths and audit logging before you move production data.
  • Testing and rollback. Run both environments in parallel, reconcile outputs, and keep a way back until the old one is switched off.

Legal notices

Trademarks and accuracy.

Trademarks. Databricks, Unity Catalog, Delta Lake, Snowflake, Horizon Catalog, Open Catalog, Microsoft Teams, PagerDuty, Ray, MLflow, vLLM and other product and company names on this page are trademarks or registered trademarks of their respective owners. Apache, Apache Iceberg, Apache Spark, Apache Kafka and Apache Polaris are names of projects of the Apache Software Foundation. They are used here only to identify products and interfaces that SparkHouse works with.

No affiliation. SparkHouse is independent. It is not affiliated with, sponsored by, endorsed by or certified by Databricks, Snowflake or any other company named on this page.

Third-party products. This page does not describe how other companies' products work. Where it refers to them, it points to their public documentation, which we read on 17 October 2026. Their products, terms and documentation change, so confirm against the current versions. SparkHouse makes no statement about the quality, price or roadmap of any other product.

SparkHouse statements. Statements about SparkHouse describe version 0.12.0 and its documentation, and can change. Where a feature is built and has not been run against a live third-party system, the page says so. "Supported" means built and documented, and is not a service-level commitment.

Not advice. This page is general information. It is not legal, tax, security or procurement advice, and it does not replace your own review of your agreements and obligations.

Corrections. If you believe a statement on this page is inaccurate or misuses a name, tell us through the contact form and we will review it promptly.

Next step

Ask for a migration assessment.

Send us your inventory: notebook count, job types, table formats and the tools that read your data. We will tell you what SparkHouse supports as is, what needs work, and what it does not do yet.

Request a pilotSee where it can run