How to Connect dbt™ to Databricks in Paradime: The Complete Setup Guide

Getting dbt™ and Databricks to play nicely together shouldn't feel like assembling IKEA furniture without instructions. Between SQL Warehouses, Unity Catalog permissions, and token strategies, there are real configuration decisions that can make or break your data pipeline.

This guide walks you through every step of connecting dbt™ to Databricks in Paradime — from understanding what you're actually configuring, to validating your connection, to fixing the issues that inevitably pop up. Whether you're an admin wiring up the workspace or a developer getting your personal environment running, this is the definitive reference.

Databricks + dbt™ in Paradime: What You're Setting Up

Before diving into configuration fields and tokens, it helps to understand the two distinct connections Paradime needs to your Databricks environment. Each serves a different purpose, uses different credentials, and targets a different phase of your development lifecycle.

Figure 1: Paradime connects to Databricks through two separate paths — one for interactive development and one for production scheduling.

IDE Connection for Development

The Code IDE connection is what every developer on your team uses day-to-day. When you write a dbt™ model in the Paradime Code IDE, compile it, or preview data with the Data Explorer, this connection is what executes queries against Databricks.

Key characteristics of the IDE connection:

• Each developer gets their own credentials — specifically their own Personal Access Token (PAT) and their own default schema (e.g., dbt_john, dbt_sarah).

• The admin configures workspace-level settings like the host, HTTP path, and Unity Catalog name once.

• Targets the dev catalog in Unity Catalog so developers can't accidentally write to production.

This separation means you can have 20 developers all working in the same Paradime workspace, each with isolated sandbox schemas, without stepping on each other's work.

Bolt Connection for Production Scheduling

The Bolt connection handles your production dbt™ runs — the scheduled jobs that build your production tables and run your tests on a cadence. This is a fundamentally different connection:

• Uses a Service Principal Token (SPT) instead of individual PATs — no single developer's credentials are tied to production.

• Targets the prod catalog with a schema like dbt_prod or whatever your organization's naming convention dictates.

• Runs on a schedule with support for standard, deferred, and Turbo CI execution modes.

The critical point: these are two separate connections in Paradime's Settings. If you only configure one, you'll either be unable to develop interactively or unable to run production jobs.

Prerequisites (Databricks)

Before you open Paradime's settings page, you need a few things ready on the Databricks side. Getting these wrong is the number one source of "why isn't my connection working?" tickets.

SQL Warehouse Endpoint Details

dbt™ on Databricks can run against two types of compute: SQL Warehouses or All-Purpose Clusters. The dbt™ documentation strongly recommends SQL Warehouses, and for good reason:

To get your SQL Warehouse connection details:

1. Log in to your Databricks workspace

2. Navigate to SQL → SQL Warehouses

3. Click on your target warehouse

4. Open the Connection Details tab

5. Copy the Server Hostname and HTTP Path

Your HTTP path will look something like: /sql/1.0/warehouses/fbc246efa4369412

Tip: Consider using a smaller SQL Warehouse for development (2X-Small or X-Small) and a larger one for production. As your project grows, you can even assign different compute per model using databricks_compute config.

Access Token / Service Principal Strategy

Databricks authentication for dbt™ requires tokens. The recommended strategy uses two different token types for the two connections:

Figure 2: Token strategy — PATs for developers, SPT for production. Never share tokens across environments.

For development (PAT):

• Each developer generates their own PAT in Databricks: User Settings → Developer → Access Tokens → Generate New Token

• Set a lifetime of 90 days or less (Databricks recommends this as a best practice; the max is 730 days)

• Developers enter their PAT in Paradime's connection settings at the user level

For production (Service Principal Token):

• Create a Databricks service principal dedicated to production dbt™ runs

• Generate a PAT for the service principal (this is done programmatically via REST API or Databricks CLI)

• Grant the service principal the minimum required permissions on your prod catalog

• Enter the SPT in Paradime's Bolt connection settings

Security note: Never use a personal developer token for production scheduling. If that developer leaves the organization or their token expires, your production pipeline breaks silently.

Unity Catalog: Catalog / Schema Mapping and Permissions

Unity Catalog changes how Databricks organizes data. Instead of the legacy Hive metastore's flat schema.table namespace, Unity Catalog introduces a three-level hierarchy: catalog.schema.table.

Figure 3: Unity Catalog's three-level namespace enables clean environment isolation for dbt™.

Create your catalogs:

Required permissions by role:

Grant these permissions in Databricks:

Important: dbt™ requires CREATE SCHEMA on the catalog because it automatically creates schemas at runtime. You don't need to pre-create developer schemas — dbt™ handles that on the first dbt run.

Create the Databricks Connection in Paradime

With your Databricks prerequisites in place, you're ready to configure Paradime. You'll create two connections: one for the Code IDE (development) and one for Bolt (production).

Settings → Connections

1. Click Settings in the top menu bar of the Paradime interface

2. In the left sidebar, click Connections

3. You'll see two sections: Code IDE and Bolt Schedules

IP Allow-listing: Before configuring connections, ensure your Databricks workspace firewall allows traffic from Paradime's IP addresses. The specific IPs depend on your selected data location in Paradime.

Host, HTTP Path, Token Configuration

Code IDE Connection (Development):

Click Add New next to the Code IDE section and select Databricks.

Bolt Connection (Production):

Click Add New next to the Bolt Schedules section and select Databricks.

Catalog / Schema Defaults

The Unity Catalog Name field in Paradime maps directly to the catalog key in your profiles.yml. When set:

• All dbt™ models default to writing into the specified catalog

• The Schema field becomes the default schema within that catalog

• Your models land at: ..

If you leave Unity Catalog Name blank, Databricks falls back to the legacy Hive metastore (hive_metastore catalog). For any new project, always set the catalog explicitly.

Behind the scenes, Paradime generates a profiles.yml that looks like this:

You never need to edit this file manually — Paradime generates it from your connection settings. But understanding the mapping helps when debugging.

Configure Development in Code IDE

With the connection in place, there are a few dbt™ project-level configurations that make multi-developer workflows smooth.

Dev Schema Strategy

The recommended pattern is one schema per developer within the dev catalog. Paradime handles this by letting each developer set their own schema name in their user-level connection settings.

To prevent developers from accidentally writing to production schemas, use the built-in generate_schema_name_for_env macro. Create this file in your project:

This macro changes behavior based on the target name:

• When target.name is prod: Uses custom schema names directly (e.g., a model with +schema: marketing lands in the marketing schema)

• When target.name is anything else (like dev): All models land in the developer's personal schema (e.g., dbt_john), regardless of custom schema configuration

This means in development, all your models are neatly contained in your sandbox. In production, they fan out to their intended schemas.

Figure 4: The generate_schema_name_for_env macro consolidates all dev models into one sandbox schema while maintaining organized production schemas.

Working with Multiple Catalogs

Some organizations need models to span multiple catalogs — for example, writing staging models to a staging catalog and mart models to a analytics catalog. dbt™ supports this via the database config (which maps to "catalog" in Databricks).

Per-folder in dbt_project.yml:

Per-model in SQL files:

For dynamic catalog switching between environments, override the generate_database_name macro:

This ensures that in development, everything writes to the dev catalog (from your connection settings), while in production, models respect their configured catalog assignments.

Configure Production in Bolt Scheduler

Bolt is Paradime's production orchestration engine. Once your Bolt connection to Databricks is configured (see the connection section above), you need to set up the actual schedules.

Prod Credentials and Separation from Dev

The Bolt connection uses a completely separate credential path from the Code IDE:

This separation is critical. Even if a developer's PAT expires or gets revoked, production continues running uninterrupted because it uses a completely different authentication path.

Schedule Configuration and Retries

Bolt supports three schedule types:

1. Standard — Runs your full dbt™ command set on a time-based schedule (cron, hourly, daily, etc.)

2. Deferred — Only runs models that have changed since the last successful run, using manifest comparison. Ideal for CI/CD workflows and saving compute costs.

3. Turbo CI — Automatically triggered on pull requests, builds only modified models and their dependencies in a temporary schema.

To create a schedule in the Bolt UI:

1. Navigate to Bolt in Paradime

2. Click Create Schedule

3. Choose your schedule type (Standard, Deferred, or Turbo CI)

4. Configure the Commands — for example:

5. Set Trigger Type: time schedule, after another job, on merge, or API call

6. Configure Notifications: email, Slack, or Microsoft Teams for failures and SLA breaches

Retry behavior: When a command fails, Bolt skips subsequent commands by default. There are exceptions: dbt source freshness failures won't block a downstream source_status:fresher+ command, and Elementary CLI commands always run (so you still get failure alerts). For deferred schedules, you can configure "Last successful run" mode, which uses artifacts from the most recent successful run — useful when retrying after a partial failure.

For YAML-based configuration (schedules as code), see the Paradime Bolt docs.

Validate with SQL Scratchpad

After configuring your connections, you need to verify everything works before running your first dbt run. Paradime's Scratchpad is the fastest way to do this — it provides a gitignored, temporary SQL environment right in the Code IDE.

SHOW CATALOGS / SHOW SCHEMAS Checks

Open a new Scratchpad file in the Code IDE and run these queries to validate your Unity Catalog access:

Expected results:

• SHOW CATALOGS should list at least dev and your source data catalog (e.g., raw)

• SHOW SCHEMAS in the dev catalog may be empty if no one has run dbt™ yet — that's fine

• If you don't see the expected catalogs, it's a permissions issue (jump to Troubleshooting)

Permission Test (Create Schema / Table if Needed)

If the basic checks pass, validate that dbt™ will be able to create its schemas and tables:

If any of these fail with a permissions error, you need to go back to Databricks and grant the appropriate privileges on the dev catalog (see the Prerequisites section above).

You can also use the Data Explorer in the Code IDE to preview compiled SQL from your actual dbt™ models — click the Data Explorer icon, select a model, and hit Preview Data to see if it compiles and returns results against your Databricks connection.

Troubleshooting

Even with everything configured correctly, things break. Here are the most common issues and how to fix them.

HTTP Path / Endpoint Mismatch

Symptom: Connection timeouts, "Invalid HTTP path" errors, or queries that hang indefinitely.

Common causes:

1. Copied the cluster HTTP path instead of SQL Warehouse path: Cluster paths look like /sql/protocolv1/o/1234567890/0123-456789-abcdefgh. SQL Warehouse paths look like /sql/1.0/warehouses/fbc246efa4369412. Make sure you're using the SQL Warehouse path.

2. SQL Warehouse is stopped or deleted: Navigate to SQL → SQL Warehouses in Databricks and verify the warehouse is running. Serverless warehouses auto-start, but classic/pro warehouses may be stopped.

3. Hostname mismatch: Double-check you're using the correct workspace hostname. If your org has multiple Databricks workspaces, it's easy to copy the hostname from the wrong one.

Fix: Go to your SQL Warehouse's Connection Details tab in Databricks and re-copy both the Server Hostname and HTTP Path. Update them in Paradime's Settings → Connections.

Unity Catalog Permission Denied

Symptom: Errors like PERMISSION_DENIED: User does not have USE CATALOG on catalog 'dev' or User does not have CREATE on CATALOG 'prod'.

Diagnosis steps:

Common fixes:

1. Missing USE CATALOG: This is the most basic permission — without it, you can't even see the catalog exists.

2. Missing CREATE SCHEMA: dbt™ needs this to create developer sandbox schemas on first run.

3. Prod vs Dev confusion: A developer's PAT should not have write access to the prod catalog. If you're getting permission denied on prod during development, check that your Paradime Code IDE connection's Unity Catalog Name is set to dev, not prod.

4. Service principal missing grants: For Bolt production runs, ensure the service principal has been explicitly granted permissions. Service principals don't inherit user-level grants.

Token Expiry and Rotation

Symptom: Previously working connections suddenly fail with authentication errors. Bolt schedules start failing on a schedule that matches your token's creation date + lifetime.

PAT expiry:

• Databricks PATs have a maximum lifetime of 730 days (2 years), but best practice is 90 days or less

• There's no automatic rotation — you must manually generate a new token and update it in Paradime

• To update: Go to Paradime → Settings → Connections → Click on your Code IDE connection → Update your Personal Access Token

Service Principal Token expiry:

• Same lifetime limits apply

• Set a calendar reminder for rotation

• Consider automating rotation using the Databricks Token Management API and updating the Paradime connection via the Paradime API

Proactive monitoring:

• In Databricks, you can create alerts for expiring PATs using system tables

• Set token lifetimes to a consistent duration (e.g., 90 days) across your team

• Document your rotation schedule — especially for the production service principal token

Figure 5: Token rotation workflow — generate, update, verify, then revoke. Don't revoke the old token until the new one is confirmed working.

Wrapping Up

Connecting dbt™ to Databricks through Paradime involves three layers of configuration: Databricks prerequisites (SQL Warehouse, tokens, Unity Catalog permissions), Paradime connections (Code IDE for dev, Bolt for prod), and dbt™ project configuration (schema strategies, catalog mapping). Getting each layer right means the difference between a smooth development workflow and hours of debugging permission errors.

The key takeaways:

• Use SQL Warehouses, not All-Purpose Clusters — they're optimized for dbt™'s SQL workload and cost less

• Separate dev and prod completely — different catalogs, different tokens, different connection configs

• Unity Catalog changes the game — embrace the three-level namespace (catalog.schema.table) and use generate_schema_name_for_env to manage environment differences

• Validate before you build — use Paradime's Scratchpad to run SHOW CATALOGS and permission checks before your first dbt run

• Plan for token rotation — 90-day lifetimes with documented rotation procedures save future pain

For the complete Paradime Databricks connection reference, see the development environment docs and scheduler environment docs. For dbt™ + Unity Catalog best practices, the dbt™ Unity Catalog guide is essential reading.