Databricks

Beta Feature: This node is currently in beta. Features and functionality may be subject to changes.

The Databricks Node lets your workflow query a table in a Databricks SQL Warehouse. Instead of writing raw SQL, you pick a table from the Unity Catalog (catalog.schema.table), tick the columns you want, and add filter conditions in the UI. The node builds a safe, parameterised SELECT statement for you and returns the matching rows.

Databricks Support: For any enquiries or issues related to Databricks, please contact the GovTech Databricks team at cdo@tech.gov.sg.

Prerequisites

Before using the Databricks Node, ensure the following:

• Databricks Workspace — You must have access to a Databricks workspace.
• SQL Warehouse — A running Databricks SQL Warehouse to execute the query.
• Unity Catalog Access — The catalog, schema, and table you want to query must be accessible to your account.
• Personal Access Token (PAT) — A Databricks personal access token for authentication. The token must be granted the sql and unity-catalog scopes. Store it as an environment variable and reference it rather than hardcoding it (see the Authentication section).

GovTech-hosted Databricks: If you select the GovTech agency option, the server hostname and SQL Warehouse ID are resolved automatically from platform configuration — you only need to provide a Personal Access Token.

How Does It Work?

The Databricks Node connects to your SQL Warehouse using the Databricks SQL Statements API. You configure the query visually, and the node constructs and runs a SELECT statement against the chosen Unity Catalog table.

Setup Steps:

1. Build your workflow using Graph Edit Mode (minimum setup: input node → Databricks node → output node, but you can add more nodes as needed)
2. Click on the Databricks node settings to open its configuration
3. Select the Agency and enter your Personal Access Token, then click Connect
4. Choose the Catalog, Schema, and Table from the cascading dropdowns
5. (Optional) Add Filters to narrow down the rows returned
6. (Optional) Select the Output Columns you want (leave all selected to return every column)

Output:

The node runs the query and returns the matching rows as an array of objects, keyed by column name:

{
  "results": [
    { "id": 1, "name": "Alice", "created_at": "2026-01-15" },
    { "id": 2, "name": "Bob", "created_at": "2026-02-03" }
  ]
}Copy to clipboardErrorCopied

• results: An array of row objects. Each object maps column names to their values for that row.

Error handling: Which status codes count as success or error is configurable — see Response Handling below. When a status code resolves to success, its response is returned as output (in the form { "status": <code>, "data": <details> }) so a downstream Decision or Switch node can branch on it. Any code that resolves to error (and network failures/timeouts) fails the node, which then follows its error path and retry policy.

Configuration

Connection

Agency

Determines how the connection to Databricks is resolved.

Server Hostname

Required when Agency is Other Agency.

The Databricks workspace hostname, without the protocol.

Example: adb-1234567890.12.azuredatabricks.net

SQL Warehouse ID

Required when Agency is Other Agency.

The ID of the Databricks SQL Warehouse that will execute the query.

Example: abc123def456

Personal Access Token

Your Databricks personal access token, sent as a Bearer token to authenticate the request. This field is required. The token must be granted the sql and unity-catalog scopes so the node can browse the Unity Catalog and run queries on the SQL Warehouse.

Tip: The Personal Access Token is sensitive. Store it as an environment variable and reference it using the field selector (e.g. {{{$env.DATABRICKS_TOKEN}}}) rather than pasting it directly.

Click Connect after entering your credentials. On success, the node discovers the available catalogs so you can configure the query.

Parameters

Databricks uses a three-level Unity Catalog namespace — catalog.schema.table. All three are required and are chosen from cascading dropdowns that populate after you connect.

Selecting a catalog loads its schemas; selecting a schema loads its tables; selecting a table loads its columns for use in filters and output selection.

Filters

Filters let you narrow down the rows returned. Each filter is a condition on a single column, and multiple filters are combined with AND. If no filters are added, all rows are returned (up to the row limit).

The operators available depend on the column’s data type:

Note: The Is Null and Is Not Null operators do not take a value. All other operators require a value. The filter value field supports template variables (e.g. {{{input.userId}}}) so you can reference data from previous nodes.

Only columns with simple (primitive) data types can be filtered. Complex types such as arrays, maps, and structs are not available in the filter column dropdown.

Output Columns

Select which columns to include in the results using the checkboxes. Use Select All / Deselect All to toggle every column at once.

If no columns are selected, the query returns all columns by default.

Row Limit

Maximum 1000 rows: The node returns at most 1000 rows per query. There is no pagination — if you need a smaller, more specific result set, add filters to narrow the query.

Example

Querying Active Users

This example retrieves the id, name, and email of users in the main.sales.users table whose status is active.

Node Settings

• Agency: GovTech
• Personal Access Token: {{{$env.DATABRICKS_TOKEN}}}
• Catalog: main
• Schema: sales
• Table: users
• Filters: status Equals active
• Output Columns: id, name, email

Output:

{
  "results": [
    { "id": 1, "name": "Alice", "email": "alice@example.gov.sg" },
    { "id": 2, "name": "Bob", "email": "bob@example.gov.sg" }
  ]
}Copy to clipboardErrorCopied

Response Handling

Response Handling lets you decide which HTTP status codes returned by Databricks count as a successful execution of the node. You list the status codes (or ranges) that should be treated as success — everything else is treated as an Error, including network failures and timeouts.

How it works:

• Treat as Success — Add the status codes or ranges that indicate success. When Databricks responds with one of these, the node completes normally and passes its response to the next node.
• Everything else → Error — Any status code not listed as success (as well as network failures and timeouts) fails the node, which then follows its error path and retry policy.

Configuring the codes:

• Add a code or range — Type a single 3-digit code (e.g. 404) or an inclusive range (e.g. 400-429) and click Add.
• Quick add — Use the shortcut chips (2xx, 3xx, 4xx, 404) to add common patterns instantly.
• Remove — Click the × on any code chip to remove it from the success list.
• Reset to defaults — Restores the default configuration.
• Check a status code — Enter a status code in the tester to preview whether it would resolve to Success or Error with your current configuration.

Default configuration (new nodes):

Tip: Most workflows only need the defaults. Extend the success list when Databricks returns a non-2xx/3xx code that your workflow should treat as a valid, non-retryable outcome — for example, treating a client 4xx as a branchable result rather than a failure.

Older nodes: Databricks nodes created before this feature was introduced keep their original behaviour until you first edit and save the Response Handling section.

This is the same Response Handling behaviour available on the API node.

Simulation Behaviour

The Databricks Node supports simulation and mock mode. During simulation, the node returns mock results instead of querying the live SQL Warehouse, so you can test your workflow without executing a real query.

When to Use It?

Use Databricks Node when:

• You need to read data from a Databricks SQL Warehouse table as part of a workflow.
• You want to query Unity Catalog data using structured filters without writing SQL.
• You need to enrich a workflow with reference or lookup data stored in Databricks.

Use the API Node instead when:

• You need to call an arbitrary Databricks REST API endpoint not covered by this node.
• You are integrating with a data source that is not a Databricks SQL Warehouse.