Your Databricks Job Graph Is YAML You Own

Ask a data team where their pipeline dependency logic lives and watch where they point. Sometimes it's a canvas in an orchestration GUI. Sometimes it's rows in a scheduler database. Often it's a diagram on a wiki that stopped matching production two quarters ago. The load order that decides whether tonight's warehouse run succeeds is real, load-bearing logic, and in most shops it's the one piece of logic nobody can diff.

‍

Orchestration You Can't Read

‍

Transformation code gets code review. Dependency graphs mostly don't, because they don't live in files. When the ordering breaks, you're clicking through a DAG viewer at 2 a.m. trying to reconstruct what was supposed to wait for what.

‍

Databricks solved the storage half of this with Asset Bundles: jobs declared in YAML, kept in source control, deployed with the CLI. Writing that YAML by hand for a real warehouse is the other half. A model with 300 loads means hundreds of task blocks, and every depends_on line encodes a decision someone has to get right today and keep right as the model grows. Nobody enjoys maintaining that file, so it rots.

‍

BimlFlex treats the job graph the same way it treats DDL and load code: as an output. You model batches, dependencies, and parallelism as metadata, and the build compiles them into a bundle you own.

‍

What a Build Hands You

‍

Build a BimlFlex project for Databricks with pushdown processing enabled (the execution model behind that is covered in push-down processing on Databricks) and the output folder contains a working Asset Bundle:

‍

• databricks.yml at the root: dev and prod targets, an include for resources/*.yml, and variables holding your job cluster definitions and an optional Git source
• one job file per batch under resources/: staging, delete detection, Data Vault, data warehouse, point-in-time, and bridge batches each get their own
• the PySpark notebooks those jobs run, plus a small Python utility module and a cluster init script

‍

Run databricks bundle deploy and the jobs exist in your workspace. There's no BimlFlex service in the execution path afterward. The workspace is running ordinary bundle-deployed jobs, the same way it would if you'd typed the YAML yourself. Getting sources connected and landing zones set up comes first, and we've written about Databricks source connections and landing zones separately. This post is about what's inside those job files.

‍

Solve Order Becomes Sub-Jobs

‍

BimlFlex derives execution layers from the dependencies in your metadata. A satellite can't load before its hub, a bridge can't build before the links it spans, and that ordering surfaces as a solve order on each object (the modeling side of this is covered in Data Vault acceleration on Databricks). In the generated bundle, every distinct solve order in a batch becomes its own sub-job, and the batch's parent job chains them in sequence:

‍

resources: 

jobs: 

EXM_DV_Job: 

name: EXM_DV_Job 

tasks: 

- task_key: EXM_DV_Job_1 

run_job_task: 

job_id: ${resources.jobs.EXM_DV_Job_1.id} 

- task_key: EXM_DV_Job_2 

depends_on: 

- task_key: EXM_DV_Job_1 

run_job_task: 

job_id: ${resources.jobs.EXM_DV_Job_2.id} 

parameters: 

- name: row_audit_id 

default: "0" 

tags: 

bfx: EXM_DV_Job 

‍

(Names shortened for the page; the structure is what the generator emits.)

‍

Each task is a run_job_task pointing at a sub-job through a bundle-internal reference, so the whole graph resolves inside the bundle with no hardcoded job ids. Every solve-order group after the first carries a depends_on for the group before it: layer two cannot start until everything in layer one has finished, which is the guarantee a Data Vault load order actually needs. Parameters are aggregated up from every notebook in the job, so a single run threads one row_audit_id through every task for the runtime audit trail, and the bfx tag makes generated jobs easy to find in a busy workspace.

‍

Threads Set the Parallelism

‍

Inside a sub-job, ordering is decided per object. Each object in a batch carries a thread number, driven by the batch's Threads setting, which controls how many groups of loads may run at once. The generator turns those numbers into chains. Consecutive tasks on the same thread get a depends_on linking them serially. A new thread number starts a fresh chain with no dependency, so it launches in parallel with the others. And an object flagged Own Thread gets a lane to itself, so a slow load never queues behind it.

‍

EXM_DV_Job_1: 

name: EXM_DV_Job_1 

tasks: 

- task_key: HUB_Customer 

notebook_task: 

notebook_path: /Workspace/bfx/EXM/awlt/HUB_Customer_00_Task 

source: WORKSPACE 

job_cluster_key: bfx_cluster 

- task_key: HUB_Order 

depends_on: 

- task_key: HUB_Customer 

notebook_task: 

notebook_path: /Workspace/bfx/EXM/awlt/HUB_Order_00_Task 

source: WORKSPACE 

job_cluster_key: bfx_cluster 

- task_key: HUB_Product 

notebook_task: 

notebook_path: /Workspace/bfx/EXM/awlt/HUB_Product_00_Task 

source: WORKSPACE 

job_cluster_key: bfx_cluster 

job_clusters: 

- job_cluster_key: bfx_cluster 

new_cluster: ${var.bfx_cluster} 

queue: 

enabled: true 

‍

Two lanes here: HUB_Customer then HUB_Order run serially on one thread while HUB_Product runs beside them. The cluster wiring is explicit too. Tasks bind to a job cluster key, or to an existing cluster id when the object's Databricks cluster settings say to reuse one, and the cluster definition itself lives as a variable in databricks.yml where your environments can override it. Serverless deployments swap the cluster binding for an environment specification instead. Sub-jobs also enable queueing, so overlapping batch runs wait rather than collide.

‍

YAML You Can Diff

‍

Here's why generation into a bundle beats an equivalent graph drawn in a UI: text has tooling. Raise the batch's Threads setting and rebuild, and the pull request shows precisely which depends_on lines vanished. A reviewer can see the parallelism change before it burns compute, because how wide a job fans out is a cost decision as much as a scheduling one (we covered that trade in cost-aware orchestration). Six months from now, git log on the job file is the history of every orchestration decision your team made, with authors and dates. Rolling back a bad one is a revert and a redeploy, not an archaeology session in a scheduler UI.

‍

Add a source table and the model's dependencies change, so the job graph changes. With a hand-drawn DAG that means someone remembers to redraw it. Here it means you rebuild, and the graph is correct because it was compiled from the same metadata as the loads themselves. Regenerate, don't redraw.