The Restart That Skips the Copies
One Catalog Flag Reruns The Databricks Job, Not The Extracts
September 9, 2026
A Databricks pushdown load has two halves, and they cost very different amounts to repeat. The first half is copy work: Data Factory copy activities pulling source tables into the lake. The second half is the Databricks job that transforms what landed. When that job dies at 2 a.m., the copies have usually already succeeded. The expensive, source-facing, window-sensitive part of the night is done. Only the compute part failed.
A naive retry throws the first half away. It re-extracts every table it already extracted, leans on source systems that already paid the I/O bill once, and only then gets around to retrying the thing that actually broke. Worse than the waste is the risk: extraction windows move. Re-copying at 6 a.m. is not guaranteed to land the same data the 2 a.m. run landed, so a full restart can quietly change what the load was loading.
BimlFlex generates a rerun that knows the difference. The recovery flag lives in the BimlCatalog you deploy and own, the skip logic lives in the pipeline JSON you generated, and the audit trail comes out whole. Here is the machinery, end to end.
Two halves, two costs
In a Databricks pushdown project, BimlFlex splits the work exactly along the line described above. The generated Data Factory batch pipeline runs the extract legs first, each one landing source data in the lake. Then a single Databricks Job activity hands off to the notebooks that do the real transformation work inside Databricks: staging, Data Vault, marts, whatever the project's model calls for. We have written an introduction to Databricks solutions with BimlFlex and a deeper look at why pushing transformation processing down into Databricks cuts cost; the short version is that data moves once and compute happens where the data already sits.
That split is great for performance and lousy for naive retries, because the two halves fail differently. Copy activities fail early, loudly, and cheaply rerunnable. A Databricks job can fail late, after every copy has finished, for reasons that have nothing to do with the data: a cluster that would not start, an exhausted pool, a transient Unity Catalog hiccup. The right recovery for that failure is obvious to a human operator. Rerun the job. Do not touch the copies. The interesting part is how the generated pipeline reaches the same conclusion without a human awake.
The flag is a D, not an R
Every generated BimlFlex pipeline logs its runs to the BimlCatalog, the operational database you deploy from a DACPAC and can open with any SQL client. We walked through that conversation in the dev diary on BimlCatalog integration. The relevant part here is the failure path. When a load fails, an error-logging activity fires and writes a recovery instruction into bfx.Execution for the run that has not happened yet. For an ordinary failure that instruction is R, the general rerun-and-recover flag whose platform-by-platform behavior is part of the restart-and-recovery state machine your BimlCatalog already owns.
A Databricks pushdown batch gets a more specific failure story. The error handler wired to the Databricks Job activity in the generated pipeline passes an extra parameter you can see in the pipeline JSON: SkipCopyOnRestart, set to true. With that parameter set, the catalog records the failure differently. The batch is still marked failed, but its next-load instruction becomes D instead of R: a Databricks-only restart. The flag encodes what the pipeline knows at that moment, which is that the failure happened downstream of the copies.
You can watch this in your own database after a bad night:
SELECT ExecutionID, OriginalExecutionID, ExecutionStatus, NextLoadStatus
FROM bfx.Execution
WHERE PackageID = @PackageID
ORDER BY ExecutionID DESC;
The dead run sits there with ExecutionStatus = 'F' and NextLoadStatus = 'D'. Not a status. An instruction to the next run.
The copies stay closed
When the next execution starts, its first activity calls the start-logging procedure in the catalog, which reads the previous run's D and starts the new execution in a dedicated Databricks restart status. That status flows back into the pipeline as part of the procedure's result, and the generated pipeline is built to route on it.
Every copy leg in the generated batch pipeline lives inside an If Condition activity. Open the deployed pipeline in the Data Factory UI, or read its JSON in your repo, and you will find expressions of this shape guarding the extract containers:
@or(equals(activity('LogExecutionStart').output.firstRow.ExecutionStatus, 'E'),
equals(activity('LogExecutionStart').output.firstRow.ExecutionStatus, 'R'))
E is a normal execution. R is a general recovery run. A Databricks restart is neither, so on a D run every one of those conditions evaluates false and every copy container closes without doing work. The pipeline flows straight through them to the Databricks Job activity, which runs as it always runs. Skipping is not a special rerun mode someone has to remember to trigger from a runbook. It is the deployed pipeline evaluating expressions that were generated into it, against state your catalog recorded at failure time.
Walking back to the first failure
One failure is the easy case. Now let the rerun fail too. Maybe the cluster pool is still exhausted at 3 a.m., and again at 4. Each failed restart writes another D, and each new attempt is one more execution row removed from the run that actually landed the data.
The start-logging procedure handles this by walking the chain. When it reads a D, it does not just look at the immediately previous run. It walks back through the contiguous chain of Databricks-restart failures until it finds the execution where the copies actually ran, the first one in the chain. That original run's identifier is what the restart carries forward, and the catalog records it in the OriginalExecutionID column of the new execution row. Run the query above after a rough night and you can read the whole story: a chain of failed restarts, each pointing home to the one execution that did the copy work.
The detail worth pausing on is what this protects. The data the transform needs is the data landed by that first run. Attempt number four at 5 a.m. must process the 2 a.m. extract, not whatever a fresh copy would have fetched. The walk-back is what pins the rerun to it.
One execution owns the rows
Threading that original identifier into Databricks is the last piece, and it is also sitting in your pipeline JSON. The generated Databricks Job activity passes a job parameter named row_audit_id, and its value is an expression:
@if(equals(activity('LogExecutionStart').output.firstRow.ExecutionStatus, 'D'),
string(activity('LogExecutionStart').output.firstRow.LastExecutionID),
string(activity('LogExecutionStart').output.firstRow.ExecutionID))
On a normal run, the notebooks stamp their rows with the current execution's ID. On a Databricks restart, they stamp with the original execution's ID instead, the one the walk-back found. So the rows the rerun writes are attributed to the execution that landed their source data, and the audit trail describes one logical load rather than a data load split across four execution IDs. Query lineage later and there is no seam: copies and transforms reconcile to the same identity, even though a human would tell you the night had four attempts in it.
That choice is easy to underrate. Plenty of orchestration setups can retry a job. Far fewer can retry a job and keep row-level audit attribution coherent across the retry boundary, because that requires the retry logic and the audit logic to share state. Here they share the catalog.
Recovery in artifacts you own
Scope honesty: this is the recovery model for Databricks pushdown processing orchestrated by Data Factory, and it is the newest of the recovery paths in the product. SSIS has its own, older recovery story with a full rollback container, and plain ADF loads lean on fenced re-execution; those are a different post. What this path shares with the others is where it lives. The flag is a column in your database. The skip guards and the audit expression are in pipeline JSON you generated and can diff in source control. The walk-back is a stored procedure you can read. Nothing in the recovery path phones home, and nothing about it requires a vendor service to be reachable during your outage.
That is not the default in this market. Reviewers of Qlik Compose report that on large volumes it "tries to do everything in one shot" without batching, which makes the one-shot itself the retry unit: when everything is one shot, recovery means the whole shot again. And the pattern where restart intelligence lives inside a vendor's bundled scheduler, rather than in artifacts you keep, is common enough that we treat catalog-owned recovery as a feature worth explaining, not a given. If you are building toward this architecture, our guide to streamlining Databricks data integration with BimlFlex covers the landing-zone and job setup this post assumes.
The next time a Databricks job dies after midnight, check bfx.Execution before you check anything else. If the copies landed, you will find a D waiting, and the morning rerun will do exactly what a careful operator would have done by hand: leave the finished work alone, restart the failed half, and file every row under the run that started the story.
