Lineage You Didn't Have to Declare
BimlCatalog Derives Package Precedence From Observed Execution Timing
August 12, 2026
Ask what runs before what in your nightly load and someone will produce a diagram. Ask when that diagram was last updated and the room goes quiet. Declared dependency documentation rots because nothing forces it to stay true. The batch changes, packages come and go, and the diagram keeps describing last quarter's warehouse.
The execution log never has that problem. Every batch run writes exact start and end times for every package into your BimlCatalog, and that history is ground truth. So BimlFlex uses it. After every successful batch, the catalog re-derives which package ran immediately before which one, and stores the answer as metadata you can query in your own database.
Declared vs Observed
There are two kinds of load order, and they answer different questions. The declared order is design: what must wait for what, compiled from your model's dependencies at build time. The observed order is operations: what actually preceded what last night, once threads, retries, and a slow source system had their say.
Most control frameworks store only the first. The BimlCatalog stores both, in the same table. Each row in bfx.Package carries the ordering the build declared, and next to those columns sits PrecedencePackageID, a column no build step and no human ever populates. The catalog fills it in itself, by watching.
Set on Every Success
When a batch completes successfully, the end-of-run logging in your BimlCatalog calls a stored procedure named bfx.SetPackageLineage and hands it the batch's execution id. Both integration styles do this: the SSIS logging path and the ADF logging path each finish a successful batch by invoking the same procedure. There's no setting to turn on and no job to schedule. Success is the trigger.
We've written about BimlCatalog integration as the audit layer around every load, and about the restart and recovery state machine that resumes a failed batch without starting over. This is a different mechanism. Recovery reads failure states; lineage inference only ever runs on success, because only a completed run is evidence of a working order.
The Last Two Runs
The procedure starts by finding the last two successful executions of that batch: rows in bfx.Execution where ParentExecutionID = -1 (batch-level runs; child executions point at their parent instead), ExecutionStatus = 'S', and an end time on record. Those two runs define the active set: every package that appeared as a child execution in either of them.
The active set is a freshness filter. A package you removed from the batch two weeks ago doesn't appear in recent runs, so the update leaves it alone rather than rewriting its lineage from history it's no longer part of. Only packages present in the recent evidence get refreshed.
Finished Before It Started
The precedence itself comes from the most recent successful run alone. The procedure lines up that run's child executions by start time and applies one rule: a package's precedence is the package that finished immediately before it started.
A worked example. Suppose last night's batch ran four packages like this:
STG_Customer: 01:00:00: 01:03:10: none
STG_Order: 01:00:00: 01:07:45: none
HUB_Customer: 01:03:15: 01:04:20: STG_Customer
SAT_Customer: 01:04:25: 01:06:00: HUB_Customer
STG_Customer and STG_Order both started at batch open, when nothing had finished yet, so neither gets a precedence. They're the openers. HUB_Customer started five seconds after STG_Customer finished, so STG_Customer is its precedence. And SAT_Customer points at HUB_Customer, not STG_Order, because STG_Order was still running when SAT_Customer started. Observed precedence records what the run actually did, parallel lanes included.
That last point is the useful one. A declared graph tells you SAT_Customer needs HUB_Customer. The observed chain tells you your staging loads really do run in parallel while the vault loads thread between them, which is the kind of fact you want in front of you when you're deciding whether the batch has room to run wider.
Query It Yourself
The result is ordinary rows in your own database:
SELECT p.PackageName,
prior.PackageName AS RanImmediatelyBefore
FROM bfx.Package AS p
LEFT JOIN bfx.Package AS prior
ON prior.PackageID = p.PrecedencePackageID
WHERE p.IsBatch = 0
ORDER BY p.PackageName;
A NULL in the second column marks a batch opener. Walk the chain with a recursive CTE and you get the observed spine of the whole batch, straight from metadata. No log parsing, no scraping a scheduler UI, no export.
This works because the control framework is a database you own: plain tables and procedures deployed to your own server, readable with the same tools you point at everything else. The procedure that maintains PrecedencePackageID is deployed right alongside the tables it updates, so a skeptical DBA can read exactly how the value is derived before trusting it.
Where It Fits
Be clear about what this is. Observed precedence is evidence, not a contract. It says that in the last successful run, this package finished right before that one started. It does not say the second required the first, and a run with different thread timing can legitimately produce a different chain. Treat it as operational lineage: the batch's own account of its most recent good night.
That still answers a lot of real questions. What does the tail of my batch look like? Did the load order actually change after the last regeneration? What ran just before the package that's suddenly slow? Those used to be log-archaeology questions. Now they're a SELECT, current as of the last successful run, because the catalog rewrites the evidence each time one completes.
Hand-maintained lineage documentation is a tax, and most teams quietly stop paying it. The whole case for metadata-driven automation is retiring that kind of manual upkeep so the same people ship more warehouse. A load-order record that rewrites itself after every successful run is the only kind that stays true. Yours is already doing it. It's one self-join away.
