The Pipeline That Knows Its Limits
ADF Caps Pipelines At 80 Activities, BimlFlex Budgets And Shards For It
August 28, 2026
Azure Data Factory has a number most teams meet at the worst possible moment: 80. A single pipeline can hold at most 80 activities. That's a platform limit, not a quota, and no support ticket raises it. Hand-built orchestration projects usually discover it months in, when the batch finally crosses the line and the next publish fails validation.
BimlFlex has been generating Data Factory pipelines since ADF v2 shipped, and the generator treats that limit the way a compiler treats a register file: as a budget to allocate against, not a surprise to hit. Every object in a batch has a known activity cost. The generator packs pipelines up to a configurable ceiling, spills the overflow into chained sub-pipelines, and when a batch can't be made to fit at all, it refuses at build time with an error that tells you what to change.
A hard cap, discovered late
Nobody designs for the 80-activity limit on day one. Day one is a dozen tables. The batch grows a few sources per sprint, each one adding a copy here, a lookup there, and the master pipeline quietly climbs toward the ceiling. Then someone adds a source system and the deployment fails.
The manual fix is miserable. You split the pipeline by hand, re-wire the dependency chains across the cut, and from then on every new object comes with a question: which shard does this one go in? The split is a design decision frozen into JSON, and it goes stale the moment the batch changes shape again. Worse, the second split is harder than the first, because now the dependencies span three pipelines instead of two.
The limit isn't going away. What can go away is the manual bookkeeping.
A budget, not a landmine
BimlFlex exposes the ceiling as a project setting: AzureDataFactoryActivityLimit, in the Azure Deployment settings category. The default is 40, and the accepted range is 40 to 80. Set it to 200 and the generator clamps it back to 80, because a pipeline that ADF rejects is not a pipeline.
Why default to half of what the platform allows? Because activities aren't all the same size, and the generator holds some back. Out of the configured limit, two activities are reserved in every generated sub-pipeline for overhead: the handoff link to the next shard and the failure-logging step. A limit of 40 means 38 activities of real work per pipeline. The conservative default leaves room for batches to grow without every regeneration living at the edge of the cap.
The setting is project-scoped, so a copy-heavy ingestion project can run at 80 while a project with more elaborate per-object logic stays at 40, and neither decision leaks into the other.
Every object has a price
Packing a pipeline only works if you know what each object costs, and the costs differ by shape:
Standard load: 1
REST source: 2
Sequential file load: 3
A standard load is one Execute Pipeline activity pointing at that object's worker pipeline. A REST source charges 2, because the generated pipeline wraps the call with an extra logging step: a REST copy that returns no rows errors out in ADF, so the generation accounts for handling it. A sequential file load charges 3: a Get Metadata activity to enumerate the files, a ForEach to iterate them in order, and the load itself inside the loop.
Custom logic is charged too. BimlFlex lets you inject your own activities into the batch through the OverrideAdfExecutePipeline extension point, and injected work doesn't ride free: the generator parses the fragment you supplied, counts its activities, including anything nested inside an Activities block, and debits them from the same budget. If the next object's price exceeds what's left, it isn't squeezed in. It opens the next shard.
Overflow becomes a chain
When the budget runs out mid-batch, the generator closes the current pipeline and starts another. The shards are named by batch, solve-order tier, and sequence: a batch pipeline called DailyLoad_Batch spills tier 2 into DailyLoad_Batch_2_0, DailyLoad_Batch_2_1, and so on. Each shard ends with an Execute Pipeline activity that calls the next one and waits for it:
<ExecutePipeline Name="DailyLoad_Batch_2_1"
PipelineName="DailyLoad_Batch_2_1"
WaitOnCompletion="true">
<Dependencies>
<Dependency Condition="Succeeded"
DependsOnActivityName="STG_AW_SalesOrderHeader" />
</Dependencies>
</ExecutePipeline>
Parameters are forwarded down the chain, dependency conditions follow the batch's precedence constraint (Succeeded, Completed, or None, exactly as configured on the batch), and objects are packed in thread order so the parallelism you modeled survives the split. The shards land in a SubBatch folder in the Factory Resources tree, under the batch they belong to, so the top-level view stays readable: one batch pipeline, with its overflow tucked underneath. This is the same batch machinery covered in the dev diary on orchestrating Mapping Data Flows, applied to the copy and load workload.
The practical effect: a 300-object batch is not a design problem. It's five or eight generated pipelines whose boundaries you never chose, never documented, and never have to maintain. Add forty objects and regenerate; the boundaries move themselves.
When the generator refuses
Sharding solves the per-pipeline math, but one thing can't be sharded away: the batch pipeline itself. Dependencies between load stages are expressed as solve-order tiers (staging before vault, vault before marts), and the outer batch pipeline carries a gate per tier. That pipeline has to fit under the cap too, which puts a hard ceiling of 14 solve-order tiers on a single batch.
Cross it and the build stops. Not the deployment, not the 2 a.m. run: the build, with an error that names the batch and states the fix:
> 'DailyLoad_Batch' has more than 14 SolveOrder values defined. Specify at most 14 solve orders to generate an ADF pipeline for this batch that will fall below the maximum ADF activity count of 40.
That message is the whole philosophy in one line. A constraint the platform would have surfaced as a failed deployment, or a failed run, gets surfaced as a build error with the batch name and the number to change. You restructure the batch or split it, on your own schedule, before anything ships.
Limits as metadata
Because the ceiling is a setting, retuning the topology is a regeneration, not a refactor. A batch of simple one-activity loads at the default limit packs 38 objects per shard; raise the limit to 80 and the same batch regenerates into half as many pipelines. Nothing else changes: the build and deploy workflow is identical whether the batch generates one pipeline or twelve.
That's the difference between orchestration you wrote and orchestration you generated. Hand-built ADF treats the activity limit as an incident. Generated ADF treats it as an input, the same way cost-aware orchestration treats spend: a constraint the pipeline design solves for before it exists, rather than one it collides with in production.
The 80-activity cap will still be there next quarter. The question is whether your project knows about it, or whether you find out from a failed publish.
