Your Databricks File Load Evolves by Default
mergeSchema Ships On, And Per-Object Settings Decide Where It Stops
October 21, 2026
Open a Databricks notebook that BimlFlex generated for a file source and look at how it reads the files. Unless someone changed a setting on that object, the read carries mergeSchema. That's true for both load shapes, read_files and COPY INTO, and for every file type the landing connection can declare: CSV, JSON, Parquet, Avro, ORC and binary. Nobody on your team typed that clause. It's a schema-evolution policy, and it was chosen for you.
It's a good default for most landing data. It's the wrong default for a feed that comes with a contract. In Part 1 of our data contracts series we covered how to classify and version schema changes. This post is the Databricks implementation detail: what the generated clause does, what it quietly tolerates, and the three object-level settings that replace it for a single object.
Two Load Shapes, One Default
BimlFlex generates one of two shapes for reading landing files, and object-level settings pick which one you get. With the defaults (DatabricksUseCopyInto off, DatabricksUseTemporaryViews on) the notebook builds a temporary view over read_files, then a second view that selects the object's columns. For a Parquet source, with the notebook variables resolved and the string escaping removed for readability, the first view looks like this:
CREATE OR REPLACE TEMPORARY VIEW <source file view> AS
SELECT *, _metadata
FROM read_files('<location_path>/<file_path>'
,format => 'parquet'
,mergeSchema => 'true');
CREATE OR REPLACE TEMPORARY VIEW <source temp view> AS
SELECT <columns from the object's metadata>
FROM <source file view>;
For a delimited file the options also carry what the metadata knows about the file: delimiter, header, and quote, escape, encoding, null value, row delimiter and skip-rows when you've set them.
,format => 'csv'
,mergeSchema => true
,delimiter => ','
,header => true
Turn DatabricksUseCopyInto on for the object, or turn temporary views off (which a project using SQL scripting has to do, since the two settings are mutually exclusive), and you get the COPY INTO shape instead:
DROP TABLE IF EXISTS <staging temp table>;
CREATE TABLE IF NOT EXISTS <staging temp table>;
COPY INTO <staging temp table>
FROM
(
SELECT <columns from the object's metadata>
FROM '<location_path>/<file_path>'
)
FILEFORMAT = CSV
FORMAT_OPTIONS ('mergeSchema' = 'true'
,'delimiter' = ','
,'header' = 'true')
COPY_OPTIONS ('mergeSchema' = 'true');
Same policy, two places. FORMAT_OPTIONS governs how the files are read. COPY_OPTIONS governs the table being written.
What Merging Actually Does
The two mergeSchema flags mean different things, and it's worth keeping them apart.
On the reader side (read_files options and FORMAT_OPTIONS), Databricks describes mergeSchema as whether to infer the schema across multiple files and merge the schema of each file. Its default there is false. So when a run picks up Monday's files, which don't have loyalty_tier, and Tuesday's files, which do, the read sees one schema with loyalty_tier in it, and Monday's rows come through with NULL.
On the table side, the generated COPY INTO writes into a table it created a moment earlier with no columns at all. That's the schemaless placeholder pattern Databricks documents, and COPY_OPTIONS ('mergeSchema' = 'true') is how the placeholder gets its columns from the incoming rows.
Now the part people miss. Look at the SELECT in both shapes. It lists the columns from the object's metadata, not whatever the files happen to contain. A producer that adds a column doesn't widen your staging table. The new column is visible to the read and then goes nowhere, until you import it into metadata and rebuild. "Evolve by default" here means the read is forgiving. It doesn't mean your warehouse grows columns on its own.
For a landing layer that's the right trade. Files from different producer versions pile up in the same folder, you add a column to metadata and rebuild, and last month's narrower files still read. Failing a night's load over that would be worse.
Where Evolve Hides a Break
Run the same logic against a feed that has a contract, say a partner's settlement file where fee_amount is required.
The partner ships a bad release, and half of tonight's files arrive without fee_amount. The merged read unions the file schemas, finds fee_amount in the other half, and fills the missing rows with NULL. The load succeeds. The batch reports green, and the break turns up later as a reconciliation gap somebody has to trace back.
If every file in the run drops the column, the metadata SELECT has nothing to resolve and the load stops. So merging doesn't hide a total break. It hides a partial one, and partial breaks are the ones that cost the most to find.
You might think the fix is mergeSchema = false. It isn't. Off means Databricks stops reconciling the files against each other, so which columns the read sees can depend on which files it looks at. That's less predictable, not more strict. Strict means declaring the shape you expect.
Tightening One Object
Three settings on the source object replace the generated reader and copy clauses. Each one is a text value that goes into the notebook verbatim in place of the generated one:
DatabricksReadFilesOptions: the options inside read_files(...): temporary view (default)
DatabricksCopyFormatOptions: the whole FORMAT_OPTIONS (...) clause: COPY INTO
DatabricksCopyOptions: the whole COPY_OPTIONS (...) clause: COPY INTO
Leave them blank and the object keeps the merged default. Set one and that object alone changes. Every other object in the project stays permissive.
For a contract feed, the read_files shape is the better place to tighten, because it accepts a declared schema. Databricks notes that you can't set a schema manually with COPY INTO. If the object is on COPY INTO, turn DatabricksUseCopyInto off for that object first (in a project without SQL scripting). Then set DatabricksReadFilesOptions on the settlement file's source object:
,format => 'csv'
,delimiter => ','
,header => true
,schema => 'settlement_id BIGINT, partner_id BIGINT, fee_amount DECIMAL(18,2), settled_at TIMESTAMP'
,enforceSchema => false
,mode => 'FAILFAST'
The generated read becomes:
FROM read_files('<location_path>/<file_path>'
,format => 'csv'
,delimiter => ','
,header => true
,schema => 'settlement_id BIGINT, partner_id BIGINT, fee_amount DECIMAL(18,2), settled_at TIMESTAMP'
,enforceSchema => false
,mode => 'FAILFAST');
The override replaces the whole options list. It lands straight after the file path, so start it with a comma and restate format, delimiter and header. It also stops following your metadata: change the delimiter on the object next quarter and this text won't notice.
As for the options themselves, schema fixes names and types instead of inferring them. With a declared schema, enforceSchema => false makes a CSV read check the header against it rather than apply the schema by position, which matters the day a partner reorders columns. mode => 'FAILFAST' stops the read on a record that doesn't parse, where the default PERMISSIVE mode keeps going.
Be clear about what this buys. Reader options pin the shape and fail on files that don't fit it. They don't turn every NULL into a violation, and they aren't a data contract by themselves. BimlFlex doesn't enforce the schema here. It writes your clause into the load exactly where Databricks reads the files. Required-field and value rules still belong in a validation step, which we laid out in Part 2 on validation gates and enforcement patterns. Before you trust the override, run one deliberately broken file through it in dev. Databricks reader behavior, including the rescued data column read_files adds by default, varies by format.
One caution on the COPY INTO shape: leave DatabricksCopyOptions alone unless you have a copy option you really need. The table that clause writes to is a placeholder rebuilt every run, and Databricks documents mergeSchema in COPY_OPTIONS as how that placeholder gets its schema. Stripping it doesn't make the load stricter. If an object has to stay on COPY INTO, tighten DatabricksCopyFormatOptions for the text formats and put the rest of the contract in the validation step.
Evolve or Enforce, Feed by Feed
These settings only exist where files get read, at the landing edge. By the time data reaches staging, persistent staging, the Data Vault and the marts, it's moving between tables through column lists your metadata already defines. So the decision is per feed, not per warehouse:
Internal exports with no contract: Evolve: Nothing, keep the default
Vendor files from several producer versions in one folder: Evolve: Nothing; merging is the job
Contract-bound feed (partner, regulatory, finance): Enforce at the read: read_files shape, DatabricksReadFilesOptions with a declared schema and a strict mode
Contract feed stuck on COPY INTO: Enforce what you can: DatabricksCopyFormatOptions with a strict mode for CSV or JSON, rules in a validation gate
Staging, Data Vault, marts: Metadata column lists: No file options apply; contract rules live in validation gates
Our rule of thumb: start every new source on the default and promote a feed to enforce the day someone outside your team signs a document about its shape. That keeps strictness where a broken file is a business event, and keeps the rest of the landing layer from failing over columns nobody has asked for yet.
If you're setting up file sources on Databricks for the first time, our walkthrough of Databricks pipelines with BimlFlex covers landing zones and the notebook flow, and structured and semi-structured formats in one framework covers how each file type is declared in metadata.
