The Varigence Blog

Tag - BimlFlex,

Combining DBT with BimlFlex

Written by Roelant Vos on 8.21.2022

During our travels, we are regularly asked how BimlFlex compares against DBT. Our BimlFlex data solution automation framework would be compared against all our competitors, of course, but DBT is the odd one out in this case. It is worth covering this in detail. This is because BimlFlex and DBT are not really competing. In fact, they can complement each other.

Dev Diary - Initial Extension Points added to Mapping Data Flows

Written by Roelant Vos on 5.26.2022

As part of ongoing improvements for our Mapping Data Flows feature, we are adding the first (of many) extension points.

Dev Diary - Customizing file paths for Delta Lake

Written by Roelant Vos on 2.14.2022

It is now possible to configure specific directory structures for your Delta Lake using BimlFlex.

Dev Diary - Configuring load windows and filters for Mapping Data Flows

Written by Roelant Vos on 2.10.2022

An overview of how to configure BimlFlex Parameters to manage load windows and apply any other filters to the data selections.

Dev Diary - Connecting to unsupported data sources using Mapping Data Flows and BimlFlex

Written by Roelant Vos on 2.4.2022

Loading data from unsupported data sources in Mapping Data Flows

Dev Diary - Adding Transformations to Mapping Data Flows

Written by Roelant Vos on 2.1.2022

BimlFlex will soon support custom transformations to be generated for Mapping Data Flows.

Dev Diary – Generating a Mapping Data Flow Staging process without a Persistent Staging Area

Written by Roelant Vos on 11.25.2021

A real-world data solution often requires a combination of many different data loading patterns. Using BimlFlex, these patterns are applied by configuring the design to work in certain ways. This post shows how to configure BimlFlex to achieve different Staging pattern outcomes.

Creating custom column-level transformations in BimlFlex

Written by Roelant Vos on 10.13.2021

In the real world, many data solutions require a high degree of flexibility so that they can cater to unique scenarios. This can be necessary because of specific systems that require integration, or simply because certain data needs bespoke logic to be interpreted correctly. BimlFlex offers ways to specify specific transformations at column level - using Custom Configurations.

Dev Diary - First look at a source-to-staging pattern in Mapping Data Flows

Written by Roelant Vos on 9.19.2021

With the 2021 BimlFlex release nearing completion, it's time to take a closer look at the patterns for Mapping Data Flows that will be made available in preview.

Dev Diary - Integration with the BimlCatalog

Written by Roelant Vos on 9.15.2021

Every data solution benefits from a robust control framework for data logistics. One that manages if, how and when individual data logistics processes should be executed. A control framework also provides essential information to complete the audit trail of how data is processed through the system and is ultimately made available to users. BimlFlex provides the BimlCatalog for this.

Dev Diary – Orchestrating Mapping Data Flows

Written by Roelant Vos on 9.7.2021

In Azure, a Mapping Data Flow itself is not an object that can be executed directly. Instead, it needs to be called from an Execute Pipeline. This pipeline can be run, and in turn it will start the data flow. BimlFlex has advanced features to manage this.

Dev Diary – Defining Mapping Data Flow Parameters with Biml

Written by Roelant Vos on 8.31.2021

BimlFlex output uses Parameters at Mapping Data Flow level to integrate with the BimlCatalog and store metadata for use inside the data flow. This post explains how to set this up in Biml.

Dev Diary - Deploying Biml-Generated ADF Data Flow Mappings

Written by Roelant Vos on 8.18.2021

BimlStudio can translate Biml into Data Flow Mappings, and this post looks into the deployment and results in Azure Data Factory.

Dev Diary - Generating ADF Data Flow Mapping using Biml

Written by Roelant Vos on 8.18.2021

In this first dev diary post we show the basic Biml syntax to create Data Flow Mappings for Azure Data Factory

Delta Lake on Azure work in progress – introduction

Written by Roelant Vos on 7.29.2021

In this development blog series, we explain how inline data sources and Delta Lake in particular will be supported in BimlFlex

Snowflake Integration Powered by BimlFlex

Written by Peter Avenant on 3.13.2019

Using BimlFlex to create an automated and integrated data warehouse

Pages: 123