Varigence Blog

The Biml Report: Azure Edition

Written by Peter Avenant on 11.23.2017

Welcome to the November issue of "The Biml Report", a newsletter for Biml Users like you. I'm Peter Avenant (@peteravenant), a senior geek at Varigence focused on technology and customer engagement.

Featured

New Biml for Azure Data Factory unveiled at PASS Summit 2017


Wednesday was the first full day of the conference and a huge day for Biml. Things kicked off with Rohan Kumar’s keynote “SQL Server 2017 and Azure Data Services – The Ultimate Hybrid Data Platform (http://www.pass.org/summit/2017/Sessions/Keynotes.aspx) ”, during which I was honored to demonstrate the upcoming Biml support for Azure Data Factory. Read more... (https://www.varigence.com/Blog/Post/75) You can watch the keynote here (https://youtu.be/NGbk9XGWTHI?t=3580) .

The Biml Book: Business Intelligence and Data Warehouse Automation is now available

The book guides you through the basics of getting your environment configured, the Biml syntax and scripting essentials for building a data warehouse automation framework. Andy Leonard (@AndyLeonard) has done an excellent of bringing a talented group of Bimlers together to co-author the book. You can order the book from Amazon (https://www.amazon.com/Biml-Book-Intelligence-Warehouse-Automation/dp/1484231341) and Apress (http://www.apress.com/gp/book/9781484231340)

Customer review on Amazon. "If this is your first exposure to a Biml framework these chapters will blow your mind. This is game changing stuff."

Webinars

Azure Data Factory v2 Automation with BimlStudio

The upcoming version of BimlStudio will include full support for Azure Data Factory v2, including new first-class Biml elements to create Gateways, Linked Services, Datasets, Pipelines, and Triggers. In this webinar, we will give you a comprehensive tour of the new functionality in the context of an end-to-end solution. Register for the webinar. (https://www.eventbrite.com.au/e/azure-data-factory-v2-automation-with-bimlstudio-tickets-40083678361)

Using Azure Data Lake for Persistent Staging

Persistent Staging (PSA) is a record, or archive, of all data deltas that can reload a Data Warehouse. Traditional approaches to creating a PSA requires a BIG database that stores all unique records loaded. In this webinar, we will show you how BimlFlex with Azure Data Lake allows for data reloads and management with minimal configuration. Register for the webinar. (https://www.eventbrite.com.au/e/using-azure-data-lake-for-persistent-staging-tickets-40084814760)

Recordings

BimlFlex Data Mart Automation using Polybase and Azure SQL Data Warehouse

Investing in designing and implementing a substantial Data Warehouse creates a foundation for a successful Data Platform architecture. Using a configurable Data Warehouse Automation solution that supports all the best bits of Azure SQL Data Warehouse as standard is essential. Watch the webinar. (https://www.varigence.com/Blog/Post/73)

Azure Data Warehouse Automation with BimlFlex

In this day and age, data warehouses simply have to be future proofed for cloud technologies. Populating this modern data warehouse should benefit from the latest innovation in ETL and data integration with near self-service data preparation. You need a data warehouse automation (DWA) solution built from the ground up specifically to empower Microsoft Data Platform users with the ability to switch between on-premises and Azure deployments. Watch the webinar. (https://www.varigence.com/Blog/Post/69)

What's new in BimlExpress 2017? by Cathrine Wilhelmsen

BimlExpress 2017 is finally here! In this session, Cathrine will take you through the exciting new features, like expanded Biml preview, global include files, and support for SSIS annotations.

She also covers some of the most important changes to the Biml API, like the updated CallBimlScript, the new MakeSsisSafe extension method for strings and the flat file extension method for generating table columns from flat file columns. Watch the webinar. (https://www.varigence.com/Blog/Post/70)

Training

We recently presented the Advance Biml Training training in Amsterdam and Copenhagen for Biml developers looking to take their skills to the next level. During this hands on training we will build a real-world data warehouse automation framework with all the source code to accelerate projects. We have requested the codes from Apress and will send attendees a complimentary copy of The Biml Book just in time for some holiday reading. We are planning training in collaboration with Roelant Vos. The plan is two days of Biml training and then three days Data Vault Implementation training. Although the training is complimentary, we will segment it into three. Varigence will present Day 1 Biml Fundamentals, Day 2 Biml Frameworks and Roelant will present Day 3-4 Data Vault Implementation. We are planning two sessions in April, one in Amsterdam and the other in the UK.

Feedback

Any feedback on content send me a mail at media@varigence.com. For the latest community content, please head on over to http://bimlscript.com/ (http://bimlscript.com/ )

Biml for Azure Data Factory unveiled at PASS Summit 2017

Written by Scott Currie on 11.19.2017

PASS Summit 2017 was an awesome conference for Varigence and Biml, both with fun traditions and some exciting firsts.

Wednesday was the first full day of the conference and a huge day for Biml. Things kicked off with Rohan Kumar’s keynote SQL Server 2017 and Azure Data Services – The Ultimate Hybrid Data Platform , during which I was honored to demonstrate the upcoming Biml support for Azure Data Factory. In just a few minutes, I showed a how a couple hundred lines of Biml code could be used to automatically read an on-premises source schema, generate an Azure Data Factory that loads the source data into an Azure Data Lake, creates SSIS packages that implement logic to scrub address fields (which would then run in the cloud using the Azure SSIS Integration Runtime), and generates PowerShell scripts to deploy it all. This can all be executed from a DevOps automation workflow to rebuild your scrubbed data lake nightly, hourly, or even on a near-realtime basis.

Wednesday afternoon, I was pleased to present BimlCatalog: Your Open Source Runtime and Orchestration Solution for Biml Projects. At Varigence, we have been working on our data warehouse automation product BimlFlex for a few years. An essential part of any automation framework, in addition to design patterns and metadata entry/management tools, is a runtime framework to manage logging, data auditing, restartability, rollback, and more. We have decided to make this runtime framework freely available and open source on GitHub as the BimlCatalog project. In my session, I gave everyone a tour of BimlCatalog capabilities, showed them how to use it in their own projects and demonstrated a few places where they might be able to extend or customize the framework.

Biml over Beers on Thursday night was a great time to reconnect with friends from the Biml community and to provide new Biml users with the opportunity to discuss usage with folks who have already implemented Biml successfully. We had about 50 Biml users over the course of the evening, and I think a good time was had by all. On Friday,

I was able to spend some time with the Azure Data Factory team to ensure that our support for ADF would be everything you’ve come to expect from Biml. While I was there, Andy Leonard gave a great session Use Biml to Automate SSIS Design Patterns which was targeted at those who are relatively new to Biml and would like to build their own reusable patterns and practices for their data automation projects.

Throughout the week, I was able to meet onsite with a few of our Seattle-based Biml users. Spending time with customers is always a great experience, and these visits didn’t disappoint. Thanks to all of you!

PASS Summit 2017 was a great conference, and I’m very much looking forward to next year.