Automate your Microsoft data warehouse with Azure Data Factory and TimeXtender

In this blog, read more about how you can automate your Microsoft data warehouse with TimeXtender and Azure Data Factory. Discover how TimeXtender and Data Factory work together, what the misconceptions about Data Factory are, and find out in which situations it's best to use it. Finally, we explain what we believe the ideal data platform looks like.

What is Azure Data Factory?

Azure Data Factory is a scalable way to build ‘data pipelines’ in Azure that can be used to move data from on-premise to Azure, or within Azure between different databases or other infrastructure components. Because Azure Data Factory is a serverless service, it can spin up resources as needed to run your data pipelines. Ultimately, this means you only pay for what you use, rather than for having a server at your disposal.

What is TimeXtender?

TimeXtender is a low-code data management platform that lets you develop up to 70% faster and save up to 80% on maintenance, because it automatically generates (ETL) scripts and documentation. With it, you develop and maintain a future-proof data warehouse (in Azure, for example) that any developer can easily work with.

How do I automate my Microsoft data warehouse with TimeXtender?

TimeXtender orchestrates your entire data platform, which also makes it possible for your platform's infrastructure to grow with you. You can start with a small number of sources and limited data, based on a virtual machine for TimeXtender with an Azure SQL DB for your data warehouse. You can later expand this, for example by growing into a Hyperscale database with Azure Data Lake and Azure Data Factory. Ultimately, TimeXtender takes care of setting up all the transports and code in Azure or on-premise, which means you can also monitor your Azure Data Factory pipelines in the usual way, for example. The great thing is that you tell TimeXtender what needs to happen, and TimeXtender then determines how best to do it.

Misconceptions about Data Factory

Of course, you can also implement your data platform without TimeXtender, but then you'll have to configure all the infrastructure yourself and write all the necessary code. Although Azure Data Factory can be used through a visual interface, you'll quickly find that setting up and managing pipelines soon becomes hard to keep track of. In general, this often means you end up being overtaken by the technical possibilities and the demands of the business. It also means that a change in infrastructure can require substantial code changes, making change less attractive. You then usually turn to other tools to automate Azure Data Factory, and before you know it, you've built a multitude of tools and dependencies into your project. On top of that, setting this up by hand and continuing to manage it can require a great deal of detailed knowledge.

When do you use Data Factory with Azure and TimeXtender?

When we implement TimeXtender for customers who use (or want to use) Azure, there are a number of scenarios where Azure Data Factory is a natural fit:

• Many source systems: it's often more efficient to use Azure Data Factory than to scale up your virtual machines and databases to enable many parallel load streams.
• Lots of data: as your data volume grows, you reach a point where Azure Data Lake is a better place to land your data. Azure Data Factory is then the most efficient way to bring data from source to lake.
• Data in an Azure Data Lake: when you land data in a lake, you'll also want to move data to databases for transformation and presentation. This transport is most efficient via Azure Data Factory.

Naturally, there are also scenarios where Azure Data Factory is not necessary, or less so. A deployment where Azure Synapse Analytics forms the core of your data platform, for example. Synapse can bring data from an Azure Data Lake into Synapse more efficiently via PolyBase, so in that case Azure Data Factory is only needed to transport data from source to lake.

The ideal data platform

By using TimeXtender to automate a data platform in Azure, you have a flexible infrastructure that can grow with the needs of the business. Experiments with advanced analytics and machine learning can be supported, and the associated infrastructure can be switched off again once such a project is over. By then using Power BI as the BI front end, you have a platform that is familiar to many users and fully integrated. In our view, this is the ideal combination for a successful data platform. Want to read more about this combination? Check out this blog about E-mergo's golden triangle.

Want to know more about TimeXtender? See the platform in action during one of our monthly live demos via Microsoft Teams, or take a look at one of our other resources.

Live Demo Resources