How to setup Source control in Azure Data Factory

Step by Step guide

You can either configure source control at the time of ADF deployment or can keep it for later.

If you prefer to set up later Check Configure Git later in the Git Configuration tab and proceed. In this blog, we will set it up later.

Now let us set up source control in already deployed ADF resource. Navigate to your Data factory UI, click on the drop-down next to the Data Factory icon, and set up a code repository.

Let us consider only Azure DevOps Git in this blog.

Nest step is to choose the mode of linking your Azure integrated Git with ADF. You can either choose.

Select repository or Use repository link

Select repository option allows you to provide repository names and Use repository link enables you to use URL link in Project to connect repository for ADF source control.

Picture

You need to set up the Azure DevOps project in Azure DevOps Organizations. Without this, you will not get any options in the dropdown shown in the Picture.! Check this to understand AzureDevops organization

Tip**** Make sure you sign out and sign in again, after clearing your browser cache in case you are not getting the Project name under the Azure Dev Ops organization you selected. This generally happens if you are configuring the DevOps project in parallel with ADF git setup and the browser cache is stale.

Common Errors

Verify that your Azure DevOps account is connected to the AAD account, the Azure DevOps repository is in the Default Directory (bla..bla..bla) tenant, and that your current ADF user account has been added to the Azure DevOps organization.

You can fix it by connecting the Azure Active directory used in your ADF resource to Azure DevOps Organizations. Follow the below steps.

Login to Azure DevOps Organization → Organization Settings → Overview→ Azure Active Directory → Connect directory

If you have not received any errors then you are through and have successfully configured Azure DevOps Git in your ADF. Below is a picture how your current setup should look like in an ideal case.

The Collaboration branch is the one used to push code to Other Data factory instances - DEV to QA for instance.

You can follow general best practices such as 'always create a feature branch for new enhancements' and 'never directly push to masters' or follow any policies designed by your organization.

Save in data factory will act as commits in Azure Git Repo's working branch.

The picture above shows how you can toggle between different branches.

adf_publish branch has the ADF ARM template, which is used for Code Migrations.

*** You can also disconnect Git Repository from Source Contol in Management Hub.

Comments

ACLs in Azure Data Lake

Access Control Lists(ACLs) in azure are an extremely powerful toolset to provision granular levels of access in Azure Data Lake. Role-Based Access Control (RBAC) is best option to setup broader access levels however with ACLs you can reach the lowest possible grains as low as a file inside a blob container. Think of a scenario where you want to add more than 1 user to a folder inside a blob container and each one of them sees only their data - Possible with ACLs Prerequisites Azure Subscription Storage blob with hierarchical namespace enabled Reader Access on the storage object via RBAC How to setup ACLs in Azure Data Lake Like any other offering, Microsoft has a broad spectrum of tools/ways to setup ACLs, ranging from Azure Portal to writing python code . All the steps involved are available in Microsoft documentation, and in a very descriptive manner, therefore needless to rephrase again in this article. Instead, lets walk through some of the challenges one can c...

n8n Tutorial: Automate Your Workflows in 4 Simple Steps

Getting Started with n8n: A Practical Guide to Workflow Automation n8n (pronounced "n-eight-n") is a powerful, open-source workflow automation tool that enables you to connect different apps and services without writing code. Think of it as a visual programming environment where you design automated sequences, known as workflows, by linking together pre-built nodes. Unlike some other automation platforms, n8n is fair-code, meaning its source code is available for you to inspect, modify, and self-host, giving you unparalleled control over your data and processes. This guide will walk you through the core concepts and initial steps to harness n8n's potential, turning complex, manual tasks into efficient, automated workflows. What is n8n? Understanding the Core Concepts At its heart, n8n is an integration platform designed to move and transform data between various applications. The fundamental building block of any n8n project is the node . Each node represents a spe...

Build smart data pipelines with AI

AI in Data Engineering Smarter Pipelines, Faster Insights Artificial Intelligence (AI) is reshaping the way businesses collect, process, and analyze data. In the world of data engineering , AI is not just an enhancement but a game-changer. By embedding intelligence into data pipelines, organizations can automate repetitive tasks, predict failures, improve data quality, and unlock actionable insights faster than ever before. Whether it's scaling data workflows in the cloud, detecting anomalies in real time, or optimizing query performance, AI-driven engineering provides efficiency, accuracy, and agility that traditional methods struggle to match. But how exactly is AI influencing this evolving discipline, and what does it mean for data engineers? Let’s dive into the distinct areas where AI is making its mark. Automating Data Pipelines Data pipelines are the backbone of analytics and machine learning projects, yet building and maintaining them can be labor-intensive. AI in...

StoredProcs

Search This Blog