Skip to main content

Learn Azure Data Factory.


Azure Data Factory aka ADF is Azure's Data offering that caters development and orchestration of Data pipelines. ADF empowers cloud developers to orchestrate their Databrick notebooks and various other codebases. This cloud-managed service is specially designed for complex hybrid ELT, ETL, and Data Integration solutions.

ETL Tool - Azure Data Factory
ADF is one among many data offerings by Azure and is designed to orchestrate data pipelines. Capabilities like Data flows make it a powerful ETL tool with an ever-growing list of data source integrations.

How Azure Data Factory is licensed?
Azure Data Factory is Azure's Platform as service (PaaS) solution.

Azure Data Factory Components.
It has a number of components wrapped in 'Author' and 'Management' options in the left pane.
Author components( GA till date) include Pipelines, Dataflows, Datasets, Power Queries.
Management Components are Integration Runtimes, Linked Services, Triggers, Global Parameters, etc.

What is a Pipeline in Azure Data Factory?
Logical grouping of activities to perform a task. The task can be data extraction, data transformation or loading etcetera!

What is an Activity in Azure Data Factory?
Activity defines the action to be performed, for instance, Data copy in the Copy activity. Based on their actions, ADF activities can be split into three categories.
  • Data Movement Activities- Copy Activity
  • Data Transformation Activities- Mapping Dataflows, Stored Procedure Activity
  • Control Activities- Until Activity, GetMetaData activity 


What are the Chaining activities in Azure Data Factory?
dependsOn (Add Activity On UI interface) property in an activity can be used in the latest ADF version to chain activities to one another. Unlike the old days when we had to configure the output of an activity as an input of the upstream activities for managing Control Flow. The snippet below explains the usage and options available.


Add Activity on Option selected for Success & Fail activity will become dependsOn property respective activities and can be viewed in XML code.

"name": "Success",
"type": "Delete",
"dependsOn": [
{
"activity": "BASE",
"dependencyConditions": [
"Succeeded"
]
}
],
Linked services in Azure Data Factory.
These are nothing but connection strings that contain connection-related information details required by ADF to connect to the source instance of a dataset. Datasets require a Linked service.

Integration Runtime in Azure Data Factory
Integration Runtime IR provides compute facility for various actions such as data movement, activity dispatching.

What are Mapping data flows?
Mapping data flow allows cloud engineers to set up visually created data transformation logic. Data flows can be executed inside a pipeline hence all the scheduling capabilities are available in case of data flows.

Data flows can be leveraged to create data pipelines for loading various types of Dimensions and Fact entities in a DW/BI application. Data flows simplify the creation of complex ETL logic which was quite a tedious task (using native ADF activities in an ADF pipeline) in the past when data flows were not available.

                                         

To enable Data Flow, While creating Azure Data Factory you need to choose version 2 with Data flows.
Currently, there are three versions available.
  • Version 1
  • Version 2
  • Version 2 with Data Flows
How can we prevent sensitive data from being displayed in Monitor logs when passing/receiving inputs across ADF Activities?
We can check/uncheck the following options under the General table of activities to securely pass sensitive information across activities.



What are global parameters in Azure Data Factory?
ADF allows you to define a parameter globally and use it across pipelines. It can be done in Global Parameters under Manage tab.



Labels:
Location: Punjab, India

Comments

  1. Anonymous28 September 2026 at 18:52

    Apache Spark is widely used for processing large datasets by distributing computation across multiple machines. PySpark provides a Python interface to Spark, allowing developers to work with distributed data processing through familiar Python programming concepts. PySpark Course

    ReplyDelete
  2. Anonymous28 September 2026 at 18:52

    Data engineering involves building reliable pipelines that collect, transform, and prepare data for analysis and other downstream applications. These workflows commonly require tools that can handle large volumes of information while supporting scalable processing and transformation operations. Data Engineering Course

    ReplyDelete
  3. Anonymous28 September 2026 at 18:52

    Big data processing becomes particularly important when datasets are too large for conventional single-machine processing. Distributed systems address this challenge by dividing data and computation across multiple processing resources, allowing large workloads to be handled more efficiently. Big Data Projects

    ReplyDelete

Post a Comment

Popular posts from this blog

ACLs in Azure Data Lake

Access Control Lists(ACLs) in azure are an extremely powerful toolset to provision granular levels of access in Azure Data Lake. Role-Based Access Control (RBAC) is best option to setup broader access levels however with ACLs you can reach the lowest possible grains as low as a file inside a blob container. Think of a scenario where you want to add more than 1 user to a folder inside a blob container and each one of them sees only their data - Possible with ACLs Prerequisites Azure Subscription Storage blob with hierarchical namespace enabled Reader Access on the storage object via RBAC How to setup ACLs in Azure Data Lake Like any other offering, Microsoft has a broad spectrum of tools/ways to setup ACLs, ranging from Azure Portal to writing python code . All the steps involved are available in Microsoft documentation, and in a very descriptive manner, therefore needless to rephrase again in this article. Instead, lets walk through some of the challenges one can c...
2 comments
Read more

n8n Tutorial: Automate Your Workflows in 4 Simple Steps

Getting Started with n8n: A Practical Guide to Workflow Automation n8n (pronounced "n-eight-n") is a powerful, open-source workflow automation tool that enables you to connect different apps and services without writing code. Think of it as a visual programming environment where you design automated sequences, known as workflows, by linking together pre-built nodes. Unlike some other automation platforms, n8n is fair-code, meaning its source code is available for you to inspect, modify, and self-host, giving you unparalleled control over your data and processes. This guide will walk you through the core concepts and initial steps to harness n8n's potential, turning complex, manual tasks into efficient, automated workflows. What is n8n? Understanding the Core Concepts At its heart, n8n is an integration platform designed to move and transform data between various applications. The fundamental building block of any n8n project is the node . Each node represents a spe...
1 comment
Read more

Integration Runtime in Azure Data Factory

  Integration runtime joins Activity and Linked services. It provides a compute environment to the Activity to process enlisted actions.   Azure IR Self-Hosted Azure-SSIS Running Dataflows, data movement inside Azure Data movement from externally hosted systems To execute SSIS packages   Azure data factory can be hosted in any azure region of Customer’s choice and IR location can be independent of ADF’s azure regions. Generally, IRs are hosted in the Azure region where data movement, activity dispatching etc. is required. IR behavior with AutoResolve Region in public network Time To Live TTL in Integration Runtime Auto resolve, Adhoc integration runtime clusters add a cluster acquisition time (approximately 4-5 mins) every time it spins up a new cluster for being used in a data flow. Thus it adds an additional compute setup time in total job time for dataflow execution, this behavior...
1 comment
Read more