Linked Service in Azure Data Factory

Linked Service provides Azure data factory with basic connection information required to connect external source. There are multiple ways to create a Linked Service in Azure - Via Manage in Azure Data Factory UI, Power Shell, Azure Portal.

Data Store Linked Service can be used to configure connection setup for data stores such as relational databases, Azure blob, on-prem FTP servers, HDFS, and many more.

Compute environments supported in ADF can be configured using Compute Linked Service. Azure Databricks, Azure Batch, HD Insights, Azure ML, and Azure Data lake Analytics are the platforms supported as of today.

Parameterizing Linked Service

The ability to parameterize a linked service makes it an extremely powerful utility. In a DW/BI, It is a fairly common scenario to have multiple source data systems using a sing RDBMS system, Oracle for instance. Using Parameters, we can configure a single linked service to connect to multiple homogeneous data systems. Parameter values can be set at datasets using this parameterized Linked Service. There is also a provision to set a default value for a parameter. Below snippets show the parameter section available while creating a new linked service.

Comments

How to work with XML files in Databricks using Python

This article will walk you through the basic steps of accessing and reading XML files placed at the filestore using python code in the community edition databricks notebook. We will also explore a few important functions available in the Spark XML maven library. Think of this article as a stepping stone in the databricks community edition. Features and functionalities elaborated herein can be scaled at the enterprise level using Enterprise editions of databricks to design reusable file processing frameworks. Requirements We will be using the Spark-XML package from Maven. **Spark 3.0 or above is required on your cluster for working with XML files. This article uses Databricks Community edition, refer to this video tutorial from Pragmatic works for getting started with Databricks Community Edition . Create your first cluster in seconds : The next step is to install the Spark-XML library on your cluster. The cluster needs to be in a running state to install this li...

Microsoft Fabric

Complete Analytics Platform In its new Software as a service offering, Microsoft basically clubbed every tool in their Analytics portfolio and gave it a new name - Fabric :). Claims are Fabric can serve every Data stakeholder ranging from a developer working with Data Lake to a Sales associate working on a self-serve Powerbi report. Microsoft has implemented tenant centric architecture in Fabric like office 365, In optimal design an organization will have 1 fabric similar to 1 office 365 tenant for entire organization. Lake centric and Open All the data and apps built on Fabric provided solutions will get stored at a single lake, It auto calculates the lineage for objects stored on a single data lake. It uses delta file format and parquet data storage for all the objects. Advantage: Table storage is shared across the fabric workspace, suppose you have a data issue in a Synapse datawarehouse query, just run a fix on the data set using Synapse data engineering python not...

Hierarchies in Oracle.

This article explores the functionality and features offered by CONNECT BY clause in Oracle with a hands-on exercise approach. Prerequisite: Oracle 9g or lastest installed, any oracle SQL client. We have used Oracle's sample schema for this article, you can download it too from the link below. Execute this SQL in your oracle client and you should be all set with data and schema. https://download.oracle.com/oll/tutorials/DBXETutorial/html/module2/les02_load_data_sql.htm Let's get started with CONNECT BY clause in Oracle. This is basically an oracle clause to place eligible datasets in a hierarchical fashion. Meaning, usage of this function is generally for creating a new resultant query that will elaborate hierarchical relations in a table. Here is the basic syntax [ START WITH condition ] CONNECT BY [ NOCYCLE ] condition START WITH is an optional keyword that can be used as a starting point for hierarchy. CONNECT BY describes the relationship between a child and parent r...

StoredProcs

Search This Blog