StoredProcs

This article explores the functionality and features offered by CONNECT BY clause in Oracle with a hands-on exercise approach. Prerequisite: Oracle 9g or lastest installed, any oracle SQL client. We have used Oracle's sample schema for this article, you can download it too from the link below. Execute this SQL in your oracle client and you should be all set with data and schema. https://download.oracle.com/oll/tutorials/DBXETutorial/html/module2/les02_load_data_sql.htm Let's get started with CONNECT BY clause in Oracle. This is basically an oracle clause to place eligible datasets in a hierarchical fashion. Meaning, usage of this function is generally for creating a new resultant query that will elaborate hierarchical relations in a table. Here is the basic syntax [ START WITH condition ] CONNECT BY [ NOCYCLE ] condition START WITH is an optional keyword that can be used as a starting point for hierarchy. CONNECT BY describes the relationship between a child and parent r...

This article will walk you through the basic steps of accessing and reading XML files placed at the filestore using python code in the community edition databricks notebook. We will also explore a few important functions available in the Spark XML maven library. Think of this article as a stepping stone in the databricks community edition. Features and functionalities elaborated herein can be scaled at the enterprise level using Enterprise editions of databricks to design reusable file processing frameworks. Requirements We will be using the Spark-XML package from Maven. **Spark 3.0 or above is required on your cluster for working with XML files. This article uses Databricks Community edition, refer to  this  video tutorial from Pragmatic works for getting started with  Databricks Community Edition . Create your first cluster in seconds : The next step is to install the Spark-XML library on your cluster. The cluster needs to be in a running state to install this li...

Start by creating your account at  Azure DevOps When you signup and log in for the first time you will get a default organization created and from there on you can customize that default organization or create new organizations per your project needs. An  Organization  can basically be considered a unit to group similar applications/projects in an enterprise. Sales and HR for instance can have two separate Azure DevOps organizations in an Enterprise.  Each organization has its share of free services including Pipelines, repos, and many more. Planing your organization structure is critically important in DevOps and a recommendation by Microsoft. Photo by  Lukas  from  Pexels

Step by Step guide  You can either configure source control at the time of ADF deployment or can keep it for later. If you prefer to set up later Check Configure Git later in the Git Configuration tab and proceed. In this blog, we will set it up later. Now let us set up source control in already deployed ADF resource. Navigate to your Data factory UI, click on the drop-down next to the Data Factory icon, and set up a code repository . Let us consider only Azure DevOps Git in this blog. Nest step is to choose the mode of linking your Azure integrated Git with ADF. You can either choose. Select repository or Use repository link Select repository option allows you to provide repository names and Use repository link enables you to use URL link in Project to connect repository for ADF source control.  Picture You need to set up the Azure DevOps project in Azure DevOps Organizations . Without this, you will not get any options in the dropdown shown in the Picture. ! Check...

  Integration runtime joins Activity and Linked services. It provides a compute environment to the Activity to process enlisted actions.   Azure IR Self-Hosted Azure-SSIS Running Dataflows, data movement inside Azure Data movement from externally hosted systems To execute SSIS packages   Azure data factory can be hosted in any azure region of Customer’s choice and IR location can be independent of ADF’s azure regions. Generally, IRs are hosted in the Azure region where data movement, activity dispatching etc. is required. IR behavior with AutoResolve Region in public network Time To Live TTL in Integration Runtime Auto resolve, Adhoc integration runtime clusters add a cluster acquisition time (approximately 4-5 mins) every time it spins up a new cluster for being used in a data flow. Thus it adds an additional compute setup time in total job time for dataflow execution, this behavior...

  Linked Service  provides Azure data factory with basic connection information required to connect external source. There are multiple ways to create a Linked Service in Azure - Via Manage in Azure Data Factory UI, Power Shell, Azure Portal. Data Store Linked Service can be used to configure connection setup for data stores such as relational databases, Azure blob, on-prem FTP servers, HDFS, and many more. Compute environments supported in ADF can be configured using Compute Linked Service.  Azure Databricks, Azure Batch, HD Insights, Azure ML, and Azure Data lake Analytics are the platforms supported as of today. Parameterizing Linked Service The ability to parameterize a linked service makes it an extremely powerful utility. In a DW/BI, It is a fairly common scenario to have multiple source data systems using a sing RDBMS system, Oracle for instance. Using Parameters, we can configure a single linked service to connect to multiple homogeneous data systems. Para...

     Azure Data Factory aka ADF is Azure's Data offering that caters development and orchestration of Data pipelines.  ADF empowers cloud developers to orchestrate their Databrick notebooks and various other codebases. This cloud-managed service is specially designed for complex hybrid ELT, ETL, and Data Integration solutions. ETL Tool - Azure Data Factory ADF is one among many data offerings by Azure and is designed to orchestrate data pipelines. Capabilities like Data flows make it a powerful ETL tool with an ever-growing list of data source integrations. How Azure Data Factory is licensed? Azure Data Factory is Azure's Platform as service (PaaS) solution. Azure Data Factory Components . It has a number of components wrapped in 'Author' and 'Management' options in the left pane. Author components( GA till date) include Pipelines, Dataflows, Datasets, Power Queries. Management Components are Integration Runtimes, Linked Services, Triggers, Global Paramet...