StoredProcs

Complete Analytics Platform  In its new Software as a service offering, Microsoft basically clubbed every tool in their Analytics portfolio and gave it a new name - Fabric :). Claims are Fabric can serve every Data stakeholder ranging from a developer working with Data Lake to a Sales associate working on a self-serve Powerbi report. Microsoft has implemented tenant centric architecture in Fabric like office 365, In optimal design an organization will have 1 fabric similar to 1 office 365 tenant for entire organization. Lake centric and Open  All the data and apps built on Fabric provided solutions will get stored at a single lake, It auto calculates the lineage for objects stored on a single data lake. It uses delta file format and parquet data storage for all the objects.  Advantage: Table storage is shared across the fabric workspace, suppose you have a data issue in a Synapse datawarehouse query, just run a fix on the data set using Synapse data engineering python not...

Access Control Lists(ACLs) in azure are an extremely powerful toolset to provision granular levels of access in Azure Data Lake. Role-Based Access Control (RBAC) is best option to setup broader access levels however with ACLs you can reach the lowest possible grains as low as a file inside a blob container. Think of a scenario where you want to add more than 1 user to a folder inside a blob container and each one of them sees only their data - Possible with ACLs Prerequisites Azure Subscription Storage blob with hierarchical namespace enabled Reader Access on the storage object via RBAC How to setup ACLs in Azure Data Lake Like any other offering, Microsoft has a broad spectrum of tools/ways to setup ACLs, ranging from Azure Portal to writing python code . All the steps involved are available in Microsoft documentation, and in a very descriptive manner, therefore needless to rephrase again in this article. Instead, lets walk through some of the challenges one can c...

  Integration runtime joins Activity and Linked services. It provides a compute environment to the Activity to process enlisted actions.   Azure IR Self-Hosted Azure-SSIS Running Dataflows, data movement inside Azure Data movement from externally hosted systems To execute SSIS packages   Azure data factory can be hosted in any azure region of Customer’s choice and IR location can be independent of ADF’s azure regions. Generally, IRs are hosted in the Azure region where data movement, activity dispatching etc. is required. IR behavior with AutoResolve Region in public network Time To Live TTL in Integration Runtime Auto resolve, Adhoc integration runtime clusters add a cluster acquisition time (approximately 4-5 mins) every time it spins up a new cluster for being used in a data flow. Thus it adds an additional compute setup time in total job time for dataflow execution, this behavior...

  Linked Service  provides Azure data factory with basic connection information required to connect external source. There are multiple ways to create a Linked Service in Azure - Via Manage in Azure Data Factory UI, Power Shell, Azure Portal. Data Store Linked Service can be used to configure connection setup for data stores such as relational databases, Azure blob, on-prem FTP servers, HDFS, and many more. Compute environments supported in ADF can be configured using Compute Linked Service.  Azure Databricks, Azure Batch, HD Insights, Azure ML, and Azure Data lake Analytics are the platforms supported as of today. Parameterizing Linked Service The ability to parameterize a linked service makes it an extremely powerful utility. In a DW/BI, It is a fairly common scenario to have multiple source data systems using a sing RDBMS system, Oracle for instance. Using Parameters, we can configure a single linked service to connect to multiple homogeneous data systems. Para...

     Azure Data Factory aka ADF is Azure's Data offering that caters development and orchestration of Data pipelines.  ADF empowers cloud developers to orchestrate their Databrick notebooks and various other codebases. This cloud-managed service is specially designed for complex hybrid ELT, ETL, and Data Integration solutions. ETL Tool - Azure Data Factory ADF is one among many data offerings by Azure and is designed to orchestrate data pipelines. Capabilities like Data flows make it a powerful ETL tool with an ever-growing list of data source integrations. How Azure Data Factory is licensed? Azure Data Factory is Azure's Platform as service (PaaS) solution. Azure Data Factory Components . It has a number of components wrapped in 'Author' and 'Management' options in the left pane. Author components( GA till date) include Pipelines, Dataflows, Datasets, Power Queries. Management Components are Integration Runtimes, Linked Services, Triggers, Global Paramet...