Skip to main content

Ensono Stacks Azure Data Platform

The Ensono Stacks Azure Data Platform solution provides a framework for accelerating the deployment of a production-ready modern data platform in Azure.

Ensono Stacks Data Overview

  1. Use the Ensono Stacks CLI to generate a new data platform project.
  2. Build and deploy the data platform infrastructure into your Azure environment.
  3. Accelerate development of data workloads and ELT pipelines with the Datastacks CLI.

The Ensono Stacks Data Platform delivers a modern Lakehouse solution, based upon the medallion architecture, with Bronze, Silver and Gold layers for various stages of data preparation. The platform utilises tools including Azure Data Factory for data ingestion and orchestration, Databricks for data processing and Azure Data Lake Storage Gen2 for data lake storage. It provides a foundation for data analytics and reporting through Microsoft Fabric and Power BI.

Key elements of the solution include:

  • Infrastructure as code (IaC) for all infrastructure components (Terraform).
  • Deployment pipelines to enable CI/CD and DataOps for the platform and all data workloads.
  • Sample data ingest pipelines that transfer data from a source into the landing (Bronze) data lake zone.
  • Sample data processing pipelines performing data transformations from Bronze to Silver and Silver to Gold layers.

The solution utilises the Stacks Data Python library, which offers a suite of utilities to support:

High-level architecture

High-level architecture

Infrastructure deployed

  • Resource Group
  • Key Vault
  • Azure Data Lake Storage Gen2
  • Azure Blob Storage
  • Azure Data Factory
  • Log Analytics Workspace
  • Databricks Workspace
  • Azure SQL Database (optional, for testing the platform)

The deployed platform can integrate with Microsoft Fabric to provide a suite of analytics tools and capabilities.

The solution is designed to be deployed within a secure private network - for details see infrastructure and networking.

Data Engineering workloads

Example data engineering workloads are provided, based upon the Datastacks templates:

Each of the ingest and data processing workloads may optionally include data quality checks.