Azure Data Engineer

We are seeking a highly skilled Azure Data Engineer with extensive experience in building, managing, and optimizing data engineering solutions on the Azure cloud platform. The ideal candidate will have a strong background in Azure Databricks and related technologies, and will be instrumental in driving the development and implementation of scalable, secure, and high-performance data pipelines and platforms.

Key Responsibilities

• Design, develop, and maintain robust data pipelines using Azure Data Factory (ADF) and Azure Databricks to support enterprise data lake architectures.

• Manage and configure Databricks asset bundles and deploy reusable data engineering components across environments.

• Implement best practices for Unity Catalog setup including configuration, lineage tracking, access control, and security to ensure data governance and compliance.

• Develop advanced data transformations and analytics workflows using PySpark, Spark SQL, and Delta Lake within Databricks.

• Automate job orchestration and management using Databricks REST APIs and Databricks Workflows.

• Utilize Python and PowerShell scripts to support automation, monitoring, and operational tasks.

• Maintain code quality and deployment workflows with Git-based version control and CI/CD pipelines leveraging tools such as GitHub Actions or Azure DevOps.

• Provision and manage cloud infrastructure using Infrastructure-as-Code (IaC) tools, preferably Terraform, to ensure scalable and repeatable deployment of data platform resources.

• Collaborate closely with security and networking teams to implement Azure security best practices, including Managed Identities, Private Endpoints, Virtual Networks (VNETs), Role-Based Access Control (RBAC), and Network Security Groups (NSGs).

• Work in enterprise-grade environments adhering to data governance frameworks and industry best practices.

• Troubleshoot, optimize, and document data engineering processes for operational efficiency and knowledge sharing.

Required Skills & Experience

• 7+ years of professional experience in data engineering, with 5+ years focused on Azure Data Engineering, specifically using Azure Databricks.

• Deep expertise in Unity Catalog for data governance, security, and lineage.

• Strong proficiency in PySpark, Spark SQL, Delta Lake, and Databricks native features.

• Experience with Databricks REST APIs, workspace management, and job orchestration.

• Solid skills in Python programming and PowerShell scripting.

• Hands-on experience with Azure Data Factory (ADF) for orchestration and data integration.

• Familiarity with Git version control systems and integration of CI/CD pipelines (GitHub Actions, Azure DevOps, or similar).

• Practical knowledge of Terraform or other IaC tools for provisioning and managing Azure resources.

• Comprehensive understanding of Azure security, networking, and identity management including Managed Identities, Private Endpoints, VNETs, RBAC, and NSGs.

• Experience working with large-scale enterprise data lake environments and governance frameworks.

Preferred Qualifications

• Azure certifications related to data engineering (e.g., Microsoft Certified: Azure Data Engineer Associate).

• Experience with other Azure services such as Azure Synapse Analytics, Azure SQL Database, or Azure Event Hubs.

• Knowledge of data modeling and metadata management.

We’re an equal opportunity employer committed to increasing diversity and inclusion in today’s workforce. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. Minorities, women, LGBTQ candidates, and individuals with disabilities are encouraged to apply. If you require an accommodation, please review our accessibility policy and reach out to our accessibility officer with any questions.