Senior Data Engineer

Assignment Description

We are seeking a Senior Data Engineer to own the technical foundation of a central Microsoft Fabric data platform. Working with the digital lead and a data analyst, you will build a medallion architecture that turns raw data into reliable, business-ready data products for technical product teams. The hybrid arrangement requires on-site attendance exactly three days per week.

In this role, you work independently and take technical ownership, while communicating and documenting trade-offs clearly. You have strong communication skills and thrive in a small, autonomous team.

Architecture and Pipelines

You will develop Bronze, Silver and Gold lakehouse layers and robust, idempotent pipelines with quality checks between layers. You will define and document data products and models, specifying clear ownership and SLAs, and work with architects to establish and follow data architecture standards.

Data Quality and Documentation

You will implement data validation, testing and monitoring to ensure accuracy, completeness and consistency across platforms, investigate and resolve data issues, and document flows, schemas, pipelines and processes.

Security and Compliance

You will ensure data handling meets security, privacy and regulatory requirements through access controls, encryption and secure data patterns, supporting audits and compliance activities as required.

Collaboration

You will work with data scientists, analysts and business units to understand data needs and provide technical guidance and support. Your approach centres on data products and domain ownership, with an understanding of production pipeline operations and the ability to define the interface between the platform and dependent teams.

Required Qualifications:

• A sound understanding of how data is modelled and databases are designed.
• Experience applying version control and CI/CD practices through GitHub Actions.
• Knowledge of how agentic coding is used together with tools and standards for validation and data quality.
• In-depth experience with PySpark, Spark SQL and Delta Lake, including strategies for partitioning, patterns for MERGE operations, and streaming versus batch processing.
• A senior-level profile with a proven record of hands-on delivery, along with experience working in cross-functional teams.
• Production experience using either Microsoft Fabric or Azure-based Databricks, together with hands-on experience in Azure data platforms and Microsoft Fabric, and knowledge of Azure cloud services.