Data Engineer

Assignment Description

We are seeking an experienced Data Engineer for hands-on software and data engineering in AWS. You will take data pipelines and engineering services through design, development, testing and deployment into secure, reliable operation. These capabilities will supply data to business applications, operational processes, reporting and AI workloads. Technical work motivates you, and you enjoy creating solutions that combine reliability with scalability.

As part of a cross-functional engineering team, you will contribute to common platform capabilities alongside data products and pipelines for specific sources. Production code, infrastructure, automation and service reliability are central to the assignment. This is not a Data Analyst position; dashboard development and ad hoc analysis are not its primary focus. You work readily with developers, architects and other stakeholders.

The development work spans scalable pipelines built with Python and Spark, integrations with APIs and ingestion based on Kafka. You will maintain these solutions, create reusable components for ingestion and orchestration, and provision AWS resources through Terraform. Automated tests at unit, integration and data-contract level, together with reconciliation checks and data validation, will support secure, tested and maintainable code.

Taking ownership and working through complex technical problems come naturally to you. Your operational work will cover monitoring pipelines and infrastructure, investigating incidents, maintaining alerts and recovery procedures, and helping prevent problems from recurring. Quality, security and operational stability matter to you. You will implement access controls, encryption and secrets management, identify performance constraints and optimise resource usage and cloud expenditure.

You will also contribute to design decisions, estimates, migration work and controlled deployments. Maintaining technical documentation, sharing knowledge, communicating clearly and proactively flagging risks or issues are part of the role. Approved AI-assisted development tools will support implementation, testing and documentation, with their generated code properly reviewed and tested. Expected outcomes include dependable pipelines that deliver data on time, effective recovery, fewer recurring incidents and delivery against agreed commitments.

Required Qualifications:

• A degree in Computer Science, Engineering or a related subject, or equivalent practical experience.
• A fluent command of English and strong technical communication skills, including the ability to explain the reasoning behind engineering decisions.
• Experience working with containerised workloads and Kubernetes / EKS.
• Knowledge covering data quality, monitoring, observability and production support, plus the ability to diagnose technical problems.
• Experience using SQL and working with data structures, schemas, file formats and data modelling.
• Experience creating pipelines for both batch and streaming data with Apache Spark, Kafka and APIs, together with experience using orchestration tools such as Apache Airflow.
• Practical AWS experience, particularly involving S3, IAM, encryption, logging and networking, and an understanding of infrastructure as code, preferably Terraform.
• Strong practical experience in data engineering and development in cloud environments, with a proven record of creating, deploying and maintaining data pipelines or data platform services in production. This includes strong Python development skills, strong hands-on experience with Python and AWS, and experience building production-grade data pipelines.
• Experience in development workflows involving Git, code reviews, automated testing and CI/CD, along with experience working with reliable operations and maintainable code.

It is considered meritorious if you have worked in banking or another regulated industry, hold relevant AWS certifications, have migrated legacy data, or have experience using dbt and modern practices for data transformation.