SC Cleared Senior Data Engineer

Senior Data Engineer – Multi-Cloud
Location: London, United Kingdom (Hybrid)
Salary: £65,000 – £85,000 per annum

An experienced Senior Data Engineer is required to design, build, and optimise scalable data platforms for complex transformation programmes. This position demands strong expertise across cloud-based data engineering, architecture, and DevOps practices, working within both AWS and Azure environments.

You’ll play a key role in enabling seamless data integration, supporting advanced analytics, and delivering robust, reusable data solutions aligned with client data strategies.

Key Responsibilities

• Design and develop scalable, automated data pipelines using cloud-native technologies (e.g. Spark), integrating with on-premise sources where required.

• Implement event-driven data architectures and optimise pipeline performance, parallelisation, and risk detection (e.g. fraud use cases).

• Build and maintain data access layers and ETL frameworks to support self-service and ad-hoc onboarding of data assets.

• Develop reusable engineering patterns and tooling to improve scalability and consistency across programmes.

• Apply DevOps and software engineering best practices, including Infrastructure as Code (IaC) and CI/CD pipelines.

• Collaborate with cross-functional teams to ensure high-quality data integration, governance, and performance.

• Deliver secure, resilient, and scalable data solutions across hybrid cloud environments (AWS and Azure).

• Configure and optimise data services including:

  • AWS: EC2, EMR, S3, Lambda

  • Azure: Blob Storage and related services

• Ensure alignment with client data strategies, governance frameworks, and security standards.

• Support knowledge sharing and capability uplift across teams.

Skills & Experience – Essential

• Strong experience in data engineering and data architecture in complex programme environments.

• Proven ability to develop and manage automated data pipelines using technologies such as Spark.

• Expertise in event-driven architectures and pipeline performance optimisation.

• Deep understanding of ETL processes, data integration, and self-service onboarding models.

• Hands-on experience across both AWS and Azure platforms.

• Practical knowledge of:

  • AWS: EC2, EMR, S3, Lambda

  • Azure: Blob Storage and related services

• Understanding of cloud networking concepts in AWS and Azure.

• Experience with Terraform (IaC) and GitLab (CI/CD).

• Solid grounding in DevOps and modern software engineering practices.

Desirable

• Experience in government or regulated sectors.

• Familiarity with large-scale data platforms and data access layers.

• Exposure to fraud detection or risk analytics use cases.

• Knowledge of data governance, security, and compliance frameworks.

Security Requirements

• This role requires Security Clearance (SC).

• Candidates must hold active clearance or be eligible and willing to undergo the process.

Engineering Practices

• Strong commitment to automation, scalability, and reusability.

• Advocate for DevOps principles, CI/CD, and Infrastructure as Code.

• Experienced working in agile, collaborative teams.

• Dedicated to delivering robust, production-grade data solutions.