Senior Data Engineer

Senior Data Engineer (active SC clearance required)
Location: UK (Remote, occasional London travel) | Permanent: £60-85K

Role Overview
Hands-on Data Engineer working with public sector clients to design, build, and operate production data platforms. Lead technical architecture across cloud environments, mentor delivery teams, and embed analytics/AI capabilities in agile, client-facing projects.

Key Responsibilities

• Translate complex business challenges into scalable data pipelines, models, and analytics solutions

• Build/operate production-grade pipelines using SQL/Python/Spark across lakehouses and event-driven systems

• Architect semantic models and data visualisations for business intelligence platforms

• Deploy cloud data/AI solutions (AWS/Azure) with robust CI/CD, testing, and security practices

• Integrate ML models, experimentation frameworks, and governance workflows

• Deliver client-ready dashboards, insight narratives, and consulting materials

• Collaborate across multidisciplinary teams in agile, iterative delivery cycles

Technical Environment
Core: Advanced SQL/Python, PySpark/Pandas, Spark orchestration tools
Cloud: AWS (Databricks/Redshift equivalents), Azure analytics platforms, Git/CI-CD
Architecture: Data lakehouses, event streaming, semantic modelling for BI tools
AI/ML: Model APIs, prompt engineering, experimentation frameworks
Governance: Data catalogues, lineage tracking, security/access controls, GDPR

Essential Requirements

• Expert SQL/Python for data modelling, transformation, and automation

• Production data engineering: Spark/PySpark, cloud platforms (AWS or Azure)

• Data architecture experience: lakehouses, event-driven systems, BI semantic models

• Software engineering: Git workflows, CI/CD pipelines, testing frameworks

• Analytics: experimentation design, statistical analysis, insight storytelling

• Consulting mindset: stakeholder collaboration, client deliverables, agile delivery

• ACTIVE SC clearance required

Desirable

• Public sector/government project delivery

• Advanced BI semantic modelling (Power BI/Tableau equivalents)

• Operational intelligence platforms or data mesh implementations

• Data governance frameworks (lineage/catalogues/access management)