Senior Data Engineer

Our client, a leading UK telecommunications provider, is running a programme of data engineering delivery on Azure and Databricks. The work covers data migration, legacy system modernisation and the ongoing build-out of a Delta Lake-based platform. Delivery is spread across several concurrent projects, and the client needs a senior engineer who can own the technical direction on them.

You will join a Data Engineering team of developers and QA engineers inside IT Delivery. The team works alongside Report Engineering, Data Governance, Data Migrations and Automation, so you will be building and handing over work across all of these functions.

What you’ll deliver

• Design and build scalable pipelines and lakehouse architecture on Databricks, using Unity Catalog, Delta Lake and Apache Spark
• Move data from legacy systems into the new platform, with a focus on quality and reliability
• Write and review production-grade Python and SQL, setting the engineering standard across projects
• Build custom solutions using Databricks APIs and integrate them with external systems
• Apply advanced Delta Lake features (time travel, vacuum operations, Z-ordering) to keep performance and cost under control
• Support planning and estimation, and help shape the delivery roadmap
• Mentor a distributed team of data engineers across multiple workstreams
• Work with operations, technology and business stakeholders to turn requirements into delivered data products

What you’ll bring

• 8+ years in data engineering, including at least 1 year in a lead capacity
• Proven Microsoft Azure expertise
• Strong, hands-on Databricks experience with Scala (Spark) and PySpark
• Experience delivering large-scale data engineering and analytics workloads on Databricks
• A track record on data migration or legacy modernisation projects
• Databricks Data Engineer or Azure Data Engineer certification (desirable)

Why this project

• A 12-month engagement with a clear delivery scope
• Senior ownership of technical decisions across several live projects
• Remote
• Outside IR35