Azure AI DataBricks Engineer

You will lead the data-engineering thread across a fast-growing, multi-project enterprise programme. Most of the active workstreams are Databricks-led — this is the engineer who unlocks them. You will design, build and own production-grade Azure \+ Databricks pipelines that feed AI-powered insights, natural-language querying and downstream CRM and operational activation — and you will do it client-facing, in UK hours, alongside a small, senior team.

What you'll own

• Architecture leadership.

Define and document target architectures across Azure \+ Databricks for new and in-flight projects; review designs from other engineers.

• End-to-end pipeline delivery.

Build production ingestion → bronze/silver/gold (medallion) → curated data products using ADF, PySpark, Delta Lake and Databricks Workflows / DLT.

• AI-grounded data products.

Partner with the GenAI engineer to expose Gold data to RAG agents, Azure AI Search and Azure OpenAI — including chunking, embeddings, citations and evals.

• CI/CD \+ IaC discipline.

Use Databricks Asset Bundles, Terraform/Bicep and Azure DevOps Pipelines to make every change reviewable, testable and roll-back-able.

• Governance.

Apply Unity Catalog, lineage, Purview labels and least-privilege access on day one — not as an afterthought.

• Client interface.

Lead design reviews, status calls and stakeholder workshops in UK hours; turn ambiguous client asks into scoped, estimable work.

• Mentorship.

Coach mid-level engineers, set patterns, run code reviews and quality bars across the data team.

Must-have experience and skills

Technical (deep)

• 8\+ years in data engineering, with 4\+ years on Azure data services in production.
• Expert in Azure Databricks — Delta Lake, PySpark, SQL, Workflows / Jobs, cluster tuning, MLflow basics.
• Strong with Azure Data Factory (or Synapse Pipelines), ADLS Gen2, Azure SQL, Key Vault.
• Medallion / lakehouse architecture experience — bronze / silver / gold modelling with real production trade-offs.
• Hands-on with at least one IaC tool (Terraform or Bicep) and CI/CD via Azure DevOps or GitHub Actions; Databricks Asset Bundles a plus.
• Working knowledge of Azure AI Search and Azure OpenAI — including RAG patterns, embeddings, chunking strategies and grounded prompting.
• Comfortable with Unity Catalog, role-based access and data governance practices (Purview, sensitivity labels).
• Strong Python \+ SQL; clean Git practices; test-driven mindset.

Delivery \& engagement

• Demonstrated experience leading multi-stream programmes — running design reviews, estimation, scope conversations.
• Proven client-facing track record: comfortable presenting to senior stakeholders, defending a design, navigating ambiguity.
• Excellent written English; structured, concise communication.