Senior Data Engineer - Data Products and AI

About Beacon Beacon Intelligence delivers data and insights that help R&D scientists develop better pharmaceuticals, faster than ever before. Our focus is simple: we curate high-accuracy data enriched and transformed with AI. This is delivered through a range of platforms, increasingly via AI-native products and features that enable customers to extract insights faster than ever before. This is a rare opportunity to make a meaningful impact on patients' lives worldwide. The Role As a Senior Data/AI Engineer, you will play a central role in building and scaling our data products. This is not a traditional back-end data engineering role. You will work closely with product, commercial and technical colleagues, transforming datasets into commercial, customer-facing solutions. You will combine deep technical expertise with a strong understanding of how data can be structured, enriched and operationalised through AI systems. What You'll Be Doing Data Product Development (Core Focus)

• Design and build scalable data products underpinning Beacon's commercial offerings
• Transform raw and third-party data into structured, enriched, product-ready datasets
• Partner with product and commercial teams to define how data is packaged, accessed and monetised
• Enable delivery via APIs, internal tools and customer-facing platforms AI & Data Enrichment
• Apply AI and LLM capabilities to enrich and enhance data (e.g. classification, tagging, summarisation, insight generation)
• Design and build LLM-powered product features
• Integrate RAG into pipelines to improve data quality and unlock new features
• Support development of AI-driven products such as recommendation engines, search and insight tools
• Ensure systems are optimised, well-governed and resilient Platform & Engineering
• Support deployment and lifecycle management of data and AI systems
• Own and evolve the data platform architecture (Databricks, Azure, Airbyte) to support scalability
• Own performance and cost optimisation across Azure Databricks and supporting Azure services, including compute selection, autoscaling, workload monitoring and resource-efficiency improvements.
• Build and maintain robust data pipelines (batch and streaming) delivering reliable, production-ready datasets
• Ensure high standards in data quality, testing and observability
• Improve efficiency through automation, CI/CD and engineering best practice Commercial & Stakeholder Impact
• Partner with senior stakeholders to identify high-value data product opportunities
• Translate business needs into practical, scalable data and AI solutions
• Act as a bridge between technical teams and commercial/product stakeholders
• Contribute to prioritisation based on business impact Leadership & Growth
• Shape the evolution of our data product and AI strategy
• Mentor junior engineers and promote a strong engineering and product mindset
• Help build a high-performing, commercially aware data and AI engineering team What We're Looking For Core Engineering Experience
• Deep, hands-on experience designing, building and operating production data platforms using Azure Databricks and Azure Cloud environments
• Strong experience with Apache Spark, PySpark, Spark SQL and Delta Lake, including incremental processing, schema evolution, performance optimisation and batch and streaming workloads.
• Strong experience implementing Unity Catalog, including catalog and schema design, access control, data lineage, discovery and governance of production data assets.
• Proven experience designing lakehouse and Medallion architectures and converting raw and third-party data into governed, reusable data products.
• Advanced Python and SQL, focused on production-quality, scalable solutions
• Proven experience designing and building data pipelines and models
• Experience working with APIs and data delivery mechanisms (critical for productisation)
• Experience deploying Databricks workloads through automated CI/CD, preferably using Azure DevOps or GitHub Actions, automated testing and Databricks Declarative Automation Bundles.
• Experience integrating Azure Databricks with ADLS Gen2, Azure identity, Key Vault, APIs and external data sources. Data Product Mindset (Key Differentiator)
• Experience building or supporting data products or customer-facing data solutions
• Strong understanding of how to structure, expose and scale data for end users or clients
• Ability to think beyond pipelines and focus on commercial value
• Experience working closely with product or commercial teams AI Engineering
• Familiarity with generative AI / LLMs (Azure OpenAI, LangChain etc.) and experience applying them to real-world use cases
• Experience evaluating, monitoring and optimising AI systems
• Experience working with unstructured data (text, documents, web data)
• Familiarity with embeddings, vector databases or semantic search
• Understanding of AI governance and data ethics Desirable
• Experience with Airbyte or modern data ingestion tools
• Experience with Terraform or infrastructure-as-code
• Exposure to LLM frameworks (e.g. LangChain, LlamaIndex)
• Experience with experimentation or A/B testing frameworks
• Background in building data-driven / AI-enabled products
• Domain experience in life sciences How You Work
• Focused on shipping products, not just experimentation
• Think in terms of products, not just pipelines
• Commercially aware and motivated by impact
• Combine technical excellence with pragmatism and pace
• Collaborate effectively across disciplines and influence stakeholders
• Curious about AI and emerging technologies, with a focus on reliable, scalable delivery Why Join Beacon Intelligence
• Build products, not just infrastructure – directly helping scientists treat and cure diseases
• Shape our AI future – define how we embed AI into products and workflows
• High-impact role – influence product strategy and commercial performance
• Modern, evolving stack – work with cutting-edge tools across data engineering and AI
• Growth and ownership – join early and grow as we scale our capabilities
• Collaborative environment – stakeholders who actively value data and insight Location & Working Pattern
• London (hybrid working) – 2 days per week minimum in office
• Flexible working arrangements available Time Spent (Directional)
• 60% hands-on technical
• 20% management and mentoring
• 20% exploration Pay range and compensation package Salary range: £80,000 to £90,000 + Bonus Equal Opportunity Statement Beacon Intelligence is committed to diversity and inclusivity in the workplace. Apply