*Palantir Foundry experience is mandatory*

Role Purpose

The Senior Data Engineer will be responsible for hands-on operation, stabilisation, and improvement of critical data engineering pipelines, supporting their transition into sustainable BAU operations. This role requires Palantir Foundry Experience.

The role focuses on:

• Keeping mission-critical pipelines running reliably
• Reducing operational risk and manual intervention
• Supporting knowledge transfer and service readiness
• Enabling BAU support teams to operate pipelines confidently and independently
• Palantir Foundry experience is mandatory.

This is a delivery-critical role, operating close to production systems, data flows, and operational processes, in collaboration with Data Architects, service managers, and Healthcare pipeline SMEs.

Key Responsibilities

1. Critical Pipeline Operation & Stabilisation

• Develop and understanding of day-to-day operation of critical data engineering pipelines.
• Investigate and resolve pipeline failures, data issues, and processing delays.

Identify and address:

• Fragility in pipeline execution
• Manual or error-prone operational steps
• Recurrent failure patterns
• Contribute directly to ensuring operational continuity for high-risk pipelines.

2. Pipeline Analysis & Technical Understanding

• Deeply understand how critical pipelines work end-to-end, including:
• Source systems and ingestion mechanisms
• Processing jobs, transformations, and validations
• Intermediate datasets and temporary storage
• Outputs and downstream consumers
• Support data flow analysis activities by providing detailed, implementation-level insight.
• Work closely with the Data Architect to validate architecture and data flow documentation.

3. Business Logic & Data Processing Support

• Implement pipeline transformations in line with:
• Documented business rules
• Statutory and regulatory requirements
• Assist in making implicit or tribal business logic explicit, supporting:
• Documentation
• Knowledge repositories
• Operational playbooks

Help identify where business logic is:

• Core and mandatory
• Historical or technical debt
• A candidate for simplification

4. BAU Service Model Enablement

Support creation and validation of:

• Operational runbooks
• Incident response playbooks
• Monitoring and alerting configurations
• Ensure pipelines are supportable, repeatable, and observable in BAU.

5. Monitoring, Alerting & Incident Support

Refine pipeline-level monitoring and alerting, focusing on:

• Failures
• Data quality issues
• Performance degradation
• Support root-cause analysis for incidents and near misses.
• Reduce reliance on reactive or manual discovery by improving proactive alerting.

6. Knowledge Transfer & Handover

• Actively participate in knowledge capture and transfer from Healthcare pipeline SMEs.

Produce and validate:

• Runbooks
• Troubleshooting guides
• Known-issue playbooks
• Support shadow and supervised operation periods for pipelines transitioning into BAU.
• Enable formal handover and operational sign-off where required.

7. Pipeline Improvement (Tactical Focus)

Identify tactical pipeline-level improvements that:

• Reduce operational burden
• Automate manual steps
• Improve reliability and observability
• Assess improvements against:
• Platform modernisation programme
• Effort vs benefit
• Operational risk reduction
• Implement agreed tactical improvements within project constraints.

Required Skills & Experience

Essential

• Strong experience as a Senior Data Engineer working with production data pipelines.

Proven ability to:

• Support and troubleshoot live data workflows
• Work hands-on with batch and scheduled processing

Experience operating within:

• BAU/run-and-maintain environments
• SLA or service-critical contexts
• Comfortable working with:
• Incomplete documentation
• Legacy or complex pipeline designs
• Strong collaboration skills with architects, operations teams, and SMEs.
• Palantir Foundry is Mandatory

Desirable

• Experience in public sector or healthcare data environments, including regulated or statutory datasets.

Familiarity with:

• Change, incident, and problem management processes
• Knowledge transfer and service transition activities
• Exposure to large, centralised data processing platforms and cloud-hosted data estates.

Working Style & Expectations

• Hands-on and delivery-oriented
• Operationally pragmatic (keeps services running first)
• Comfortable working under time-boxed Phase 1/Phase 2 delivery pressures
• Focused on stability, clarity, and repeatability
• Produces artefacts that support operations, not just engineering teams

Key Outcomes This Role Supports

• Stable and reliable operation of critical pipelines
• Reduced operational risk and manual intervention
• Operational playbooks and documentation validated for BAU use
• Successful knowledge transfer and pipeline handover
• Tactical pipeline improvements implemented or prioritised

Summary

This role is suited to a Senior Data Engineer who thrives in live environments, can quickly understand complex pipelines, and is motivated by:

Keeping critical data flowing, reducing operational risk, and enabling others to successfully run and support data pipelines in BAU. This role requires Palantir Foundry Experience.