We are looking for a Data Engineer to join our growing data engineering team at Our Future Health. The Data Engineer will bring an in-depth knowledge of NHS data and data solutions to help solve some of the key challenges around a programme of work at industrial scale with global significance.
Our data engineers work with our participant data (including various health records from the NHS, genetic data, Our Future Health’s own questionnaires) and build pipelines and systems to process it, control quality and create data releases for researchers. You’ll take a lead on developing specific pipelines related to questionnaire and physical measurements data.
Our Future Health will be the UK’s largest ever health research programme, bringing people together to develop new ways to detect, prevent, and treat diseases. We are a charity, supported by the UK Government, in partnership with charities and industry. We work closely with the NHS and with public authorities across all nations and regions of the UK.
What you’ll be doing:
Your key responsibilities will include but not be limited to:
- Support the design and building of data pipelines from data providers to our primary data store and trusted research environment, using languages such as Python.
- Produce logic for data transformation steps as code, which meets the requirements for our end users and builds well curated, accessible, and quality controlled data for analysis.
- Keep abreast of best practice in data engineering across industry, research and Government and facilitating the adoption of standards.
- Working with researchers to understand the data requirements and work with them to deliver the data needed for their projects.
- Communicate to and between technical and non-technical stakeholders as well as facilitate discussions within a multidisciplinary team including scientists, software engineers, product managers and other data engineers.
Requirements
The successful Data Engineer can listen to the needs of technical and business stakeholders and interpret them, and effectively manage stakeholder expectations. To succeed in this role, you will also have some of the following skills:
- Understanding and working knowledge of information governance and data security approaches appropriate for sensitive health data.
- You have the detailed knowledge and understanding of NHS data such as hospital administrative data, disease registries or primary care data, and how they can be used to support research.
- You have the knowledge of health record coding systems and data standards (e.g., ICD, READ and SNOMED codes).
- Familiarity with a range of data structures and standards, with experience with OMOP and FHIR being desirable.
- You have experience with the design and deployment of complex data schemas.
- You have experience in validating and cleaning complex datasets.
- Experience building and maintaining robust, scalable and efficient pipelines capable of processing very large amounts of data from one or multiple systems. You know how to create repeatable and reusable products.
Technical requirements:
- Highly proficient in Python.
- Experience working with data lakes; experience with Spark, Databricks.
- Understanding of common data transformation and storage formats, e.g. Apache Parquet.
- Good understanding of cloud environments (ideally Azure), and workflow management systems (e.g. Dagster, Airflow, Prefect).
- Follow best practices like code review, clean code and unit tests.
- Familiar with version control and Git/GitHub.
- Understanding of containerisation (e.g. Docker) and deployment (e.g. Kubernetes).
Benefits
- Up to £60,000 per annum basic salary.
- Generous company pension package with employer contributions of up to 12%.
- 30 days annual leave (plus bank holidays.)
- Individual development budget
- Flexible and remote working arrangements and a lovely new office in Holborn, Central London.
Join us - let’s prevent disease together.
