Senior Data Engineer London Full Time February 20, 2025 Find out more Senior Data Engineer

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Job summary:

D+G is transforming into a technology-powered product business serving customers around the world. Our products and services rely heavily on compelling digital experiences and data-led journeys for our B2B and B2C clients and customers.

This is a key lead engineering role with D&G’s technology team which presents a challenging and exciting opportunity which will require real enthusiasm and modern Data engineering experience to stabilise, enhance and transform D&G’s operational Customer Databases as they move from legacy systems to new scalable cloud solutions across the UK, EU and US. The role will require an experienced Data engineer with good knowledge of IBM Datastage & DB2, AWS & Databricks pipelines who is able to excel in challenging environments with the confidence to help the teams steer the right course in the development of the data platform alongside supporting any required tooling decisions.

The role will enable D+G to deliver a modern data services layer, delivered as a product, and which can hence be consumed by key service channels and stakeholders on demand.

Strategic Impact:

Quality Customer Data is the lifeblood of D&G’s operations which allows us to serve our customers with outstanding propositions and outcomes. This role will be integral to supporting this through the following areas of delivery:

On-Prem Customer Data Platform Stabilisation

This role will initially help stabilise existing on-prem Customer Data Platforms to help serve our customers and protect the one-billion-pound revenue across the UK and EU. Targets will be to reduce merge and compliance incident backlog, promote more automation and support onboarding of 3rd party to provide managed break / fix service.

Support Data Growth in UK and US Markets

Supporting further growth in UK / EU markets through enhancement of the Customer on-prem IBM platforms to ensure they remain available, robust and secure for growing data demands in UK / EU whilst leading on delivery of cloud based solutions for the US pipelines and Data platform.

Knowledge, Expertise, Complexity and Scope:

Knowledge in the following areas essential:
 

Data Engineering Experience:

Databricks: Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases. AWS Cloud: Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM. IBM Skills: DB2, Datastage, Tivoli Workload Scheduler, Urban Code Programming Languages: Proficiency in Python, SQL. Data Warehousing & ETL: Experience with modern ETL frameworks and data warehousing techniques. DevOps & CI/CD: Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog). Familiarity with big data technologies like Apache Spark, Hadoop, or similar. Test automation skills ETL/ELT tools and creating common data sets across on-prem (IBM Datastage ETL) and cloud data stores Leadership & Strategy: Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures. Customer Data Platform Development: Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs. Data Governance & Best Practices: Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards. Pipeline Automation & Optimisation: Drive the automation of data pipelines and workflows to improve efficiency and reliability. Team Management: Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards. Cross Company Collaboration: Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems. Cloud Management: Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform. Performance Monitoring: Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers. Key Responsibilities: Manage outcomes for the on-prem customer platform break / fix service. Build and deliver automated and secure data pipelines that provisions data for all business users and applications (including operational and insight) Work with the DevOps developer and testers to help support and create our AWS & Databricks infrastructure and continuous delivery pipelines.

Ensure all developments are tested and deployed within the automated CI / CD pipeline where appropriate.

Version and store all development artefacts in the agreed repository

Ensure all data are catalogued and appropriate documentation is created and maintained for all ETL code and associated NFR’s.

Collaborate with the product owner (Data) & business stakeholders to understand the requirements and capabilities.

Collaborate with the lead architect, CCOE to align to the best practice delivery strategy.

Participate in the teams agile planning and delivery process to ensure work is delivered in line with the Product Owners priorities

Create low level designs for Epic’s and Stories and where required, support the lead architect to create the designs to enable the realization of the Data Lake, Operational Customer DB, Warehouse and marts while ensuring scalability, security by design, ease of use and high availability & reliability.

Identify the key capabilities needed for success and the technology choices, coding standards, testing techniques and delivery approach to deliver reliable data services

Learn emerging technologies to keep abreast of new or better ways of delivering the Data Pipeline

Welcomes a challenge as a new opportunity to learn new things and make new friends whilst always thinking of better techniques to solve problems

At Domestic & General, we are proud of our 100-year legacy and excited about our future growth plans. We are expanding our horizons, entering new markets and territories internationally and we need your expertise to help us on the journey.

Apply now Latest Opportunities Commercial Finance Analyst London Full Time

Commercial Finance Analyst Location: Flexible working with visits to our office in Wimbledon as and when required Salary: Competitive Contract Type: Permanent We have a fantastic opportunity for a Commercial Finance Analyst to join our Commercial...

View + apply Customer Sales Advisor - Outbound - Part Time Nottingham Part Time

D&G’s purpose is to make the world a better place, one repair at a time with a mission to keep or customer’s world running. You will be responsible for delivering an excellent experience to our customers first time, every time using all...

View + apply Procurement Administrator London Full Time

Procurement Administrator Location: Wimbledon – Hybrid/Remote Options available subject to interview process. Salary: Competitive Contract Type: Permanent We have a fantastic opportunity for a Procurement Administrator to join our Group...

View + apply Head of Propositions Growth Markets Madrid Full Time

Role Title: Head of Propositions Location: Spain Salary: Competitive Contract Type: Permanent We have an exceptional opportunity for a Head of Propositions to join our Product & Technology Division. The core responsibility of this role will...

View + apply OBTM Telesales Agent Portugal Madrid Full Time

Resumo do trabalho: • O propósito da D&G é tornar o mundo num lugar melhor, uma reparação de cada vez, com a missão de manter o mundo do cliente a funcionar. Será responsável por entregar uma excelente experiência aos nossos clientes...

View + apply Sales Agent (m/w/d) Wiesbaden Full Time

Domestic & General ist der führende Spezialist für Garantieprodukte und Services im Bereich Haushaltsgeräte und Unterhaltungselektronik. Unsere weltweit über 2.000 Mitarbeiter betreuen aktuell über 16 Millionen Kunden in 15 Ländern. Damit...

View + apply Business Change Manager (6 month FTC) Nottingham Full Time

We are looking for a Business Change Manager to join our Transformation Team on a fixed term basis to champion best practice in organisational change management and drive improvements. Domestic & General is a company with an ambitious future....

View + apply Beko Europe Team Leader Nottingham Full Time

Contract Type: Permanent, Full time Location: Flexible working with visits to our office in Nottingham Salary: £30,000- £38,000 per annum + Bonus Closing Date: Friday 7th March Are you a highly driven sales and people leader who has a...

View + apply Quality Assurance Officer Nottingham Full Time

The Role: Quality Assurance Officer Location: Flexible working – both the option to work remotely and office based available. Contract type: Full time 37.5hrs per week Salary: £24,800 per annum We are excited to announce that we are now...

View + apply Senior MI Analyst London Full Time

Role Title: Senior MI Analyst Location: Wimbledon (Flexible) Salary: Competitive Contract Type: Permanent Reporting line: Head of Commercial MI Closing Date: 21st February 2025 This is an exciting opportunity to join the Group MI department, in a...

View + apply Director of Brand and Communications London Full Time

Director of Brand and Communications Location: Hybrid working with visits to our office in Wimbledon and all other D&G offices as and when required. Salary: Competitive Contract Type: Permanent We have a fantastic opportunity for a Director...

View + apply Customer Support Agents - U.S Norwalk Full Time

Customer Resolutions Agent Location: Norwalk, Connecticut – with Hybrid & Remote Options available Salary: $22 per hour Contract Type: Permanent, Full Time We have a great opportunity for a Customer Resolutions Agent to join our US Contact...

View + apply