We are seeking an experienced data engineer to join our team. As a data engineer, you will be responsible for designing, implementing, and maintaining data structures and pipelines for efficient data processing and analysis. Your role will involve working with various databases, including DB2, Cloudant, PostgreSQL, and REDIS. You will collaborate with cross-functional teams to understand data requirements, optimise data flow, and ensure data integrity. Your expertise in data modelling, ETL processes, and database technologies will be instrumental in building robust and scalable data solutions.Responsibilities:Design and implement data models and database structures to meet business requirements.Develop and maintain ETL processes to extract, transform, and load data from various sources.Optimise data pipelines and workflows for efficient data processing and analysis.Collaborate with data scientists and analysts to define data requirements and deliver actionable insightsEnsure data quality, integrity, and security throughout the data lifecycle.
Monitor and troubleshoot data pipelines, resolving issues and performance bottlenecks.Implement data governance and data management best practices.
Stay updated with emerging trends and technologies in data engineering and database management.Collaborate with cross-functional teams to define data integration strategies and implement data-driven solutions.Implement data security measures and ensure compliance with relevant regulations and standards.Document data structures, processes, and workflows for knowledge sharing and future reference.
Mentor junior data engineers and provide technical guidance and support.Tasks:
Design and implement data models for DB2, Cloudant, PostgreSQL, and REDIS databases.Develop ETL processes using tools and frameworks such as Apache Spark, Apache Kafka, or Talend.Optimise data pipelines for performance, scalability, and reliability.Collaborate with data scientists and analysts to understand data requirements and develop data solutions.Implement data governance practices, including data quality monitoring and metadata management.Monitor and analyse data pipelines, identifying and resolving issues or bottlenecks.Implement data security measures, including access controls and encryption.Collaborate with database administrators to ensure optimal database performance and availability.Conduct data profiling and analysis to identify data quality issues and propose solutions.
Implement and automate data validation and testing processes.
Explore and evaluate new tools and technologies for data engineering and database management.