Job Overview:

We are seeking a talented and motivated Senior Data Engineer with 6 to 10 years of experience to join our dynamic team. The ideal candidate will possess a strong background in designing, building, and maintaining scalable data pipelines and databases. You will work closely with engineering teams and product owner to translate requirements into resilient, secure, and high-performance software. Key responsibilities include designing, coding, testing, and implementing new or existing software products based on specifications. A crucial aspect of this role is a commitment to continuous learning and adaptability to evolving technologies and market trends.

Key Responsibilities:

Data Pipeline Development:Design, build, and maintain robust and scalable data pipelines to process and transform large volumes of data from various sources.Implement ETL (Extract, Transform, Load) processes to ensure data is accurately and efficiently integrated into data warehouses or data lakes.Database Management:Develop, build and optimize database queries and schema designs to enhance performance and support analytical needs.Data Integration:Collaborate with cross-functional teams to gather data requirements and integrate data from diverse sources including APIs, flat files, and databases.Ensure data consistency, accuracy, and integrity across various systems.Data Quality and Monitoring:Implement data quality checks and monitoring processes to identify and resolve data issues proactively.Develop and maintain data documentation and metadata for transparency and reproducibility.Performance Tuning:Monitor and tune the performance of data pipelines and databases to ensure optimal operation and scalability.Troubleshoot and resolve data-related issues in a timely manner.Collaboration and Communication:Work closely with various stakeholders to understand their data needs and provide appropriate solutions.Communicate technical concepts and solutions effectively to both technical and non-technical team members.Continuous Learning and Adaptation:Stay current with emerging technologies, trends, and best practices in data engineering.Demonstrate a proactive attitude towards learning new tools and technologies, and adapt quickly to changes in the market and industry requirements.

Qualifications:

Education:Bachelor’s degree in Computer Science, IT, Engineering or a related fieldExperience:6 to 10 years of hands-on experience in data engineering or a related role.Proficiency with data warehousing solutions and ETL tools.Technical Skills:Proficient in SQL: Extensive experience in writing complex queries, stored procedures, and functions in SQL and Azure SQL Managed Instance (SQL-MI).Data Pipeline Orchestration: Hands-on experience with data pipeline orchestration tools such as SSIS, Azure Data Factory, and Azure Synapse.Knowledge of Microsoft Fabric: Familiarity with end-to-end analytics and data platform Microsoft FabricProgramming Skills: Proficiency in Python and PySpark for data manipulation and analysis.Azure DevOps: Experience with Azure DevOps for version control, continuous integration, and deployment.Repository Management: Strong practical experience with code repositories such as Azure Repos and Bitbucket.Database Concepts: Solid understanding of Relational Database Management Systems (RDBMS) and data warehousing concepts.Cloud Platforms: Experience working with AzureNice to have:Data Visualization: Knowledge / familiarity with data visualization tools like Power BI to create insightful and interactive reports.Programming Knowledge: Familiarity with additional programming languages such as .NET, Java, or front-end technologies like React/JavaScript is a plus.Project Management & Issue Tracking: Familiarity/exposure in using Jira - creating/managing stories, sub-tasks, tasks, bugsSoft Skills:Excellent problem-solving skills and attention to detail.Strong analytical and organizational abilities.Ability to work effectively both independently and as part of a team.Effective communication skills to interact with various stakeholders.Willingness to learn new technologies and adapt to rapid changes in the data engineering landscape.