Responsibilities

Build data pipelines: Managed data pipelines consist of a series of stages through which data flows (for example, from data sources or endpoints of acquisition to integration to consumption for specific use cases). These data pipelines must be created, maintained and optimized as workloads move from development to production for specific use cases.Architecting, creating and maintaining data pipelines will be the primary responsibility of the data engineer.Drive Automation through effective metadata management:The data engineer will be responsible for using innovative and modern tools, techniques and architectures to partially or completely automate the most-common, repeatable and tedious data preparation and integration tasks in order to minimize manual and error-prone processes and improve productivity.

 

The data engineer will also need to assist with renovating the data management infrastructure to drive automation in data integration and management.
This will include (but not be limited to):

Learning and using modern data preparation, integration and AI-enabled metadata management tools and techniques.Tracking data consumption patterns.Performing intelligent sampling and caching.Monitoring schema changes.Recommending or sometimes even automating existing and future integration flows.

 

Collaborate across departments: The newly hired data engineer will need strong collaboration skills in order to work with varied stakeholders within the organization. In particular, the data engineer will work in close relationship with data science teams and with business (data) analysts in refining their data requirements for various data and analytics initiatives and their data consumption requirements.

 

Educate and train: The data engineer should be curious and knowledgeable about new data initiatives and how to address them. This includes applying their data and/or domain understanding in addressing new data requirements. They will also be responsible for proposing appropriate (and innovative) data ingestion, preparation, integration and operationalization techniques in optimally addressing these data requirements.
Participate in ensuring compliance and governance during data use: It will be the responsibility of the data engineer to ensure that the data users and consumers use the data provisioned to them responsibly through data governance and compliance initiatives.

 

Candidate Specification :

3 to 5 years experience plus appropriate degree; Received a bachelor’s in the fields of Science or Mathematics or related degree;Hands on experience working with BI developers and data analysts to understand the reporting requirements and build out data models for reporting. Strong hands on experience with SQL, preferably in SQL Server, SSRS/SSAS/SSISExperience with Python will be preferred.Strong understanding and hands-on experience with ELT and ETL postulates.Demonstrated expertise and hands-on experience with Azure big data technologies like Azure Data Lake Storage, Azure Data Factory, Azure Logic Apps etc. is required.Experience in batch and real-time data processingDemonstrated expertise and hands-on experience in building scalable and optimized data pipelines along with optimizing performance of database queries.Demonstrated expertise and hands-on experience working with complex and large data sets in a multi-sector environment Proficient in creating optimized stored procedures and other integration techniques within the database;Experience with the following tools:  Microsoft SharePoint, Excel, Power BI, PowerPoint, Word.Proficiency in Azure DevOps is required