Line of Service

Internal Firm Services

Industry/Sector

Not Applicable

Specialism

IFS - Internal Firm Services - Other

Management Level

Manager

Job Description & Summary

At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth.

In data engineering at PwC, you will focus on designing and building data infrastructure and systems to enable efficient data processing and analysis. You will be responsible for developing and implementing data pipelines, data integration, and data transformation solutions.

Job Summary:

The ideal candidate will have a strong background in Data warehouse and  Data migration projects, with extensive expertise in ETL processes, data comparison/profiling  etc., and source to target validation across various platforms, including relational and non-relational databases, APIs, BI tools, flat files, Excel, CSV, XML, Json’s, and others. The ETL Test Automation Architect will play a key role in assessing project feasibility, recommending solutions, collaborating with vendors, and leading automation strategies. Proficiency in Python and experience with Datagaps tool are highly desirable.

Experience - 10 to 13 years

Key Responsibilities:

1. ETL Project Feasibility and Design:

- Assess ETL project feasibility based on project requirements and business goals.

- Design and implement ETL automation frameworks/solutions aligned with industry best practices.

- Evaluate and recommend tools, such as Datagaps and Python plugins, to streamline ETL validation and testing processes.

2. Validation and Troubleshooting:

- Conduct thorough validation of all data sources, including relational databases, non-relational databases, APIs, BI tools, flat files, Excel, and CSV.

- Provide guidance and troubleshooting support to ETL Automation Leads and Engineers.

- Assist with complex ETL automation troubleshooting and debug issues related to pipeline configuration, tool bugs, and data quality.

3. Vendor Collaboration:

- Serve as a liaison with vendors to address issues related to Datagaps and other automation tools.

- Follow up with vendors on bugs, tool fixes, and feature enhancements.

4. Automation Strategy and Frameworks:

- Develop and implement the overall ETL automation strategy for company programs.

- Create POCs (Proof of Concepts) and pilot projects to assess the feasibility of new market technologies and trends, including GenAI and next-generation tools.

- Design and execute CI/CD pipelines for ETL processes, leveraging Datagaps tools and Azure DevOps.

5. Training and Documentation:

- Conduct demos and training sessions for next-generation automation tools and solutions.

- Develop comprehensive documentation, including training materials, frameworks, and process guidelines.

6. Evaluation and Innovation:

- Stay up-to-date with market trends and emerging ETL technologies.

- Evaluate new tools and requirements for ETL automation, providing recommendations to stakeholders.

- Drive innovation through the adoption of cutting-edge technologies and practices.

Qualifications:

Required Skills:

- Extensive experience in data warehouse and data migration projects.

- Strong knowledge of ETL processes, database validation, and diverse data source validation.

- Expertise in tools like Datagaps for ETL validation and automation.

- Proficiency in Python for scripting and plugin development.

- Hands-on experience with various data sources, including relational databases (e.g., SQL Server, Oracle, MySQL) and non-relational sources (e.g., NoSQL, MongoDB).

- Familiarity with APIs, BI tools, flat files, Excel, and CSV data integration, Databricks.

- Experience with CI/CD pipeline configuration for ETL processes.

Preferred Skills:

               -Datagaps experience

- Knowledge of GenAI and its applications in automation.

- Understanding of cloud platforms (e.g., AWS, Azure, GCP) for ETL and data workflows.

- Strong analytical and troubleshooting skills.

- Excellent collaboration and communication skills to work effectively with cross-functional teams and vendors.

Additional Requirements:

- Ability to manage multiple projects and deliver results in a fast-paced environment.

- Willingness to take ownership and lead initiatives.

- Strong organizational and problem-solving skills.

Education (if blank, degree and/or field of study not specified)

Degrees/Field of Study required:

Degrees/Field of Study preferred:

Certifications (if blank, certifications not specified)

Required Skills

Optional Skills

Accepting Feedback, Accepting Feedback, Active Listening, Agile Scalability, Amazon Web Services (AWS), Analytical Thinking, Apache Hadoop, Azure Data Factory, Coaching and Feedback, Communication, Creativity, Data Anonymization, Database Administration, Database Management System (DBMS), Database Optimization, Database Security Best Practices, Data Engineering, Data Engineering Platforms, Data Infrastructure, Data Integration, Data Lake, Data Modeling, Data Pipeline, Data Quality, Data Transformation {+ 23 more}

Desired Languages (If blank, desired languages not specified)

Travel Requirements

0%

Available for Work Visa Sponsorship?

No

Government Clearance Required?

No

Job Posting End Date