Primary DetailsTime Type: Full time

Worker Type: Employee

Key Responsibilities:

Design, develop, and optimize data pipelines and workflows within Azure Databricks using Python for data processing and automation.Collaborate with the platform team to ensure that the underlying infrastructure supports the scalability, performance, and security requirements of Databricks workloads.Maintain and enhance ETL pipelines, ensuring they are efficient, reliable, and meet business requirements.Troubleshoot and resolve issues related to data processing, pipeline failures, and performance bottlenecks within Databricks.Develop and implement strategies to optimize data pipelines for speed, cost-efficiency, and performance, ensuring alignment with business objectives.Contribute to the improvement of data architecture, data models, and overall infrastructure within the Azure ecosystem.Lead efforts to monitor, maintain, and scale data workflows and data storage solutions within Azure Databricks.Collaborate with fellow data engineers, data scientists, and analysts to ensure data integrity and smooth flow across systems.Stay up-to-date with the latest industry trends, tools, and best practices in data engineering and cloud technologies.

Requirements:

3+ years of experience as a data engineer with expertise in building and optimizing data pipelines in a cloud-based environment, specifically Azure.Proficiency in Azure Databricks and Python for developing data processing scripts and optimizing workflows.Experience with ETL processes, data pipeline design, automation, and troubleshooting.Hands-on experience with Apache Spark within Databricks, including performance tuning and optimization.Knowledge of Azure Data Lake, Azure SQL, and other Azure-based data storage and processing services.Strong understanding of cloud data platforms, including networking, storage, security, and performance considerations within Azure.Ability to work effectively in cross-functional teams to coordinate and deliver high-quality data solutions.Excellent problem-solving and analytical skills, with the ability to identify and resolve issues related to data workflows.Strong communication skills, both technical and non-technical, with the ability to collaborate effectively with different teams.

Preferred Qualifications:

Experience with Azure Synapse Analytics, Azure Data Factory, or similar data integration tools.Familiarity with CI/CD practices and version control systems like Git.Experience in Agile methodologies and project management tools (e.g., Jira, Trello).Familiarity with data governance principles and best practices in ensuring data quality and security.

Skills:

Adaptability, Agile Application Development, Communication, Critical Thinking, Customer Service, GitLab CI/CD, Intentional collaboration, Managing performance, Process Improvements, Risk Management, Scalability Testing, Software Development, Software Development Life Cycle (SDLC) Methodologies, Stakeholder Management, Team Management

Application Close Date: 10/03/2025 11:59 PM

How to Apply:

To submit your application, click "Apply" and follow the step by step process.

Equal Employment Opportunity:

QBE is an equal opportunity employer and is required to comply with equal employment opportunity legislation in each jurisdiction it operates.