UST is looking for a highly skilled Azure Data Architect with 10+ years of experience to join our team and play a key role in designing and implementing efficient data solutions on the Azure cloud platform. The ideal candidate should have a strong background in data engineering, possess expertise in Azure services, and demonstrate proficiency in data processing technologies, particularly PySpark.

Responsibilities:

Data Pipeline Development:

·       Design, implement, and optimize end-to-end data pipelines on Azure, focusing on scalability and performance.

·       Develop and maintain ETL workflows for seamless data processing.

Azure Cloud Expertise:

·       Utilize Azure services such as Azure Data Factory, Azure SQL Database, and Azure Databricks for effective data engineering.

·       Implement and manage data storage solutions on Azure.

Data Transformation with PySpark:

·       Leverage PySpark for advanced data transformations, ensuring high-quality and well-structured output.

·       Implement data cleansing, enrichment, and validation processes using PySpark.

Performance Optimization:

·       Optimize data pipelines, queries, and PySpark jobs to enhance overall performance and scalability.

·       Identify and address performance bottlenecks within data processing workflows.

Requirements:

·       Proven experience as a Data Engineer, emphasizing expertise in Azure.

·       Proficiency in Azure Data Factory and other relevant Azure services.

·       Expertise in PySpark for data processing and analytics is a must.

·       Experience with data modeling, ETL processes, and data warehousing.