Design, develop, and maintain scalable data processing pipelines using Azure Databricks and PySparkScala.
Implement and optimize machine learning workflows using Databricks integrated machine learning capabilities.
Collaborate with data scientists, engineers, and other stakeholders to gather requirements and deliver solutions.
Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure.
Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
Explore data, build secure and compliant pipelines, and produce cleansed and enhanced datasets for analysis using various Azure data services and frameworks (1.) To be responsible for providing technical guidance to a team of developers, enhancing their technical capabilities and increasing productivity. (2.) To conduct comprehensive code reviews, establish and oversee quality assurance processes, performance optimization , implementation of best practices and coding standards to ensure succeful delivery of complex projects. (3.) To ensure process compliance in the assigned module, and participate in technical discussionsorreview as a technical consultant for feasibility study (technical alternatives, best packages, supporting architecture best practices, technical risks, breakdown into components, estimations). (4.) To collaborate with stakeholders to define project scope, objectives, deliverables and accordingly prepare and submit status reports for minimizing exposure and closure of escalations.