Responsibilities:
- Design, develop, implement, and maintain scalable and robust data pipelines using Azure Data Factory, Databricks, and related Azure services.
- Collaborate with data scientists, analysts, and other developers to understand data requirements and translate them into efficient data solutions.
- Develop and optimize ETL/ELT processes for data ingestion, transformation, and loading from various sources into data lakes and data warehouses.
- Ensure data quality, integrity, and security throughout the data lifecycle.
- Monitor and troubleshoot data pipelines, identifying and resolving issues promptly.
- Participate in code reviews, ensuring adherence to coding standards and best practices.
- Stay up-to-date with emerging technologies and trends in the data engineering space, evaluating and recommending new tools and approaches.
- Document data solutions, including data models, data flows, and technical specifications.
- Collaborate with cross-functional teams to define and implement data governance policies and procedures.
Qualifications:
- Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.
- 5+ years of experience in data engineering, ETL development, or a similar role.
- Proficiency in Azure Data Factory, Databricks, Azure Synapse Analytics, and other Azure data services.
- Strong programming skills in Python and SQL.
- Experience with data modeling, data warehousing concepts, and dimensional modeling.
- Familiarity with big data technologies (e.g., Spark, Delta Lake).
- Knowledge of version control systems (e.g., Git).
- Excellent problem-solving, analytical, and communication skills.
- Ability to work independently and collaboratively in a fast-paced environment.
- Strong understanding of data governance and security principles.
- Experience with Agile development methodologies.
