Senior Data Engineer
Job Summary:
We are seeking an experienced and highly motivated Senior Azure Data Engineer to join a Data & Analytics team.
The ideal candidate will be a hands-on technical leader responsible for designing, developing, implementing, and managing scalable, robust, and secure data solutions on the Microsoft Azure platform.
This role involves leading a team of data engineers, setting technical direction, ensuring the quality and efficiency of data pipelines, and collaborating closely with data scientists, analysts, and business stakeholders to meet data requirements.
Key Responsibilities:
Lead, mentor, and provide technical guidance to a team of Azure Data Engineers. Design, architect, and implement end-to-end data solutions on Azure, including data ingestion, transformation, storage (lakes/warehouses), and serving layers.
Oversee and actively participate in the development, testing, and deployment of robust ETL/ELT pipelines using key Azure services. Establish and enforce data engineering best practices, coding standards, data quality checks, and monitoring frameworks.
Ensure data solutions are optimized for performance, cost, scalability, security, and reliability. Collaborate effectively with data scientists, analysts, and business stakeholders to understand requirements and deliver effective data solutions.
Manage, monitor, and troubleshoot Azure data platform components and pipelines. Contribute to the strategic technical roadmap for the data platform.
Qualifications & Experience:
Experience:
· Minimum 6-8+ years of overall experience in data engineering roles.
· Minimum 3-4+ years of hands-on experience designing, implementing, and managing data solutions specifically on the Microsoft Azure cloud platform.
· Proven experience (1-2+ years) in a lead or senior engineering role, demonstrating mentorship and technical guidance capabilities.
· Education: Bachelor’s degree in computer science, Engineering, Information Technology, or a related quantitative field (or equivalent practical experience).
Technical Skills:
· Core Azure Data Services: Deep expertise in Azure Data Factory (ADF), Azure Synapse Analytics (SQL Pools, Spark Pools), Azure Databricks, Azure Data Lake Storage (ADLS Gen2). Data
Processing & Programming:
· Strong proficiency with Spark (using PySpark or Scala) and expert-level SQL skills.
· Proficiency in Python is highly desired.
Data Architecture & Modelling:
· Solid understanding of data warehousing principles (e.g., Kimball), dimensional modelling, ETL/ELT patterns, and data lake design.
Databases:
· Experience with relational databases (e.g., Azure SQL Database) and familiarity with NoSQL concepts/databases is beneficial.
Version Control:
· Proficiency with Git for code management.
Leadership & Soft Skills:
· Excellent leadership, mentoring, problem-solving, and communication skills, with the ability to collaborate effectively across various teams.