Tech Lead – Azure/Snowflake & AWS Migration

Key Responsibilities

• Design and develop scalable data pipelines using Snowflake as the primary data platform, integrating with tools like Azure Data Factory, Synapse Analytics, and AWS services.

• Build robust, efficient SQL and Python-based data transformations for cleansing, enrichment, and integration of large-scale datasets.

• Lead migration initiatives from AWS-based data platforms to a Snowflake-centered architecture, including:

o Rebuilding AWS Glue pipelines in Azure Data Factory or using Snowflake-native ELT approaches.

o Migrating EMR Spark jobs to Snowflake SQL or Python-based pipelines.

o Migrating Redshift workloads to Snowflake with schema conversion and performance optimization.

o Transitioning S3-based data lakes (Hudi, Hive) to Snowflake external tables via ADLS Gen2 or Azure Blob Storage.

o Redirecting Kinesis/MSK streaming data to Azure Event Hubs, followed by ingestion into Snowflake using Streams & Tasks or Snowpipe.

• Support database migrations from AWS RDS (Aurora PostgreSQL, MySQL, Oracle) to Snowflake, focusing on schema translation, compatibility handling, and data movement at scale.

• Design modern Snowflake lakehouse-style architectures that incorporate raw, staging, and curated zones, with support for time travel, cloning, zero-copy restore, and data sharing.

• Integrate Azure Functions or Logic Apps with Snowflake for orchestration and trigger-based automation.

• Implement security best practices, including Azure Key Vault integration and Snowflake role-based access control, data masking, and network policies.

• Optimize Snowflake performance and costs using clustering, multi-cluster warehouses, materialized views, and result caching.

• Support CI/CD processes for Snowflake pipelines using Git, Azure DevOps or GitHub Actions, and SQL code versioning.

• Maintain well-documented data engineering workflows, architecture diagrams, and technical documentation to support collaboration and long-term platform maintainability.

Required Qualifications

• 9+ years of data engineering experience, with 3+ years on Microsoft Azure stack and hands-on Snowflake expertise.

• Proficiency in:

o Python for scripting and ETL orchestration

o SQL for complex data transformation and performance tuning in Snowflake

o Azure Data Factory and Synapse Analytics (SQL Pools)

• Experience in migrating workloads from AWS to Azure/Snowflake, including services such as Glue, EMR, Redshift, Lambda, Kinesis, S3, and MSK.

• Strong understanding of cloud architecture and hybrid data environments across AWS and Azure.

• Hands-on experience with database migration, schema conversion, and tuning in PostgreSQL, MySQL, and Oracle RDS.

• Familiarity with Azure Event Hubs, Logic Apps, and Key Vault.

• Working knowledge of CI/CD, version control (Git), and DevOps principles applied to data engineering workloads.

Preferred Qualifications

• Extensive experience with Snowflake Streams, Tasks, Snowpipe, external tables, and data sharing.

• Exposure to MSK-to-Event Hubs migration and streaming data integration into Snowflake.

• Familiarity with Terraform or ARM templates for Infrastructure-as-Code (IaC) in Azure environments.

• Certification such as SnowPro Core, Azure Data Engineer Associate, or equivalent.