We are looking for a Data Engineer to join our Team to build, maintain, and enhance scalable, high-performance data pipelines and cloud-native solutions. The ideal candidate will have deep experience in Databricks, Python, PySpark, Elastic Search, and SQL, and a strong understanding of cloud-based ETL services, data modeling, and data security best practices.
This is a hands-on technical role that requires strong problem-solving skills and the ability to work independently while collaborating with cross-functional teams to deliver business-critical data solutions.
Key Responsibilities:Design, implement, and maintain scalable data pipelines using Databricks, PySpark, and SQL.
Develop and optimize ETL processes leveraging services like AWS Glue, GCP DataProc/DataFlow, Azure ADF/ADLF, and Apache Spark.
Build, manage, and monitor Airflow DAGs to orchestrate data workflows.
Integrate and manage Elastic Search for data indexing, querying, and analytics.
Write advanced SQL queries using window functions and analytics techniques.
Design data schemas and models that align with various business domains and use cases.
Optimize data warehousing performance and storage using best practices.
Ensure data security, governance, and compliance across all environments.
Apply data engineering design patterns and frameworks to build robust solutions.
Collaborate with Product, Data, and Engineering teams; support executive data needs.
Participate in Agile ceremonies and follow DevOps/DataOps/DevSecOps practices.
Respond to critical business issues as part of an on-call rotation.
Must-Have Skills:Databricks (3+ years): Development and orchestration of data workflows.
Python & PySpark (3+ years): Hands-on experience in distributed data processing.
Elastic Search (3+ years): Indexing and querying large-scale datasets.
SQL (3+ years): Proficiency in analytical SQL including window functions.
ETL Services:
AWS Glue
GCP DataProc/DataFlow
Azure ADF / ADLF
Airflow: Designing and maintaining data workflows.
Data Warehousing: Expertise in performance tuning and optimization.
Data Modeling: Understanding of data schemas and business-oriented data models.
Data Security: Familiarity with encryption, access control, and compliance standards.
Cloud Platforms: AWS (must), GCP and Azure (preferred).