Role : Mid level/Senior Data Engineer

Experience : 6-8 Years

Mandatory Skill: Python, Astronomer, Apache Airflow

Location: Hyderabad

Notice Period:  less than 60 Days

Job Description:

Skills:  Apache Airflow, Astronomer, Python, SQL, Kafka

Responsibilities:

Data Pipeline Development: Design, build, and maintain robust and scalable data pipelines using Apache Airflow and Astronomer to automate complex Business processes. Workflow Orchestration: Collaborate with Automation engineers, and business stakeholders to develop and manage complex Business workflows Performance Optimization: Continuously monitor the performance of Airflow DAGs and optimize them for efficiency, reliability, and scalability. Python Development: Write clean, maintainable, and efficient Python code for data processing, transformation, and analysis tasks. Best Practices: Establish and promote best practices for workflow orchestration, data pipeline design, and coding standards within the team. Testing and Maintenance: Implement testing strategies for data pipelines and workflows, ensuring high availability and minimizing errors. Documentation: Create and maintain comprehensive documentation for data pipelines, workflows, and architectural designs for team knowledge sharing and onboarding. Mentorship: Provide mentorship and guidance to junior developers and team members on best practices, tools, and data engineering concepts. Tool Integration: Stay updated on emerging technologies and platforms related to data engineering, and advocate for tool adoption and process enhancements.

            

Qualifications:

Expert knowledge of Apache Airflow, including DAG creation, scheduling, and debugging. Proficiency with Astronomer for deploying and managing Airflow applications. Strong programming skills in Python, with experience in developing data processing applications and libraries. Familiarity with Azure cloud platform and services related to data processing and storage. Good understanding on distributed systems and experience building real-time integrations with Kafka Experience with version control (e.g., Git) and CI/CD practices. Knowledge of containerization technologies (e.g., Docker) and orchestration tools (e.g., Kubernetes) is preferred. Strong analytical and troubleshooting skills, with the ability to work independently as well as part of a collaborative team. Excellent communication skills, able to articulate technical concepts to both technical and non-technical stakeholders

Preferable

6+ years of professional software development experience 4+ years of Python, DAG and Airflow development 2+ years of Cloud Experience; preferably with Amazon Web Services (AWS) 2+ year of experience working in an Agile-based development environment using Agile concepts such as Continuous Integration (CI/CD) and Test-Driven Development (TDD) Strong experience with monitoring and observability and building scalable services Understanding of various database solutions (SQL, NoSQL) Knowledge of Azure, GCP, AWS