Job Description: Data Engineer
Position Overview: We are seeking a skilled and motivated Data Engineer to join our team. The ideal candidate will have a strong background in designing, developing, and optimizing data pipelines and ETL processes, with hands-on experience in Python, Databricks, and/or IICS (Informatica). This role requires a collaborative mindset, a passion for data engineering best practices, and the ability to deliver scalable and efficient data solutions that align with business objectives.

Key Responsibilities:

Data Pipeline Development: Design, develop, and implement data pipelines and ETL processes using Python and Databricks and/or IICS (Informatica).Collaboration: Work closely with data scientists and analysts to understand data requirements and deliver solutions that meet business needs.Optimization: Optimize existing data workflows for performance, scalability, and reliability.Monitoring and Troubleshooting: Monitor and troubleshoot data pipeline issues, ensuring timely resolution to maintain data integrity and availability.Documentation: Document data processes, workflows, and technical specifications to ensure clarity and maintainability.Continuous Learning: Stay updated with the latest industry trends and technologies related to data engineering, cloud services, and data integration.

Qualifications:

Technical Expertise:Proven experience in Python programming and data manipulation.Mid-level knowledge of Databricks and its ecosystem (e.g., Spark, Delta Lake).Experience with IICS (Informatica) or similar cloud-based data integration tools.Familiarity with SQL and database management systems (e.g., PostgreSQL, MySQL).Understanding of data warehousing concepts and best practices.Experience with cloud platforms (e.g., AWS, Azure, GCP).Problem-Solving and Optimization:Mid-level problem-solving skills with a focus on addressing performance issues in data workflows.Experience implementing optimization techniques and evaluating performance improvements using measurable metrics.Best Practices:Understanding of data engineering best practices, including data quality, reliability, and maintainability.Knowledge of robust monitoring practices and data governance principles.Soft Skills:Strong communication and collaboration skills.Familiarity with Agile development methodologies.

Preferred Skills:

Hands-on experience with data storage solutions such as data lakes and data warehouses.Knowledge of data orchestration frameworks and their practical application in past projects.Insights into common pitfalls in the data lifecycle and strategies to avoid them.