Position Qualification Summary:

The Data Engineer will join the Americas IT software and architecture team. This business centric IT team allows for business-focused development of new, cutting edge, solutions as well as integrating overall global/regional IT initiatives into the business. In this role, we are seeking a talented and motivated Data Engineer with hands-on experience in Apache Spark and Microsoft Fabric to join our dynamic data engineering team. In this role, you will be at the heart of our data initiatives, designing robust data pipelines, optimizing data architectures, and enabling data-driven decision-making across the organization.

The Data Engineer would be expected to work independently and/or in conjunction with appropriate business unit/IT leaders or cross-functional teams during any engagement. They would be tasked with understanding technological and business problems and focused on delivering cutting-edge solutions. The Data Engineer must possess the ability to architect or integrate the appropriate solution from the ground up based on a set of business requirements.

Key Responsibilities:

Design and Develop Data Pipelines: Architect, build, and maintain scalable data pipelines leveraging Apache Spark and Microsoft Fabric to process large volumes of structured and unstructured data from diverse sources.Data Integration: Integrate data from internal and external sources, ensuring accuracy, consistency, and reliability throughout the data lifecycle.Performance Optimization: Monitor and optimize ETL/ELT processes for performance, scalability, and cost-efficiency, proactively identifying bottlenecks and implementing improvements.Data Quality and Governance: Implement best practices for data quality, data cataloging, and lineage, and support governance policies to ensure compliance and high standards.Collaboration: Work closely with data scientists, analysts, business stakeholders, and cross-functional IT teams to gather requirements, deliver insights, and support analytical models.Documentation: Prepare and maintain detailed documentation for data workflows, pipeline architectures, data schemas, and processes.Continuous Improvement: Stay current with emerging trends and technologies in data engineering, suggesting and implementing innovative solutions to enhance the data platform.Troubleshooting: Diagnose, debug, and resolve data pipeline and infrastructure issues to ensure robustness and reliability of the data ecosystem.Security: Adhere to and enforce data security and privacy guidelines, ensuring sensitive data is handled appropriately.