Greenwood Village, CO, USA
4 days ago
Data Engineer
Kforce has a client in Greenwood Village, CO that is seeking a skilled Data Engineer to support a high-impact data engineering initiative focused on AWS infrastructure, Spark-based transformations, and orchestration tools. This individual contributor role requires hands-on experience building data pipelines, processing large-scale JSON messages, and deploying solutions in cloud environments. The ideal candidate has a solid foundation in data engineering within enterprise environments, strong Spark/PySpark expertise, and experience with AWS-native services and modern orchestration tools. Responsibilities: * Build and optimize scalable data pipelines to process and transform structured and semi-structured data using PySpark and SparkSQL * Work with JSON objects to parse messages and send MDK messages via Kafka to S3 * Create and manage data endpoints with defined schemas, orchestrated through RabbitMQ and Kafka * Execute data transformations from JSON to RDDs using Spark on AWS EMR/EC2 * Support orchestration through AWS Step Functions with a future transition to Airflow * Use SQL to extract, transform, and load data into reporting and dashboarding tools * Collaborate with DevOps and CI/CD teams to maintain GitLab pipelines and automate infrastructure deployment using Terraform * Maintain version-controlled ETL code in GitLab and participate in testing, deployment, and monitoring workflows * Participate in cross-functional engineering discussions, aligning with best practices and timelines
Por favor confirme su dirección de correo electrónico: Send Email