CTG is seeking to fill a Data Scientist – GenAI Models and AI Agentic Systems opening for our client in Armonk, NY. Location: Remote Duration: 15 months Duties: Design rigorous evaluation pipelines for Generative AI models, focusing on open-source alternatives hosted on Hugging Face Hub. Define metrics and build test suites to assess model behavior across factuality, coherence, bias, safety, hallucination, and robustness. Leverage tools such as Ragas, TruLens, Giskard, or LM Eval Harness to compare performance across prompts, tasks, and domains. Create compelling dashboards and visual narratives to track model performance, error distributions, and drift over time. Build visuals using libraries like Plotly, Dash, Seaborn, or Matplotlib for both real-time and retrospective evaluations. Summarize and present findings to cross-functional teams, including product owners, compliance stakeholders, and LLM developers. Develop and validate multi-agent systems using frameworks such as LangGraph, CrewAI, LangChain Agents, or AutoGen. Optimize agent roles, memory, and tool selection for task-oriented pipelines in legal, enterprise, or research domains. Evaluate orchestration strategies (sequential vs. parallel, static vs. dynamic role assignment) for quality and efficiency. Skills: Strong expertise in Generative AI model evaluation and open-source LLMs (e.g., LLaMA 3, Mistral, Falcon, Gemma). Fluency in Python and common data science tools (Pandas, NumPy, Scikit-learn, Jupyter). Proficiency with data visualization libraries (Plotly, Dash, Seaborn, Matplotlib). Familiarity with GenAI evaluation frameworks and benchmarks (TruthfulQA, MMLU, BBQ, Hellaswag). Hands-on experience with Hugging Face Transformers and Datasets. Strong understanding of agentic frameworks (LangGraph, CrewAI, AutoGen, LangChain). Experience: 3+ years in data science, NLP, or ML research roles, or equivalent academic/industry background. Practical experience evaluating open-source LLMs and building evaluation pipelines. Experience developing multi-agent workflows for enterprise or research use cases. Prior exposure to RAG pipelines, vector databases (FAISS, Weaviate, Chroma), and hybrid retrieval strategies preferred. Experience in regulated environments with emphasis on explainability, auditing, or trustworthy AI is a plus. Cloud deployment and monitoring experience in platforms such as AWS SageMaker, Azure ML, or similar is preferred. Education: Minimum Master’s degree in Computer Science, Data Science, Mathematics, or a related field. Excellent verbal and written English communication skills and the ability to interact professionally with a diverse group are required. CTG does not accept unsolicited resumes from headhunters, recruitment agencies, or fee based recruitment services for this role. To Apply: To be considered, please apply directly to this requisition using the link provided. For additional information, please contact Laura Dominguez at Laura.Dominguez@ctg.com. Kindly forward this to any other interested parties. Thank you! The expected base salary for this position ranges from $90,000 to $115,000. Salary offers are based on a wide range of factors including relevant skills, training, experience, education, market factors, and where applicable, licensure or certifications obtained. In addition to salary, a competitive benefits package is also offered. About CTG CTG, a Cegeka company, is at the forefront of digital transformation, providing IT and business solutions that accelerate project momentum and deliver desired value. Over nearly 60 years, we have earned a reputation as a faster and more reliable, results-driven partner. Our vision is to be an indispensable partner to our clients and the preferred career destination for digital and technology experts. CTG leverages the expertise of over 9,000 team members in 19 countries to provide innovative solutions. Together, we operate across the Americas, Europe, and India, working in close cooperation with over 3,000 clients in many of today's highest-growth industries. For more information, visit www.ctg.com. Our culture is a direct result of the people who work at CTG, the values we hold, and the actions we take. In other words, our people define our culture. It's a living, breathing thing that is renewed every day through the ways we engage with each other, our clients, and our communities. Part of our mission is to cultivate a workplace that attracts and develops the best people, reflected by our recognition as a Great Place to Work Certified™ company across many of our global operations. CTG will consider for employment all qualified applicants including those with criminal histories in a manner consistent with the requirements of all applicable local, state, and federal laws. CTG is an Equal Opportunity Employer. CTG will assure equal opportunity and consideration to all applicants and employees in recruitment, selection, placement, training, benefits, compensation, promotion, transfer, and release of individuals without regard to race, creed, religion, color, national origin, sex, sexual orientation, gender identity and gender expression, age, disability, marital or veteran status, citizenship status, or any other discriminatory factors as required by law. CTG is fully committed to promoting employment opportunities for members of protected classes.