GenAI Engineer – Test Agent / CI Integration
Turing • Remote
Company
Turing
Hourly rate
$12 – $35/hr
Location
Remote
Listed
135d ago
- Min. degree:
- Master's
- Field:
- Software Engineering
Not ready to apply?
Join our talent pool and let labs like Turing find you first as you build up experience.
Set up your profile →About this Role
GenAI Engineer – Test Agent / CI Integration
Experience: 4–8 Years
Employment Type: Full-Time
Job Description:
We are looking for a highly skilled GenAI Engineer – Test Agent / CI Integration to build and scale intelligent AI-powered testing systems for next-generation applications. The ideal candidate will work on automated test-agent frameworks, synthetic data generation, evaluation harnesses, and CI/CD-integrated AI testing pipelines.
This role requires strong expertise in Python backend development, LLM evaluation frameworks, retrieval-grounded testing systems, and modern DevOps practices.
What you'll do
- AI Test Agent Development
- Design and develop autonomous AI-driven test agents for validating GenAI and LLM-powered applications
- Build systems for:
- Synthetic data generation
- Test-case synthesis
- Scenario generation
- Adversarial and edge-case testing
- Develop reusable evaluation harnesses for benchmarking model quality, accuracy, safety, and reliability
What you need
- Technical Skills
- Strong proficiency in Python
- Experience with:
- Pytest
- CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI, etc.)
- REST APIs & backend development
- Hands-on experience with:
- LLM evaluation frameworks (DeepEval, Ragas, LangSmith, custom evaluators)
- RAG systems and retrieval pipelines
- Synthetic dataset generation
- Prompt engineering and evaluation strategies AI/ML & GenAI Expertise
- Strong understanding of:
- Large Language Models (LLMs)
- Agentic systems
- AI evaluation methodologies
- Context grounding and knowledge retrieval
- Familiarity with vector databases, embeddings, and knowledge graphs
Nice to have
- Experience working with knowledge graphs or graph databases
- Exposure to LangChain, LlamaIndex, or similar orchestration frameworks
- Familiarity with Kubernetes, Docker, and cloud platforms (AWS/GCP/Azure)
- Experience in enterprise-scale AI platform engineering
Context-Aware Test Generation
• Integrate test agents with BLK’s knowledge/context graph for retrieval-grounded testing
• Enable contextual test generation using RAG pipelines and graph-based retrieval systems
• Ensure generated tests align with enterprise knowledge sources and real-world workflows
CI/CD & Automation
• Integrate AI test agents into CI/CD pipelines as first-class pipeline jobs
• Automate regression testing, evaluation runs, and quality scoring during deployments
• Build scalable validation workflows for continuous model monitoring and release gating
Evaluation Frameworks & Quality Engineering
• Work with LLM evaluation frameworks such as:
• DeepEval
• Ragas
• Custom evaluation frameworks
• Develop automated scoring mechanisms for:
• Hallucination detection
• Faithfulness
• Relevance
• Toxicity
• Response quality
• Integrate with pytest and existing QA ecosystems
Backend & Infrastructure
• Build and maintain Python backend services powering evaluation workflows
• Optimize distributed evaluation execution for scalability and performance
• Collaborate with platform, MLOps, and DevOps teams for production deployment
DevOps & Automation
• Experience integrating AI workflows into CI/CD environments
• Understanding of automated quality gates and testing orchestration
Important Note
This is a niche requirement and not a regular GenAI developer role. We are specifically looking for candidates with experience in:
• AI validation/testing
• QE automation
• Python backend
• CI/CD integration
• LLM evaluation frameworks
• RAG and retrieval-grounded systems
Skills & Categories
Frequently Asked Questions
How do I apply for the GenAI Engineer – Test Agent / CI Integration role at Turing? +
Use the Apply button on this page. It opens Turing's own application form, so you apply directly with them.
What does this Turing role pay? +
Turing lists $12 – $35/hr. Confirm the exact rate and terms in their application.
Is this role remote? +
The listing is remote. Check Turing's posting for any country or time zone requirements.
Do I need a degree? +
The listing asks for a Master's or equivalent. Turing makes the final call on what counts.
Interview Prep
This listing calls for this tool directly. Prep for the technical screen:
Related Roles
Browse All Full-Time Jobs at Turing
Turing works with leading AI labs on model training and evaluation, and hires specialists in coding, languages, and domain expertise for that work.
View All Turing Roles →