Senior Quality Engineer (AI) - E-Learning
About Truelogic
At Truelogic we are a leading provider of nearshore staff augmentation services headquartered in New York.
For over two decades, we’ve been delivering top-tier technology solutions to companies of all sizes, from innovative startups to industry leaders, helping them achieve their digital transformation goals.
Our team of 600+ highly skilled tech professionals, based in Latin America, drives digital disruption by partnering with U.S. companies on their most impactful projects.
Whether collaborating with Fortune 500 giants or scaling startups, we deliver results that make a difference.
Our client is committed to building innovative and scalable solutions that drive efficiency and impact.
We foster a culture of continuous learning, collaboration, and proactive problem‑solving.
If you're looking for an environment where you can grow and make a difference, we want to hear from you!
Job Summary
As we advance our AI development efforts, we recognize the need for more than a traditional QA engineer.
We are seeking a GenAI Quality Coach—a strategic and hands‑on role that blends test innovation, prompt effectiveness analysis, and user feedback insights.
This individual will help shape and evolve our QA practices specifically for GenAI systems.
They will partner closely with developers, product owners, and SMEs to ensure we are building robust, safe, and high‑quality AI features.
Responsibilities
- Define GenAI‑specific QA strategy
- Develop a QA framework tailored to GenAI systems and workflows
- Design tests for:
- Prompt behavior across varied inputs and user tasks
- Hallucination detection
- Factual consistency and groundedness
- Blend manual and automated test design for deterministic and stochastic outputs
- Collaborate with teams to obtain or create sample data with clear target outputs
- Own end‑to‑end test strategy and execution for GenAI‑powered features (Test Plan Ownership)
- Ensure coverage across:
- Diverse prompt phrasing, user intents, and failure modes
- Multiple GenAI features (e.g., summarization, generation, classification)
- High‑risk, edge‑case, and compliance‑driven scenarios
- Lead the design and implementation of prompt and model evaluation protocols:
- Alignment between user input and intended behavior
- Output fluency, tone, and coherence
- Clarity, coverage, and relevance of responses
- Use golden datasets and benchmark prompts to establish evaluation baselines
- Design and manage SME‑driven review workflows (Human‑in‑the‑Loop evaluation)
- Facilitate structured reviews focused on:
- Correctness/accuracy based on metrics and SME feedback
- Capturing edge‑case failures
- Reporting and KPIs
- Define and track QA effectiveness using metrics such as:
- Pass rate for high‑risk use‑cases
- HITL reviewer agreement rates and flagging critical issues
- Use‑case specific measures of “quality”
- Deliver clear, actionable dashboards and reports to leadership on AI quality, safety, and readiness
Qualifications And Job Requirements
You’re a great fit if you:
- Are excited by the complexities and challenges of GenAI testing
- Think like a product owner, act like a tester, and communicate like a coach
- Thrive in ambiguity and enjoy shaping new standards
- Are passionate about safe, responsible AI development
What we offer
- 100% Remote Work – all you need is a laptop and reliable internet
- Highly competitive USD pay – market‑leading compensation beyond typical offers
- Paid Time Off – policies that allow you to unwind and recharge
- Work with autonomy – focus on results, not the clock
- Work with top American companies – grow your expertise on innovative, high‑impact projects
Why You’ll Like Working Here
- A culture that values you – well‑being, work‑life balance, and dynamic teams
- Diverse, global network – connect with over 600 professionals in 25+ countries
- Team with skilled professionals – senior talent ensuring you work with the best in the field
Apply now!
#J-18808-Ljbffr