We run an AI-powered SMS bot and an AI-graded call training system for a private school
network. Both systems have extensive test suites (400+ test files) and a RAG-based knowledge
base that drives bot behavior
We need someone to own the full quality cycle:
- Review production conversations to find bad bot responses
- Create test files (JSON) that reproduce the issue
- Fix the knowledge base content (Markdown files, WRONG/RIGHT examples)
- Run test suites to verify fixes and catch regressions
- Maintain and expand the knowledge base as our program evolves
This is NOT a software engineering role. The infrastructure is built. You're operating it — editing
JSON, writing Markdown, running CLI scripts, reading test reports.
What You'll Work With
- JSON test files (you'll write and edit these daily)
- Markdown knowledge base documents
- Terminal commands (copy-paste and run scripts)
- Git (commit, push, basic branching)
- VS Code or similar editor
Requirements
- Native-level English fluency (non-negotiable). You'll be writing realistic SMS conversations
between parents and our school. The language has to sound like a real person texting, not a
corporate chatbot. You'll also be writing precise pass/fail evaluation criteria.
- Comfortable editing JSON and Markdown in an IDE
- Can run commands in a terminal without hand-holding
- Extreme attention to detail
- Ability to learn a complex domain quickly (education, state government programs, etc)
Nice-to-Haves
- Experience writing test cases or QA documentation
- Experience with chatbot QA, conversational AI testing, or LLM evaluation
Hours & Ramp-Up
- 30 hrs/week to start, ramping up as needed. US time zones required.
- The first 1-2 weeks will be focused on learning our domain — how our school works, how the
state voucher program works, compliance rules, etc. We have extensive internal documentation.
