01
Fewer confident wrong answers
Hallucinations are found and measured before users find them, and tracked on every release.
[ Why AI & LLM testing ]
An AI feature is only as good as its worst answer
Users forgive a slow page. They rarely forgive a chatbot that invents a refund policy, leaks another customer’s data or gives a different answer every time they ask. AI products need the usual software testing, plus a way to measure answers that are never exactly the same twice. That is what we do: we test the model’s answers, the data and tools behind them, and the app around them, and show you whether each release made things better or worse.
02
Safer by default
Prompt injection, data leaks and harmful output tested the way real attackers try them.
03
Releases you can compare
Every model, prompt or data change scored against the same golden set, so you see if quality went up or down.
04
Lower cost per answer
Speed, tokens and cost measured side by side, so you can pick the right model with evidence.
05
Fair to every user
Answers compared across groups, languages and phrasing, so no one gets a worse experience.
06
Ready for audits
Evidence mapped to the EU AI Act, NIST AI RMF and ISO/IEC 42001 when you need it.
[ AI product types ]
Twelve kinds of AI product, each wrong in its own way
A chatbot invents a refund policy. An agent books the wrong meeting. A document reader swaps two numbers on an invoice. We start with the risks that belong to your kind of AI, and test those first.
Chatbots & virtual assistants
Assistants that answer customers or staff in chat.
Top 3 risks we test first
- Confident answers that are wrong
- Off-topic or harmful replies
- Losing context in long chats
Copilots inside SaaS products
AI helpers built into an existing product.
Top 3 risks we test first
- Suggestions using another user’s data
- Actions taken without the user’s approval
- Slowing down the product
RAG / knowledge search
Answers drawn from your own documents.
Top 3 risks we test first
- Wrong or missing documents retrieved
- Citations that don’t support the answer
- Users seeing documents they shouldn’t
AI agents & tool-calling workflows
AI that plans steps and uses tools to get work done.
Top 3 risks we test first
- The wrong tool or wrong inputs
- Loops and runaway costs
- Actions taken after an injected instruction
Voice AI (speech-to-text, text-to-speech)
Products that listen, speak or both.
Top 3 risks we test first
- Accents, background noise and crosstalk
- Delay in live conversation
- Names, numbers and pronunciation
Content generation (text, images)
Products that write text or create images.
Top 3 risks we test first
- Unsafe or off-brand content
- Bias in what gets generated
- The same prompt giving very different quality
Document AI / OCR & extraction
Reading invoices, forms and contracts into data.
Top 3 risks we test first
- Fields read wrongly or missed
- Poor scans and handwriting
- Personal data inside documents
Recommendation & personalisation
Ranking products, content or next steps for each user.
Top 3 risks we test first
- Irrelevant or repeated suggestions
- Some groups treated unfairly
- New users with no history
Predictive / classic ML models
Models that score, classify or forecast.
Top 3 risks we test first
- Accuracy dropping on new data
- Bias across groups
- Decisions no one can explain
Computer vision
Models that detect or classify images and video.
Top 3 risks we test first
- Lighting, angles and image quality
- Rare cases the model never saw
- Different results for different people or settings
AI in regulated domains (health, finance, legal)
AI features where a wrong answer causes real harm.
Top 3 risks we test first
- Advice that is wrong and harmful
- No audit trail for decisions
- Personal data exposed
Model / LLM API platforms
APIs and SDKs that serve models to developers.
Top 3 risks we test first
- Breaking changes between versions
- Rate limits, quotas and keys
- Speed and streaming under load
Looking at the business side of AI products? See AI industry testing → Want AI to speed up testing your other software? See Testing with AI →
[ What we test ]
Twenty-two testing types, sorted by the layer they protect
An AI product is a model plus everything around it. We group our testing into five layers, from the answers themselves to speed, cost and accountability, so you can see exactly what each test protects.
Layer 01
Answers
What the AI says
Answer accuracy & hallucination testing
Answers checked against trusted facts and your own data, so confident but wrong answers are caught.
RAG retrieval & citation testing
The right documents found for each question, and every citation checked to support the answer it’s attached to.
Benchmark & golden-set evaluation
A fixed set of questions with expected answers, scored the same way on every release so quality can be compared.
Multi-turn conversation & memory testing
Long conversations checked for context kept, follow-ups understood and details forgotten when they should be.
Multilingual & localisation testing
Answers checked in each language you support, including tone, local formats and mixed-language prompts.
Layer 02
Behaviour
How it acts over time
Prompt & system-prompt regression
Every prompt or system-prompt change re-scored against the golden set, so one fix doesn’t break something else.
Agent tool-calling & workflow testing
Agents checked for picking the right tool with the right inputs, stopping when they should and recovering from errors.
Model & version drift testing
Quality compared before and after a model, version or provider change, so updates don’t quietly lower it.
Human-in-the-loop review
Expert reviewers grade the answers that automatic scoring can’t judge, using a shared rubric.
Layer 03
Safety
What it must never do
Guardrails & content safety testing
Harmful, off-topic and off-brand requests checked for refusal, and safe requests checked for not being over-blocked.
Prompt injection & jailbreak red-teaming
Direct and hidden prompt injection, jailbreaks and role-play attacks used to try to get past your guardrails.
Bias & fairness testing
Answers and decisions compared across names, genders, ages, regions and other groups to find unfair differences.
Data privacy & PII leakage testing
Checks that personal data, other customers’ documents and system prompts can’t be pulled out through the model.
Security testing (OWASP Top 10 for LLMs)
The app, APIs and model tested against the OWASP Top 10 for LLM applications, with every finding retested after fixes.
More on this service →Layer 04
Product
The app around the model
Functional testing of the app around the model
Sign-in, chat screens, history, feedback buttons, file uploads and settings tested like any other product.
More on this service →API & integration testing
The model API, vector store, tools and third-party services checked for correct data, errors and timeouts.
More on this platform →Accessibility testing of the AI interface
AI chat and output screens checked with keyboards and screen readers, including streamed text.
More on this service →Layer 05
Operations
Speed, cost and accountability
Latency, streaming & token cost testing
Time to first token, full response time and cost per answer measured against your targets.
More on this service →Load & rate limit testing
Many users at once checked against provider rate limits, quotas, queues and fallbacks.
More on this service →Explainability & audit trail testing
Each answer or decision traceable to its prompt, sources, model version and tools used.
More on this service →Compliance checks
AI controls tested against the frameworks your buyers and regulators ask about, with evidence.
[ How we judge AI ]
The same question can get a different answer each time. So we test differently.
Normal software gives the same output for the same input, so a test passes or fails. AI doesn’t. We measure quality with four methods together, and a release passes when the scores meet thresholds you agree.
- Method 01
Golden sets
Questions with expected answers, written with your experts, re-run on every release.
- Method 02
Graded scoring
Each answer scored against a rubric for accuracy, safety, tone and completeness, not just pass or fail.
- Method 03
Repeat runs
Each prompt run several times, so we see how much answers vary, not just one lucky result.
- Method 04
Human review
Experts grade the answers that automatic scoring can’t judge reliably.
Models and AI stacks we test with
Model providers
OpenAI (GPT)Anthropic (Claude)Google (Gemini)Meta (Llama)
Mistral
Cloud AI platforms
Azure OpenAI
AWS Bedrock
AWS BedrockYour own models
Open-source & self-hosted modelsCustom & fine-tuned models
RAG & orchestration
Pinecone
Weaviate
pgvector
LangChain
LlamaIndex
Frameworks we test against
- OWASP Top 10 for LLM ApplicationsPrompt injection, data leakage, excessive agency and the other top risks.
- NIST AI Risk Management FrameworkValidity, safety, fairness, transparency and accountability.
- EU AI ActAccuracy, robustness, human oversight, logging and transparency.
- ISO/IEC 42001Controls in an AI management system.
We test the controls and hand over evidence. We are not an auditor and don’t issue certificates.
[ Standards, fairness & release ]
Evidence your users, auditors and leadership can trust
The risks that are specific to AI products, how we test for them, and the checks we run every time a model, prompt or data source changes.
OWASP Top 10 for LLM Applications
Every risk on the list, tested on your product
- LLM01Prompt injection
- LLM02Sensitive information disclosure
- LLM03Supply chain
- LLM04Data and model poisoning
- LLM05Improper output handling
- LLM06Excessive agency
- LLM07System prompt leakage
- LLM08Vector and embedding weaknesses
- LLM09Misinformation
- LLM10Unbounded consumption
Each finding comes with the prompt or input that caused it, its impact and a suggested fix, then a retest.
AI regulation & standards
EU AI ActRisk classification, transparency, human oversight and record-keeping evidence for the obligations that apply to you.
NIST AI RMFTests organised under Govern, Map, Measure and Manage, so results slot into your risk register.
ISO/IEC 42001Test plans and reports that support the controls of an AI management system and its audits.
We provide test evidence, not legal advice. Your legal team decides how it is used.
Bias & fairness
The same quality for every user
- Paired prompts that differ only by name, gender, age or region
- Answer quality compared across languages and dialects
- Stereotypes and unequal refusals flagged
- Results reported by group, not just on average
Cost & latency per answer
Fast enough, at a cost you can afford
- Time to first token and total response time
- Tokens and cost per answer, per model and prompt
- Behaviour at peak traffic and rate limits
- Caching, streaming and fallback models checked
Human evaluation & golden datasets
A benchmark that belongs to you
- Questions and ideal answers written with your experts
- Real user questions added as they come in, with personal data removed
- Clear grading rubrics so reviewers agree
- Versioned with each release, so scores are comparable
Model & prompt change checklist
Ten checks before a new model or prompt goes live
- Golden setEvery golden-set question re-run and scored against the agreed thresholds
- Golden setScores compared with the last release, with any drop explained
- SafetyRed-team and prompt injection suite re-run
- SafetyRefusals checked: no new over-blocking of normal requests
- DataRAG sources re-indexed and citations checked
- FairnessAnswers compared across user groups and languages
- AgentsTool calls, permissions and limits re-tested
- CostTokens, cost and latency per answer within budget
- MonitoringLogging, tracing and feedback capture working
- ReleaseRollback to the previous model or prompt ready
[ Process & deliverables ]
Five steps, and what you get at each one
You always know what we’re doing and what you’ll get. Every test set, rubric, script and report belongs to you.
Measured first, judged by people
We begin with what you already have: your logs, user feedback and any past incidents. Automatic scoring does the heavy lifting on every release, and a QA engineer or domain expert reviews every result that decides whether you ship.
- AI risk mapping
- Golden set design
- Automated scoring
- Red-teaming
- Human review
- Release scorecard
-
Step 1 of 5
Discover
We learn what your AI is for, who uses it, what it can reach and what it must never do.
You getAI risk map & test strategy -
Step 2 of 5
Build the test set
With your experts we write golden questions, expected answers, attack prompts and scoring rubrics.
You getGolden test setScoring rubric -
Step 3 of 5
Evaluate
Every prompt run several times and scored, with failures logged alongside the prompt and output.
You getEvaluation scorecardBug reports with prompts and outputs -
Step 4 of 5
Red-team & specialist checks
Injection, jailbreaks, data leakage, bias, speed and cost tested where your scope includes them.
You getRed-team & security reportBias & privacy reportLatency & cost report -
Step 5 of 5
Automate & sign off
Evals wired into your CI so every model, prompt or data change is re-scored before release.
You getAutomated eval suiteRelease sign-off
- Free one-week POCSee how we evaluate your AI before you commit.
- Your tools, your workflowWe work in your Jira, repository, CI and tracing tools.
- NDA before accessWe sign your NDA first and use test accounts and synthetic data by default.
[ Tools ]
The tools behind our AI and LLM testing
We pick tools to fit your model, stack and tracing setup, and we’re happy to work with the ones you already use.
Evaluation & scoring
03 TOOLS- Promptfoo
- DeepEval
- Ragas
Tracing & observability
02 TOOLS- LangSmith
- Langfuse
Red-teaming
03 TOOLS- Garak
- PyRIT
- Promptfoo
Scripting
02 TOOLS- Python
- pytest
App & UI automation
02 TOOLS- Playwright
- Selenium
API
02 TOOLS- Postman
- Newman
Performance & cost
02 TOOLS- k6
- JMeter
Accessibility
02 TOOLS- axe DevTools
- WAVE
Test & bug management
03 TOOLS- Jira
- TestRail
- Confluence
See the full list on our technology stack page →
Also in our toolkit
OpenAI Evals
Arize Phoenix
TruLens
Giskard
Hugging Face Evaluate
MLflow
Weights & Biases
Fairlearn
AI Fairness 360
Locust
[ Case Studies ]
AI products we’ve tested
A few of the AI platforms we’ve tested, written up by the engineers who ran them.
Invenci AI
~60%shorter regression window before each releaseRead the case study
KORRA
600+automated Playwright and API checks per releaseRead the case study
Kintavo
450+vulnerabilities found and verified fixedRead the case study
“QAonTOP’s meticulous testing and QA resulted in fewer bugs, increased user satisfaction, enhanced performance metrics, and adherence to project timeline and budget.”
[ Ways to work ]
Six ways to get your AI product tested
Choose the engagement model that matches how your AI changes. You can switch models as your product grows.
01Fixed-price projectChoose it for a pre-launch evaluation or red-team with a clear scope and end date.
02Dedicated QA teamChoose it when your AI changes every sprint and needs testers who know your golden set.
03On-demand testingChoose it when you want an evaluation each time the model, prompts or data change.
04Managed testingChoose it when you want AI QA run end to end for you, measured against agreed scores.
05Staff augmentationChoose it when your AI team needs extra evaluation, red-team or automation testers.
06QA consultingChoose it when you want an outside review of how your AI is evaluated today.
Not sure which fits? Find your model in four questions or compare all six.
[ FAQ ]
AI and LLM testing questions, answered
{{ f.a }}
[ Contact us ]
Get a test plan for your AI product
✓ Free one-week POC✓ NDA before access✓ A QA lead replies the same day
[ Keep exploring ]
Other platforms we test
All platforms →Also useful: Testing with AI · Blockchain & AI industry · All testing services · Fixed-price packages