- Client
- KORRA
- Industry
- AI Platform · Enterprise knowledge
- Engagement
- Ongoing · 4+ years
- QA team
- 2 full-stack QA engineers
- Platforms
- Desktop Web · Mobile Web · API · Database
- Manual Testing
- Automation Testing
- AI Algorithm Result Analysis
- Accessibility Testing
- AI Performance Validation
Jira
ConfluencePostman
PostgreSQL
GitHub
Playwright
Python
BrowserStack
LLMs
JMeter
Axe DevTools
WAVE
About the project
Korra is a document-centric, visually powered AI knowledge platform. Enterprises load their technical documentation — equipment manuals, wiring schematics, Excel part lists, recorded training — and Korra answers questions with the text, image or video moment that supports each answer. Its users are manufacturers and industrial teams, where a wrong answer means downtime, not an awkward chatbot reply.
The product has grown considerably during our engagement: from AI search across documents and video, to multimodal answers with diagrams embedded inline, part matching across PLM data, maintenance-ticket analysis and fully on-prem, air-gapped deployments. Every one of those capabilities added new ways for an answer to be wrong.
QAonTOP works as Korra’s embedded QA team: two full-stack engineers inside the sprint who own the regression suite and validate AI output with the same rigour as the functional product.
What made this hard
Testing an AI knowledge platform is two jobs at once. The software around the model has to behave like any enterprise SaaS product — and the answers coming out of the model have to be correct, cited and repeatable.
- CHALLENGE 01
“Correct” is not pass or fail
An answer can read well and still cite the wrong page, quote a figure from an outdated manual revision, or miss the one diagram that matters. Conventional test cases had no way to express that.
- CHALLENGE 02
Every document type breaks differently
Scanned PDFs, multi-sheet Excel files, annotated schematics and long videos each take a different ingestion path. A parser change for one could quietly degrade another.
- CHALLENGE 03
Model changes arrive without a code diff
Retrieval tuning, prompt edits and LLM version upgrades can shift answers overnight, with nothing in the pull request for a tester to review.
- CHALLENGE 04
Enterprise users, enterprise expectations
Technicians on tablets and phones, accessibility requirements in customer procurement, and document libraries large enough that query speed becomes a feature.
How QAonTOP tested KORRA
-
01
Map the product and its risks
Audited the release process, bug flow and existing coverage, then wrote a test strategy that split quality into two tracks: platform behaviour (ingestion, permissions, admin, player, API) and answer quality (retrieval, grounding, citation).
Tools Jira · Confluence
-
02
Build a golden question set
With Korra’s team, built benchmark libraries per document type: questions with a known correct answer, the source that should be cited and the passage that should be highlighted. Every release is scored against it and compared with the last.
Tools Python · LLMs · PostgreSQL
-
03
Automate the regression backbone
Playwright suites cover upload and ingestion, search, deep links into the document and video player, workspace permissions and the editor. Postman checks API contracts; PostgreSQL queries confirm that indexing and metadata land correctly.
Tools Playwright · Postman · PostgreSQL · GitHub
-
04
Validate AI output on every release
Python scripts run the benchmark through the API and flag answers whose citation, figure or source revision has drifted. LLM-assisted comparison speeds up triage, but a QA engineer makes the final call on every flagged answer.
Tools Python · LLMs
-
05
Accessibility, devices and load
Axe DevTools and WAVE audits against WCAG 2.1 AA, manual keyboard and screen-reader passes, BrowserStack runs on real phones and tablets, and JMeter load tests on query and ingestion endpoints ahead of large customer rollouts.
Tools Axe DevTools · WAVE · BrowserStack · JMeter
The engagement in numbers
Continuous engagement, every release since 2022
Test cases maintained across platform and AI tracks
Automated Playwright and API checks per release
Benchmark questions scored against known sources
Less manual regression effort per release
Accessibility baseline audited and maintained
Before and after the engagement
| Area | Before QAonTOP | With QAonTOP |
|---|---|---|
| Regression | Three to four days of manual checks before each release, with areas skipped when time ran short. | An overnight Playwright and API suite, followed by a focused one-day manual pass on what actually changed. |
| AI answer quality | Checked ad hoc by whoever built the change, on whichever questions came to mind. | Every release scored against a fixed benchmark — answer, cited source and highlighted passage — with drift reported release over release. |
| Document ingestion | Parser regressions discovered by customers when a new file type or layout failed silently. | A reference library of PDFs, spreadsheets, schematics and videos re-ingested and verified in the database before sign-off. |
| Accessibility | No formal audit; issues surfaced during customer procurement reviews. | Axe and WAVE audits plus manual screen-reader checks, with WCAG 2.1 AA findings tracked to closure in Jira. |
| Performance | Response times unknown beyond a handful of concurrent users. | JMeter baselines for query, answer generation and ingestion, re-run before major rollouts. |
| Release decisions | Go / no-go based on open ticket count and instinct. | A documented Definition of Done, severity-based bug flow and a release report the product owner signs off. |
What we covered
Manual Testing
Exploratory and scripted testing of new features, the editor, workspaces and the document and video player.
Automation Testing
Playwright end-to-end and Postman API regression, versioned in GitHub alongside the product code.
AI Algorithm Result Analysis
Benchmark scoring of retrieval, grounding and citations, with drift analysis between model and prompt versions.
Accessibility Testing
WCAG 2.1 AA audits with Axe DevTools and WAVE, plus keyboard and screen-reader passes.
AI Performance Validation
JMeter load tests and response-time tracking for query, answer generation and document ingestion.