Web AppWebInternal product by ZoVintra

How we tested our own scanner

We hid known bugs in test projects and counted what our free scanners caught. Here are the results, and how we got them.

ZoVintra's own internal product, not a client project. The results come from test projects we built, not from customers. Real code is harder than test projects: see the numbers under the results.

How we tested our own scanner - Engineering Case Study Cover
37 of 58
Hidden bugs the code scanner caught in our test projects (64%)
45 of 60
Hidden problems the website scanner caught in our test sites (75%)
Under 0.2 s
Time the three slowest checks now take, down from 9 to 15 seconds
3 in 10
Findings on 11 real projects that were real problems (16 Oct). The rest needed a person to sort. Treat a scan as a first look.

About the Project

ZoVintra runs two free tools: an App Health Check for websites and a Code Review for code repositories. Both are our own software, built so a founder with an AI-built or half-finished app can get a first, repeatable answer to "is it safe to launch?".

A scanner is only worth trusting if its misses and false alarms are published, so we measured both. We hid known bugs ("planted defects") in test projects the rules had not been written against, and counted how many the scanner found. Recall is that share: 37 of 58 is 63.8%. A false alarm is safe code the scanner flags as a problem.

The planted-defect results were re-run on 15 November 2026. The real-repository figures date from 16 October 2026 and were not re-run.

Requirements & Objectives

  • Find real defects in JavaScript and TypeScript code and in live websites.
  • Keep false alarms low enough that a founder is not sent chasing noise.
  • Stay safe to run on other people's input: no crashes, no hangs, no unsafe requests.
  • Raise detection by 250%. This could not be met, and not for lack of effort: the scanners already found 36% to 57% of planted defects at the start, and a 250% rise on that would be above 100%.

The Engineered Solution

1Code review without running the code

Reads a repository archive without executing it, using pattern rules, a syntax-tree pass and tracing of request data across function calls.

2Website health check

Crawls a site and checks metadata, security headers, links, accessibility basics, email records and AI-crawler access.

3Test projects written separately

Separate authors wrote test code and test sites with planted defects and safe look-alikes, and were told not to read the scanner. The benchmark tracks 40 kinds of code defect and 40 kinds of website defect; one set covers only part of them (30 of 40 code kinds, 34 of 40 website kinds).

4Findings in priority order

Findings are listed most serious first and carry a confidence label. Reports are private by link, and emailing one needs a verified address.

Engineering Process

  • Wrote the rules.
  • Had test projects written separately, by authors who were told not to read the scanner.
  • Scored the scanner, fixed general causes rather than single cases, then scored again.
  • Relabelled any set used for tuning as a dev set. Dev sets score 98.4% (code) and 96.6% (website) and are not counted as evidence.
  • Wrote the scanners with AI coding assistants under human direction. The test projects were also written by language models.

Problems we hit and how we solved them

ProblemFix
Three checks could be made to hang by crafted input. In a re-run of the old code they took 12.7, 15.0 and 9.5 seconds.Rewritten so the time grows in a straight line with input size. Each now finishes in under 0.2 seconds.
About twenty rules could waste time rescanning very long lines.Long lines are cut before the rules read them, and open-ended matches got explicit limits.
A comment-stripping step re-read each line over and over, so one hostile line was very slow.A single pass with a limit, and tracing now shares the scan's time budget.
A package file with a very large number of dependencies took 99 seconds to review.Caps and faster lookups.
A damaged upload could crash the request or leave it waiting forever.The error is caught and the request ends cleanly.
A slow site could run past the platform time limit and get no report.The scan now shortens its own crawl instead of being cut off.
A rule change that raised the headline score hid a regression.Found by scoring each change separately, then undone.

What it cannot do, and how far to trust the numbers

  • Real code is much harder than planted bugs. On 11 fresh real repositories (16 October 2026, not re-run since), an estimated 28.6% of 687 findings were real problems, and about 14% (roughly 1 in 7) once import-related findings are left out. On an earlier list of 15 repositories it was about 10%. Our own results document calls this far from accurate. Expect findings that need a human to sort.
  • The most serious defects are found least often. For defects rated critical or high, recall was 64.7% on the fresh code set and 42.9% on the fresh website set.
  • Severity is often off. The scanner rated severity exactly right for 70.3% of the code defects it found and 73.3% of the website defects.
  • "Held-out" is not a sealed exam. Some of these test sets were scored more than once as the scanner changed, and an earlier version of our test tool printed per-case detail before we switched to totals only. Treat them as held-out in intent.
  • Small sets: 58 to 60 planted defects each, so differences of a few points are noise. The older 62.1% for the fresh code set against today's 63.8% is one defect.
  • Only JavaScript and TypeScript get code-level analysis. Other languages get a few simple line patterns, and the benchmarks cover JavaScript and TypeScript only.
  • Very long lines are only partly scanned.
  • Business logic, who-can-see-whose-data, and behaviour under load are out of reach for a static tool.
  • The test projects were written by language models, so their idea of a realistic bug is not independent. A set labelled by real engineers is still to do.
  • A free scan does not replace a human review.

Production Tech Stack

Layer / ComponentProduction Technologies
Runtime
Node.js
App
Next.js 16React 19
Code analysis
Babel parser
Storage
Supabase (REST interface)
Email
Nodemailer
Testing
node --test

What we measured

Measured on 15 November 2026: 63.8% of planted code defects and 75.0% of planted website defects found in fresh test sets, up from 36.2% and 56.7% at the start of the cycle (16 October). The 98.4% and 96.6% scores on the sets we tuned on are not evidence. On real repositories, most findings were false alarms when we last measured. We have not measured how many founders used the tools or what happened after. There is no client and no testimonial.

TRY IT YOURSELF

Run the scanners on your own app

Both tools are free. Expect some false alarms, and treat a scan as a first look, not a review.