Using YOLO for CAPTCHA recognition — 17 phases from demo-ware to production

The prototype took an afternoon. Production took fourteen months. Somewhere between those two numbers lives every lesson I'd now give a younger version of me about adversarial ML, motor control, and the difference between it works and it keeps working.

V-Scrape's stealth layer had to pass fifteen different CAPTCHA vendors, not just reCAPTCHA v2. The demo I shipped to a friend was a 90-line script using a pretrained YOLOv8 model and a linear mouse move — it solved hCaptcha image grids at roughly 94% accuracy and felt like magic. Six weeks later, every major vendor had quietly blocked me. The model wasn't wrong. Everything around it was.

This article is the 17-phase rebuild: what I got wrong, what the adversary actually watches for, and what the production pipeline looks like now. It's long. Code and numbers are real. The photos of me at 3am are not included.

01 · the prototypeThe demo that lied to me

The first version was embarrassingly short. Load a fine-tuned YOLOv8n checkpoint, run inference on the grid tiles, pick the cells whose bounding boxes matched the prompt, click them. The model was good — 94.1% top-1 on my held-out set. And it worked, for about forty-eight hours.

// v1 — this gets you blocked in a week
async function solveGrid(page, prompt) {
  const tiles = await page.$$('.task-grid .task-image');
  const preds = await yolo.infer(tiles, prompt);
  for (const t of preds.filter(p => p.score > 0.6)) {
    await t.element.click();    // instant, centre, zero jitter
  }
  await page.click('.verify-btn');
}

I thought the problem was vision. It wasn't. The clicks were perfect centre-of-box, 4ms between them, no mouse path, same user agent, same viewport. A reasonable heuristic on the vendor's side — "humans don't do that" — flagged the session in seconds. The bounding boxes were right; I was sending them as a robot with a spreadsheet.

02 · what's actually being measuredNine signals, not one

I dumped obfuscated client-side JS from three vendors and spent a weekend deobfuscating. The vision problem is maybe 20% of the score. The rest is behavioural and environmental:

* relative, eyeballed from observed score deltas. Not published anywhere by the vendors, obviously.

The CAPTCHA isn't the test. The motor control leading up to it is the test. Vision is a tiebreaker.

03 · seven-layer motor controlModelling a hand, not a cursor

The breakthrough came from an unrelated paper — Harris & Wolpert, 1998, "Signal-dependent noise determines motor planning". Human arm movement isn't a smooth curve; it's a noisy optimisation over a minimum-jerk trajectory, corrupted by signal-dependent motor noise that increases with intended velocity. The curve you see is already the filtered output of several lower-level controllers fighting each other.

So I built seven layers, bottom-up, each owning one property the adversary watches:

The dispatcher is the boring glue but it's where 60% of the wins came from. Too few events and the path looks teleported between waypoints. Too many and you fire at an impossible rate. Matching the browser's own event loop cadence is what sells it.

// layer 03 + 04 — the curve and the noise
function minJerk(p0, p1, t) {
  // Hogan 1984: 5th-order poly, zero vel+acc at endpoints
  const s = 10*t**3 - 15*t**4 + 6*t**5;
  return { x: p0.x + (p1.x-p0.x)*s, y: p0.y + (p1.y-p0.y)*s };
}

function sdn(velocity, sigmaBase = 0.08) {
  // signal-dependent noise: σ ∝ v
  const σ = sigmaBase * (1 + 0.5*velocity);
  return { dx: randn()*σ, dy: randn()*σ };
}

04 · the vision partQuad-signal fusion beats any single model

The 94% prototype model was fragile to every adversarial perturbation hCaptcha shipped after the rebuild: tile-edge noise, semantic overlap (a "bus" that's half a truck), occluded objects, stylised illustrations. Instead of training a bigger model, I stacked four weaker, fast ones and voted:

1. YOLOv8n fine-tuned on 140k vendor-pulled images — the workhorse.
2. CLIP-based semantic match — embed the prompt, embed each tile, cosine-similarity rank.
3. Caption-then-match — BLIP caption the tile, Levenshtein against prompt synonyms.
4. External fallback — cheap human-solver API, only when the ensemble is low-confidence.

The four signals get fused with learned weights per vendor (hCaptcha weights CLIP higher; reCAPTCHA weights YOLO higher — they use different image distributions). Ensemble accuracy is 97.8% on the held-out set and the cost is bounded because the external fallback fires on only ~6% of grids.

05 · what it costsNumbers that survived production

Compare the demo: 94.1% accuracy but effectively 0% sustained — blocked within 48 hours of any sustained use. The delta between a model and a product is motor control, telemetry, an ensemble, and a fallback. Four layers on top of the checkpoint that people usually ship alone.

06 · audio challengesVosk + Whisper, picked per vendor

Accessibility audio challenges are a different beast and worth their own treatment. Short version: Vosk for clean narration (fast, offline, 160ms p50), Whisper-small for noisy / accented clips (slower but handles distortion). Routing between them is a simple SNR threshold on the first 500ms — cheap, and it gets the right model ~92% of the time.

07 · things that brokeMistakes I'd save you from

• Using real mouse hardware timings from a Logitech dataset — turns out trackpad vs mouse users have distinguishable jitter signatures. I was always "mouse". Randomising it now.
• Pre-generating trajectories — caching Bézier control points between attempts. The adversary can cluster sessions by trajectory fingerprint. Generate fresh every time.
• One ensemble weight set for all vendors — each vendor's image distribution is different enough that per-vendor tuning is free accuracy.
• Trusting headless: false — Playwright's chromium still leaks four identifiers even when headed. Patchright patches those; plain Playwright doesn't.
• Not measuring the session — I was optimising per-CAPTCHA. The adversary is scoring you over a whole browsing session. A perfect solve after 200ms of suspicious mouse behaviour still fails.

08 · what it looks like nowSystem diagram

09 · the takeawayThe prototype is the easy part

If you've ever looked at a demo repo and felt like you were about to ship — this is a letter from the other side. The model was never the hard part. The hard part is every millisecond around the model: how your cursor moves, how long you look before clicking, how your fallback chooses when to fire, how your ensemble weights get tuned per vendor. A checkpoint is a tool; a pipeline is a product.

A checkpoint is a tool. A pipeline is a product. Don't confuse the two.

FlowScrape — sorry, V-Scrape — uses this pipeline for every auth-walled site it visits. It is, per internal telemetry, my least-loved and most-necessary subsystem. I will never write another line of CAPTCHA code if I can help it. And yet here we are.

Questions, corrections, or vendor-intel trades: hello [at] sherifbutt.dev.