Waiting for a worker
Preparing crawler
- Job picked up from queue
- Fetching robots.txt
- Parsing sitemap.xml
- Rendering page in headless Chromium
- Extracting Schema.org JSON-LD
- Probing /.well-known/ discovery files
- Walking shadow DOM trees
- AI-visibility heuristics (anti-bot, lang, feeds)
- Browser-UX audit (aria, tap targets, CLS)
- Compiling grade and breakdown
Median scan time: 20–60 seconds. Sites behind aggressive CDNs (Cloudflare Bot Fight, DataDome) can take up to two minutes while we wait out the challenge.