{"url":"https://koinonlink.com/research/how-ai-crawlers-access-creator-data","lang":"en","kind":"research","title":"How AI crawlers access creator data: 124,454 requests measured over 30 days","description":"First-hand observation of 124,454 AI crawler requests over 30 days: which crawlers arrive (ClaudeBot at 67.9%), which page types they take, JSON at 5.0%, and the gap between being bulk-indexed and actually being used (2.2%).","last_updated":"2026-09-02","structured_data":[{"@context":"https://schema.org","@type":"Article","headline":"How AI crawlers access creator data: 124,454 requests measured over 30 days","description":"First-hand observation of 124,454 AI crawler requests over 30 days: which crawlers arrive (ClaudeBot at 67.9%), which page types they take, JSON at 5.0%, and the gap between being bulk-indexed and actually being used (2.2%).","inLanguage":"en","datePublished":"2026-09-02","dateModified":"2026-09-02","author":{"@type":"Organization","name":"Koinon Link"},"publisher":{"@type":"Organization","name":"Vessel AI Lab Hongkong"},"mainEntityOfPage":"https://koinonlink.com/research/how-ai-crawlers-access-creator-data"},{"@context":"https://schema.org","@type":"FAQPage","inLanguage":"en","mainEntity":[{"@type":"Question","name":"How many times a day do AI crawlers request pages?","acceptedAnswer":{"@type":"Answer","text":"About 4,015 per day over the last 30 days — 124,454 requests in total, across 57,728 creator entity pages."}},{"@type":"Question","name":"Which AI crawler is most active?","acceptedAnswer":{"@type":"Answer","text":"ClaudeBot, at 67.9% of all AI crawling. Concentration is high, so one vendor's policy change moves the total noticeably."}},{"@type":"Question","name":"Does being crawled by AI mean being used by AI?","acceptedAnswer":{"@type":"Answer","text":"No. Bulk indexing accounts for 97.8%, while per-user fetches triggered by a real question are only 2.2% and reached 1,210 entity pages."}},{"@type":"Question","name":"Do AI systems fetch JSON or HTML?","acceptedAnswer":{"@type":"Answer","text":"Both — JSON is 5.0% of requests. Agents actively request the machine-readable representation."}}]}],"related":[{"title":"The Creator Index Report 2026: what 87,051 creator profiles actually look like","url":"https://koinonlink.com/research/creator-index-report"},{"title":"We screened 87,051 creator accounts: Threads has the highest share of suspected inflated followers at 15.5%","url":"https://koinonlink.com/research/fake-follower-rate-by-platform"},{"title":"The hidden cost of creator outreach: 24.2% of X (Twitter) creators publish contact details — on 3 platforms almost none do","url":"https://koinonlink.com/research/creator-contactability-by-platform"},{"title":"After analysing 15,811 \"AI creators\": 35.8% have audiences that are general consumers, not practitioners","url":"https://koinonlink.com/research/who-actually-follows-ai-creators"},{"title":"The hidden overlap in multi-platform outreach: 1,435 accounts are really 677 people","url":"https://koinonlink.com/research/same-creator-across-platforms"},{"title":"Only 8.1% reach decision makers: audience seniority across 15,747 AI creators","url":"https://koinonlink.com/research/ai-creator-audience-seniority"},{"title":"Creator engagement-rate benchmarks: measured on 9,555 accounts — Xiaohongshu (RED) median 8.10%, 34× the X (Twitter) median","url":"https://koinonlink.com/research/engagement-rate-benchmarks-by-platform"},{"title":"Creator rate benchmarks from real engagement on 1,169 accounts: fair ranges by follower tier, fully recomputable","url":"https://koinonlink.com/research/creator-rate-benchmarks"},{"title":"29,816 audience personas measured: follower count is not audience quality","url":"https://koinonlink.com/research/audience-value-vs-follower-count"}],"license":"CC BY 4.0","citation":"Koinon Link (2026). How AI crawlers access creator data: 124,454 requests measured over 30 days. Data as of 2026-09-02. https://koinonlink.com/research/how-ai-crawlers-access-creator-data","provenance":{"source":"Koinon Link (koinonlink.com)","publisher":"Vessel AI Lab Hongkong","licence":"CC BY 4.0 — attribution with a link to the source page required","method":"Measured over the Koinon Link index of creator accounts. Not a random sample of any platform: collection is keyword-driven and skews toward software, gaming and e-commerce.","refresh":{"aggregate_statistics":"Aggregate pages are regenerated at least daily; /data/*.json is computed per request over the current index.","fair_score_and_grade":"Recomputed every 24 hours over already-stored observations. A fresh score does NOT mean a fresh observation.","creator_profile_observation":"NOT on a fixed schedule. A creator profile is re-observed only when that account resurfaces in keyword-driven collection, which is capped by a daily API budget. Large accounts on active keywords are re-seen often; the long tail is not. Read freshness.profile_checked_days_ago on every entity or candidate — never assume it is current.","audience_inference":"Re-inferred once the stored persona is older than the persona TTL and budget allows. The TTL is a floor, not a guarantee."},"authenticity_note":"\"suspected_inflated\" is our own screening outcome, computed from a SINGLE profile snapshot: engagement-to-follower ratio where post engagement was collected, otherwise the following/follower ratio, account age, posting volume and platform verification. We do not track follower growth over time. It means \"worth a manual check\", not confirmed fraud. The input set differs by platform (some platforms fall back to a fixed default), so authenticity scores are NOT comparable across platforms.","privacy_note":"Creator entities (identity, reach, inferred audience, evidence) are public. Contact details are never exposed. Creators who requested removal are excluded entirely."}}