Blog
The latest on technical enforcement of crawling preferences.
Scrapling and Crawlee: How Open-Source Scraping Tools Get Detected
A technical analysis of Scrapling and Crawlee, two popular open-source scraping frameworks, examining their anti-detection features and the behavioral signals that content-layer defenses can exploit.
What Defensive Coordination Actually Looks Like in 2026
A year after Poison Fountain launched anonymously, no AI lab has acknowledged it, no publisher has named it, and no one has measured it. Compared against Anubis, AIPREF, and Cloudflare Pay Per Crawl, the contrast shows what real defensive coordination requires.
AI Crawler Compliance, Mid-2026: The Blocked-but-Cited Trap
Publishers that blocked AI crawlers via robots.txt lost 23.1% of monthly traffic on average, and got only weakly correlated reductions in AI citation. The 2026 data inverts the case for blocking-as-defense.
Anubis at One Year: What Production Operators Are Actually Reporting
A year of public Anubis deployments yields concrete operator numbers, a Codeberg cautionary tale, and a project trajectory shift toward layered defenses. What the data says about proof-of-work anti-scraping.
AI Poisoning Threat Models: Backdoors, RAG, and Supply Chain
Backdoor attacks, model degradation, and RAG poisoning explained. Technical analysis of who can attack, defense costs, and power dynamics in AI training data.
How Much Does It Cost to Scrape the Web at Scale?
Bulk residential proxy pricing, Web Unlocker tiers, and headless browser farms put real per-page scraping costs at $0.001-$0.005, not the widely-quoted $0.01. AI training-data licensing deals show why the economics keep working for scrapers.
AIPREF After Toronto: What the IETF Decided in April
The IETF AIPREF working group reached consensus on AI training scope at its April 2026 Toronto interim, made progress on AI search wording, and deferred the contested AI input category. Status update on the standard.
Where AI Training Data Actually Comes From in 2026
A canonical reference for the six-layer AI training-data stack: Common Crawl, lab crawlers, curated open datasets, licensed feeds, contractor pipelines, and synthetic data. With the comprehensive licensing-deal table, current numbers, and what the labs do not disclose.
Defensive Data Poisoning: Ethics, Risks, and Alternatives
Analyzing ethical tradeoffs of defensive data poisoning: proportionality, collateral damage, and safer alternatives like proof-of-work and AIPREF standards.
Data Poisoning FAQ: Technical, Legal, and Policy Answers
Answers to common questions about data poisoning, web crawling, robots.txt, AIPREF, legal status, and enforcement mechanisms for AI training defense.
What Is Data Poisoning in Machine Learning?
Data poisoning manipulates AI training data to alter model behavior. Learn how defensive tools like Nightshade protect content from unauthorized AI training.
How AI Scraping Infrastructure Works: Proxies, Evasion, and Scale
Inside the technical infrastructure AI companies use to scrape the web: residential proxy networks, fingerprint emulation, CAPTCHA solving, and why traditional defenses fail.
Who Owns the Residential Proxy Industry That Feeds AI Scraping in 2026
A follow-the-ownership map of the residential proxy networks behind AI scraping: the Lithuanian and Israeli conglomerates that sell consumer privacy VPNs and the scraping infrastructure that drains publishers, plus why this structure breaks per-crawler defense.
Who Pays the Proof-of-Work Tax: The Accessibility Cost of Anti-Scraping Walls
Proof-of-work anti-scraping like Anubis is not a flat fee. It is a regressive tax: at difficulty 5, about 2 seconds on a flagship laptop, up to 2 minutes on an old phone, and a hard wall for non-JS and screen-reader users, while AI scrapers pay near zero.
The AI Crawler Compliance Crisis: Who Plays by the Rules?
AI crawler robots.txt compliance dropped from 96.7% to 70% in one year. Analysis of which crawlers comply, what it costs publishers, and what comes next.
Understanding AIPREF: The IETF Standard for AI Content Preferences
AIPREF extends robots.txt with standardized vocabulary for AI training preferences. How the IETF standard works, its syntax, and what it means for publishers.
Publisher Defenses Against AI Scraping: Cost Imposition vs Poisoning
Comparing defense strategies against AI scraping: proof-of-work systems impose costs, data poisoning degrades value. Who pays and what works for publishers.
How AI Data Laundering Uses Non-Profit Research to Shield Commercial Models
AI data laundering routes web-scraped content through academic and non-profit datasets so commercial labs can train on material they could not legally collect themselves. The mechanism, the LAION case, the 2025 Hamburg ruling, and why robots.txt cannot reach it.
Why VENOM Exists: From robots.txt to AI Data Enforcement
When robots.txt fails, enforcement mechanisms emerge. VENOM analyzes data poisoning, proof-of-work, and technical countermeasures for AI training governance.