Mike's Notes
This gem from Roberto Di Cosmo, Director of Software Heritage, explains much of the web's history and the current swarm of AI crawlers.
"The "good enough" trap: Why the web keeps breaking itself
Every day, swarms of AI crawlers storm the web like "digital locusts," repeatedly re-downloading entire websites just to discover what changed. It's wildly inefficient, drives up massive infrastructure costs, and is actively forcing the open web to lock its doors behind bot filters and API paywalls.
The most frustrating part? We solved this decades ago. In his new series, Roberto Di Cosmo, Director of Software Heritage, traces how our digital infrastructure keeps falling into the "good enough" trap: local decisions that work fine for individual actors, but compound into an aggregate disaster for everyone else.
Back in 1998, long before Google dominated search and decades before LLMs arrived, Di Cosmo co-authored an IETF Internet-Draft proposing a simple, push-based alternative called the Remote Update Protocol (RUP). A web server knows when its data changes, so by broadcasting updates directly, it spares crawlers the endless need to ask, “Anything new?"
Despite being technically sound and independently re-invented by others, the protocol quietly expired. It lacked a dedicated institution to champion, maintain, and deploy it. Fast-forward to today, and everyone pays the price for relying on "good enough" brute-force scraping.
The mission of Software Heritage is to address this directly by archiving public source code once so the world doesn't have to collect it over and over.
Di Cosmo’s series isn't just a historical autopsy—it’s a warning. Without collective, sustained investment in shared digital infrastructure, the open web will continue to disappear into proprietary silos." - Software Heritage
Resources
- https://www.softwareheritage.org/
- https://www.dicosmo.org/good-enough/
- https://en.wikipedia.org/wiki/Roberto_Di_Cosmo
- https://annex.softwareheritage.org/public/press/2024/2024-09-27-ORCWG@LeMondeInformatique.pdf
- https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/
- https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/
- https://www.dataprovenance.org/consent-in-crisis
- https://arxiv.org/abs/2407.14933
- https://datatracker.ietf.org/doc/draft-rfced-exp-cosmo/
- https://www.dicosmo.org/RUP/
- https://www.dicosmo.org/Piege/cybersnare/
- https://www.dicosmo.org/HoldUp/
- https://www.dicosmo.org/RUP/draft-RUP-01.txt
- https://www.dicosmo.org/Inria40/
- https://datatracker.ietf.org/doc/draft-ietf-webi-rup-reqs/
- https://www.dicosmo.org/RUP/RupJava/RobotServer.html
- https://www.rfc-editor.org/rfc/rfc9309.html
- https://www.sitemaps.org/protocol.html
- https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
- https://www.dicosmo.org/Inria40/
References
- Consent in Crisis: The Rapid Decline of the AI Data Commons, Data Provenance Initiative, July 2024 (arXiv:2407.14933).
- R. Di Cosmo and P. E. Martínez López, “Distributed Robots: a Technology for Fast Web Indexing”, written January 1998.
- Hijacking the World: the dark side of Microsoft, October 1998.
Repository
- Home > Ajabbi Research > Library > Subscriptions > Software Heritage
- Home > Handbook >
Last Updated
28/08/2026
Good enough is not good enough
An alumnus of the Scuola Normale Superiore di Pisa, with a PhD in Computer Science from the University of Pisa, Roberto Di Cosmo was associate professor for almost a decade at Ecole Normale Supérieure in Paris. In 1999, he became a Computer Science full professor at University Paris Diderot, where he was head of doctoral studies for Computer Science from 2004 to 2009. President of the board of trustees and scientific advisory board of the IMDEA Software institute and chair of the Software chapter of the National Committee for Open Science in France, he is currently on leave at Inria.
His research activity spans theoretical computing, functional programming, parallel and distributed programming, the semantics of programming languages, type systems, rewriting and linear logic, and, more recently, the new scientific problems posed by the general adoption of Free Software, with a particular focus on static analysis of large software collections. He has published over 20 international journals articles and 50 international conference articles.
In 2008, he has created and coordinated the european research project Mancoosi, that had a budget of 4.4Me and brought together 10 partners to improve the quality of package-based open source software systems.
Following the evolution of our society under the impact of IT with great interest, he is a long term Free Software advocate, contributing to its adoption since 1998 with the best-seller Hijacking the world, seminars, articles and software. He created in October 2007 the Free Software thematic group of Systematic, that helped fund over 50 Open Source research and development collaborative projects for a consolidated budget of over 200Me. From 2010 to 2018, he was director of IRILL, a research structure dedicated to Free and Open Source Software quality.
He created in 2015, and now directs Software Heritage, an initiative to build the universal archive of all the source code publicly available, in partnership with UNESCO.
...
A series about why the open web is collected so badly, why that has survived thirty years of everyone knowing better, what it is now costing all of us, and what is needed to go from “good enough” to “doing things right”.
What this is
The series runs on LinkedIn. This page is where it lives afterwards: every post in order, and under each one the sources for every claim it makes — dates, references, links, commit identifiers, intended to make it possible for you to check the facts directly.
It is also here for a more practical reason. What follows is partly about things quietly disappearing from the web, and about what it costs to depend on a single platform. Leaving the only copy inside somebody else's feed would have made the point rather too well.
The posts
- 01 The wasteful sentence 4 Aug
- 02 The alternative was written down 11 Aug
- 03 How good enough wins 18 Aug
- next: The mechanism: an externality, not a villain
The series
POST 01 The wasteful sentence 4 August 2026
This way of collecting data is very wasteful.
That is how an Akamai specialist described the way many AI crawlers gather material from the web.[1] The report around the quotation gives the concrete mechanism: some botnets revisit an entire site every day merely to discover what changed, transferring the same material again and again. This is not new: search engines have been doing it for decades, in the very same wasteful way, but the scale of "scraping" is now such that what we have known as the "open web" is now closing down, an API at a time[2], a "bot filter" at a time[3], effectively reducing the global value for everybody, while increasing the waste of resources.
The striking part is not that the method is wasteful. Everybody running a sizeable website knows that by now. The striking part is that we speak as though this were a law of nature.
It is not. We knew how to avoid most of this waste in the 1990s, and it turns out I actively tried to push forward a concrete solution, when the problem was still manageable. We'll get back to this.
For nearly three decades, I have repeatedly encountered the same pattern: a system that works well enough for every actor taken separately, while imposing a growing cost on everybody taken together. Each local decision is reasonable. Their aggregate result is damaging. Since nobody owns the aggregate problem, the better solution remains nobody's job.
It is time to call out this state of affairs, and act upon it... again: I personally tried many times, and now I can build on that experience. The first step is to clearly understand what is happening and why: which actor does what and why, who gains, who loses. In the coming weeks I will reconstruct one instance of the pattern from dated documents. It begins with web crawlers and ends with the institutions we build—or fail to build—for shared digital infrastructure.
Let me point at the destination from the outset: I direct Software Heritage, a non-profit organisation that collects public source code once so that others do not all have to collect it again. This is my latest effort to contribute to systematically address this long standing problem.
It is a problem that starts with two simple words: good enough.
POST 02 The alternative was written down 11 August 2026
A record card for the Internet-Draft "Distributed Robots: a Technology for Fast Web Indexing", by Di Cosmo and Martinez Lopez, recorded 4 March 1998, the document printing "EXPIRES SEPT 1998" with no day given. Beneath the card, a caption reads: "What today's AI crawlers do, answered on paper in 1998."
One of my first encounters with the "good enough" curse came in the mid-nineties, during the ferocious battle to control the rapidly expanding cyberspace.
WebCrawler, AltaVista, Alexa, Yahoo… Google, the eventual winner, had not even been born yet.
What stunned me was that all of them did essentially the same thing: download the Web again and again to discover what had changed.
Sounds familiar? It should: it is exactly what AI crawlers do now.
The Web was much smaller then, but bandwidth, compute and storage were scarcer too, so the waste was already real.
And there was an obvious alternative.
In January 1998, with Pablo E. Martínez López — aka Fidel — I wrote Distributed Robots: a Technology for Fast Web Indexing. It entered the IETF record as an individual Internet-Draft on 4 March 1998,[4] carrying the wonderfully explicit line: "EXPIRES SEPT 1998."
The idea was elementary, familiar to any systems programmer: interrupt-driven beats busy-wait.
A web server knows, cheaply, when its files change. Let it say so, and let crawlers fetch only what is new — instead of asking every server, over and over, "Anything changed?"
In May 1998 an IETF Area Director sent us a generous, detailed critique. By then I had moved on: that March I had put Piège dans le Cyberespace online — CyberSnare in English — which went viral, started thirty years of work on free and open source software, and became a book with Dominique Nora.[5]
It took me seven months to answer point by point. I first asked what he thought of the revised structure.
The answer was an automatic out-of-office message. The thread ended there.
We produced the revised draft anyway, dated 20 April 1999, incorporating the review.[6] Fidel sent it to the RFC Editor. No answer came.
But the problem had not disappeared. In 2001 the IETF's own WEBI group independently produced Requirements for a Resource Update Protocol.
That expired too.[7]
Years later, looking at my web server logs, I found that in 2009 Googlebot had fetched one unchanged page twenty-four times.[8]
Students of mine implemented the Remote Update Protocol twice, in Java and OCaml.[9] The technical idea was not the hard part.
And this is the point: a better technical solution does not deploy itself.
Someone must maintain it, persuade others to adopt it, integrate it into existing systems, and keep pushing after the prototype works. Advocacy on the side of a full-time job is not an institution.
The alternative was written down almost thirty years ago, then rediscovered independently inside the IETF.
It was not defeated by a better idea.
It was not shown to be technically impossible.
It simply never acquired enough organised support to become infrastructure.
Meanwhile, the "good enough" solution kept scaling.
Why?
POST 03 How good enough wins 18 August 2026
A French sentence, "le systeme actuel fonctionne assez bien", above its footnote, "Good enough, comme on dit chez nos amis anglosaxons." From an article revised into 2011, written up from a talk at Inria in December 2007.
So why did it survive?
Not because anyone defended it. Because it was made liveable, one patch at a time.
Let's be precise about what "good enough" means: it does not mean bad. A good enough system does not do the right thing, but it does get the thing done. It is a hack, a wooden leg that gets you across the room… And because it gets you across the room, you stop looking for a better leg.
Downloading the whole web to find out what changed was never sensible. It was patched until it was bearable.
In 1994 Martijn Koster proposed robots.txt, so a site could say: not this, not here. It worked well enough that nobody standardised it until 2022.[10]
In 2006 came sitemaps, so a site could say: here is what I have, and here is when it last changed.
Read the specification closely. The freshness field is "a hint and not a command", and crawlers may ignore it.[11] So a site can say exactly what changed, and you are free not to believe it and download everything anyway. Which is what happened.
Then the crawlers got cleverer about where to spend. Google's own documentation says it plainly: "URLs that are more popular on the Internet tend to be crawled more often."[12] And if your server struggles, the crawler backs off.
Add it up and you have a system that works. Big sites get visited often, small sites rarely, a struggling server gets a break. Nobody is delighted, but nobody is ruined.
That is what good enough looks like from the inside: not a catastrophe, but a tolerable arrangement everyone has adapted to.
And then the second thing happens, which is worse: it becomes normal.
You are almost certainly reading this on a QWERTY keyboard. Nobody chose it this morning. Everybody knows it, every keyboard has it, and whoever switches first pays the whole cost alone. So it stays: not because anyone re-decided it, but because it is installed.
That is where web crawling ended up. Installed. It always worked like this.
In December 2007 I gave a talk at Inria's fortieth anniversary in Lille, written up as an article I revised into 2011. I wrote that the current system was working « assez bien ».[13] French had no phrase for what I meant, so I borrowed one, in a footnote: « Good enough, comme on dit chez nos amis anglosaxons. »
I was not complaining. I was describing. It was accurate.
Lock-in is harmless when the stakes are low. Nobody is much hurt by a keyboard layout.
Repeated crawling is not. It spends bandwidth, electricity and machine time on a planetary scale to fetch what has not changed.
Thirty years on from robots.txt, the patches are not holding, and everyone can see it.
So the question is not why nobody noticed the waste. Everybody noticed.
The question is why a cost that large stays invisible to the people who could act on it, until it is too late to act easily.
That is not a technical question. Economists have a name for it, and it is the whole point that we need to delve into.
References
Every source for every claim, numbered in the order the posts cite them. Each one carries a way back to the place it was cited from.
[1] post 01 Darius Rafieyan, “Like digital locusts, OpenAI and Anthropic AI bots cause havoc and raise costs for websites”, Business Insider, 19 September 2024. Archived copy: annex.softwareheritage.org. The speaker is David Senecal, principal product architect for fraud and abuse at Akamai. The sentence in full: “This way of collecting data is very wasteful,” he said, “but until the mindset on data sharing changes and a more evolved and mature way to share data exists, scraping will remain the status quo.” The sentence that follows in the post — botnets crawling a whole site daily — is the reporter’s prose summarising him, not a quotation, and is given as reported speech for that reason. ↩ back to the text
[2] post 01 GitHub tightened its limits on unauthenticated access on 8 May 2025, citing “an increase in scraping activity targeting our API”. Smaller operators went further: SourceHut placed a proof-of-work challenge in front of its web interface after a week-long crawler incident in March 2025, and GNOME, KDE, Fedora and Codeberg have each done some version of the same. ↩ back to the text
[3] post 01 Longpre et al., Consent in Crisis: The Rapid Decline of the AI Data Commons, Data Provenance Initiative, July 2024 (arXiv:2407.14933) — an audit of ~14,000 web domains behind three widely used AI training corpora, finding a sharp rise in crawl restrictions across a single year. ↩ back to the text
[4] post 02 R. Di Cosmo and P. E. Martínez López, “Distributed Robots: a Technology for Fast Web Indexing”, written January 1998. Record: datatracker.ietf.org. Full text, with everything that followed it: dicosmo.org/RUP/. The document prints “EXPIRES SEPT 1998” and gives no day, so none is claimed here. ↩ back to the text
[5] post 02 « Piège dans le Cyberespace », and the book that came out of it. The essay appeared in Multimédium (Canada) on 17 March 1998. It is online free and complete, in five languages, at dicosmo.org/Piege/cybersnare/ — French, English as CyberSnare, German as Falle im Cyberspace, Italian as Trappola nel Cyberspazio, Spanish as Trampa en el Cyberespacio — with a Chinese version at PiegeCN.html. The book is Le Hold-up planétaire : la face cachée de Microsoft, with Dominique Nora, Calmann-Lévy 1998, ISBN 2-7021-2923-4; Kirk McElhearn’s English translation, Hijacking the World: the dark side of Microsoft, followed in October 1998. When the publisher stopped reprinting in July 2006 the two authors recovered their rights and put the book out under Creative Commons Attribution-NonCommercial-NoDerivs. dicosmo.org/HoldUp/ carries the French, English and Spanish texts in full — the Spanish, El Asalto Planetario, never having had a print edition at all. ↩ back to the text
[6] post 02 The review, the resubmission, and the silence. On 10 May 1998 an IETF Applications Area Director reviewed the draft in the IESG and asked that the objects be defined as MIME types and distinguished from the FIND working group’s CIP and SOIF and from the W3C’s RDF. The point-by-point answer went back on 8 December 1998, accepting the MIME and SOIF encapsulation and asking one question before resubmitting; the reply, ten seconds later, was an automatic out-of-office, and no further correspondence followed. draft-RUP-01.txt, The Remote Update Protocol (RUP). Part I: RUP Architecture, is dated 20 April 1999 and does what the review asked. It was submitted: Martínez López sent it to the RFC Editor on 16 April 1999, with the file attached, describing it as “an update for the internet-draft <draft-rfced-exp-cosmo-00.txt>”. No reply ever came — the same address had answered the 1998 submission twice within hours — and six days later he wrote asking whether to chase them or wait a little longer. No IETF record of draft-RUP-01 exists. Nobody behaved badly: a volunteer reviewer was away for a fortnight, the answer had taken seven months, and after that carrying it forward was nobody’s job. The correspondence is in the author’s own archive. ↩ back to the text
[7] post 02 Twenty-four Googlebot fetches of one unchanged page across 2009, twelve of them answered 304 Not Modified. The log is reproduced in full on page 9 of the Inria anniversary article, which is now online with the talk it came from: dicosmo.org/Inria40/. ↩ back to the text
[8] post 02 “Requirements for a Resource Update Protocol”, draft-ietf-webi-rup-reqs, IETF WEBI (Web Intermediaries) working group. The first revision is M. Hamilton (JANET Web Cache Service) and I. Cooper (Equinix), 22 February 2001; Dawn Li and Mike Dahlin joined later; the last revision is dated 4 March 2002, and it expired. Its consumers are caching proxies and surrogates, not crawlers — a different protocol for a different audience. What is shared is the diagnosis, in its own abstract’s words: such a protocol is needed where “periodic revalidation is unacceptable in terms of performance and/or cache consistency”. There is no evidence the two efforts knew of each other, and none is claimed: the point is that the same conclusion was reached twice, independently, and expired twice. ↩ back to the text
[9] post 02 Both implementations were student projects at Paris 7 — Travaux d’Étude et de Recherche on the subject “indexation rapide du Web”, which Roberto set for two years running. The RUP 1.0 Java servlet, by Yerom-David Bromberg, is at dicosmo.org/RUP/RupJava/. OCamlRup — a RUP client, a generic RUP server and an Apache CGI server, with rupinfo.txt parsing and robots.txt integration — was written in 2001 by Samuel Lasry and Xavier Patourel and is archived at ocamlrup-0.1.tar.gz. Copyright remains the authors’; no licence was ever attached and none is asserted — it is published as an archival record, with attribution. ↩ back to the text
[10] post 03 robots.txt. The Robots Exclusion Protocol was, in the words of the RFC that eventually specified it, “originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed”. It became an IETF standards-track document only in September 2022: RFC 9309, by Koster with G. Illyes, H. Zeller and L. Sassman. Twenty-eight years of running on a convention. ↩ back to the text
[11] post 03 Sitemaps, and the word “hint”. The Sitemap protocol 0.9 says of changefreq: “Please note that the value of this tag is considered a hint and not a command.” It goes on: crawlers “may crawl pages marked ‘hourly’ less frequently” than stated. The protocol lets a site describe its own freshness; it obliges no one to act on the description. That asymmetry is the whole of this post. ↩ back to the text
[12] post 03 Crawlers spending where it pays. Google’s own documentation on managing crawl budget: “URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems.” And on backing off: if a site “slows down … or responds with server errors … the limit goes down and Google crawls less.” Adaptation, not repair — the polling continues, more politely. ↩ back to the text
[13] post 03 From a talk given at Inria’s fortieth anniversary, Lille, December 2007, written up as an article revised into 2011: « le système actuel fonctionne assez bien », with the footnote “Good enough, comme on dit chez nos amis anglosaxons.” Both are now online at dicosmo.org/Inria40/ — the slides as the audience saw them, and the article, which was never finished; the footnote is on page 10. The article is cited as talk December 2007, article revised into 2011; no span of years is computed from it anywhere in this series. ↩ back to the text
Corrections
Every correction made after publication is logged here, with its date and what changed. Nothing is silently edited. If you find an error, the fastest way to have it fixed is to tell me.
— nothing logged yet —
Roberto Di Cosmo · dicosmo.org
Text licensed CC BY 4.0. Quoted material remains its authors'.
This page loads nothing from any other host: no fonts, no scripts, no analytics.
