Ethical crawling of academic websites

why on Earth would you prevent bots from accessing published papers founded by public research on public grants

Because bandwidth and hosting isn't free. They are mostly trying to prevent these works from being incorporated into AI chatbots, which do not typically cite their sources; this is a hard requirement for most academic and scientific organizations.

Bing and Google are let in because they are primarily search engines which drive organic traffic to their websites (if we ignore the fact that Google and Microsoft are also training LLMs, but I digress).

It's worth your time to contact these organizations directly about your pursuit. They might ask you some questions, and then allow your bot's traffic, provided you supply a user agent string that their servers can pick up on. They might also come back and say "pay us," in which case, you'll need to fork over some money; how feasible this is for you is debatable. They also might just blindly say "no" because AI.

But I think it's worth your time to contact these organizations, because they might have internal controls and processes to selectively allow this kind of traffic provided it meets certain criteria.

+1 for contacting these organizations directly.

As a side note, any technique that's used to bypass Cloudflare/captchas/whatever, could also be used by bad actors.

You haven't provided any details on what's being used to stop you other than Cloudflare/captchas. As noble as your intentions may be, bypassing those protections would also make it possible for to bypass as well. They're not afraid of your requests specifically, they're afraid of being DDOS'd. You avoid being flagged as the latter by getting whitelisted.

I'm sending one HTTP request per second [...] And I'm getting blocked.

Does the response specifically indicate that the blocking is due to excessive requests? If not then this sounds like a red herring.

It's equally possible that one request per second like clockwork is an indicator of a bot and gets blocked. Or a systematic traversal of the site is an indicator of a bot and gets blocked. There are any number of indicators their system could be looking for, and short of contacting them to discuss it's likely a matter of more trial and error on your part. (And accepting that this reverse-engineering effort may simply not succeed.)

Alternatively, if any given service "blocks" requests from the start then that would imply the pattern of behavior wouldn't have anything to do with it at all.

So far you're mostly debating on the merits of what you're trying to accomplish. Your position in that debate is reasonable, but the debate itself misses the mark in a couple of ways.

One... Because what purpose does it serve? What is accomplished if members of this community agree with you that this information should be freely accessible? Does that solve the problem? Probably not.

Two... The information is freely available. You can access it any time you want. What you're demanding is not free access to information, what you're demanding is that any organization which has this information should be required to expose a free and public API with no restrictions to access it.

The information is free. The API is not. That's the difference.

So contact the organizations. I've had to deal with captchas/Cloudflare when I open a private window in Firefox, and I'm not doing any automated traffic. The servers notice your request looks weird for some reason and they want to avoid being DDOS'd. Telling us why your traffic is legitimate and should be allowed through isn't going to change the way those servers analyze your requests.

You might be able to find a magic header (User-Agent maybe?) or fake cookies to get automated requests working, but that's just going to be a game of whack a mole. The selling point of Cloudflare is to keep bad actors from hitting your infrastructure, and since regular users won't be bothered by an occasional captcha or automated challenge, they're incentivized to err on caution.

Your safest, most reliable solution, is to contact these organizations and get white-listed.

What they are doing right now is ensuring the monopoly of Microsoft and Google over information retrieval systems, and indirectly feeding the privacy leakage around that.

This. So much this.

Believe me, I'm not trying to play devil's advocate here. I just think that's what's going on. Microsoft and Google have enough leverage that these organizations must let those lampreys latch onto their jugular and engorge themselves, otherwise you get excluded from the web.

Academic and scientific institutions are well aware of the lampreys on their neck. If you approach them in a way that distances yourself from the lampreys, and you can prove good intentions, and (gasp) that you could use AI for the benefit of society, you might not need to negotiate or pay. You might just need to be honest and upfront about your intentions, and get them to understand how helping you helps them or society at large.

You should really read this Wikipedia entry on Aaron Swartz!: https://en.wikipedia.org/wiki/AaronSwartz

You should also be aware that "GAFAM" do pay very large licensing fees to the scientific journal overlords for journal data scraped for AI models.

I always say it's best to be a Good Internet Citizen™ and stop scraping data once you hit a captcha or other type of barrier. At that point you know they don't want your bot crawling the site and should contact them to see if they have an API you can use.

Of course the web was created as an open information sharing platform... but that was back when only DARPA was using it. Shortly after it went public all sorts of authentication/authorizations were shoe-horned on top of the protocol. If you're seeing reverse proxies blocking your traffic on 880 sites that are completely public, then it's partly a by-product of people/organizations similar to you who are creating way too much traffic trying to gather information for AI training. Scientific journals, in particular, have aggressive gate-keepers who do make money providing access to them. (while anyone can get access at the University library, the University does pay a fee to offer those journals for free) I won't get into the ethics of that... it's just the reality. Btw, depending on what you are training your model for, there may already be a free version of that model available to you.

Gotcha. I'd suggest not doing what you are doing, but you don't have to take the advice. I'm certainly not a lawyer or anything like that... but if you're doing this as a business you really should consult with the legal division.

On a side note, I can ping Scihub with DOI directly, and get the full PDFs for free. I mean… scientific journals are already a mafia of leeches who don't pay the researchers, don't pay the peer-reviews, but instead charge the researchers who submit articles, and charge again the university libraries for access. So I truly wonder what they are for and why we need them in the first place, if they are not even gonna pay for bandwidth at the end.

Right now, I already ping the Wayback machine/Internet archive when I'm getting 403. I won't be ransomed for academic knowledge, thank you very much.

on a side note, there are some services that allow webmasters to charge scraping bots per hit... they integrate with reverse proxies like Cloudflare... and Cloudflare itself is introducing same: https://www.cloudflare.com/paypercrawl-signup/ This allows the bot to control how much it is willing to pay for the information, and the webmaster to set the rate they are willing to take per bot/per hit.

I run a couple of cheesy little web sites with niche information useful to a few people. I publish them with a community spirit on the theory that "information wants to be free".

I've had to block scrapers because of the myriad low-rent AI scrapers that have been slamming the server I rent to publish them. It breaks my heart: I put my little bit of info out there for people to use.

So, your project and mine are falling victim to this tragedy of the commons. It stinks.

OR, use an AI model that's trained to control your computer's mouse and keyboard and act just as a human would. Think a "robot" type framework, but powered by AI. (I'm sure this is already happening...) You'd probably rotate between many sites just as a human might. Any method that is not detected as bot traffic is a little nicer for webmasters as they can still be paid for ads served. (just don't block those ads)

if you really want to by-pass these captchas/blocks, there are methods available... I just don't think it's necessarily morally correct to do it. (and on the edge of being illegal because you're bypassing a protection mechanism) It's a moving target, but you not only need to sort of customize your requests and browser(if you decide to use one), but also send your traffic through residential proxies. (many of which are built using hijacked IOT devices or routers... you're starting down a dark road at that point...) Remember that Google and Microsoft obey robots.txt files, and do pay fees for scraping data used for AI training. (though there were some lawsuits regarding things like thumbnail images, cached pages, etc..., that led them there... they did depart from the "Do no evil" mindset at some point, and had to be forced into compliance)

Not what I was asking. That leaves me with ~880 organizations to contact. What was the web for again ?

Does the response specifically indicate that the blocking is due to excessive requests? If not then this sounds like a red herring.

Here is an example of a spoofing sequence for https://journals.sagepub.com/doi/abs/10.1177/1046496408328703. Note that the link is followed from a blog post reference, not from methodically crawling the journal itself, and fails with 403 error from the first hit, when trying to get the page headers (not even the page content).

https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 403 {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14.6; rv:129.0) Gecko/20100101 Firefox/129.0', 'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
https://journals.sagepub.com/doi/abs/10.1177/1046496408328703/ 403 {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14.6; rv:129.0) Gecko/20100101 Firefox/129.0', 'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 403 {'User-Agent': 'Virtual Secretary', 'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
https://journals.sagepub.com/doi/abs/10.1177/1046496408328703/ 403 {'User-Agent': 'Virtual Secretary', 'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 403 {'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
https://journals.sagepub.com/doi/abs/10.1177/1046496408328703/ 403 {'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
http://journals.sagepub.com/doi/abs/10.1177/1046496408328703 403 {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14.6; rv:129.0) Gecko/20100101 Firefox/129.0', 'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
http://journals.sagepub.com/doi/abs/10.1177/1046496408328703/ 403 {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14.6; rv:129.0) Gecko/20100101 Firefox/129.0', 'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
http://journals.sagepub.com/doi/abs/10.1177/1046496408328703 403 {'User-Agent': 'Virtual Secretary', 'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
http://journals.sagepub.com/doi/abs/10.1177/1046496408328703/ 403 {'User-Agent': 'Virtual Secretary', 'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
http://journals.sagepub.com/doi/abs/10.1177/1046496408328703 403 {'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
http://journals.sagepub.com/doi/abs/10.1177/1046496408328703/ 403 {'Connection': 'keep-alive', 'Accept': 'application/pdf,text/html,application/xhtml+xml,text/xml,application/xml;q=0.9,/;q=0.8'}
Headers: https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 -> https://web.archive.org/web/20240806191614/https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 : 403
Content: https://web.archive.org/web/20240806191614/https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 -> https://web.archive.org/web/20240806191614/https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 : 403
Page content error No page found
Content: https://web.archive.org/web/20240806191614/https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 -> https://web.archive.org/web/20240806191614/https://journals.sagepub.com/doi/abs/10.1177/1046496408328703 : 403
couldn't download https://web.archive.org/web/20240806191614/https://journals.sagepub.com/doi/abs/10.1177/1046496408328703

When that happens, my code falls back to the Wayback machine, which is blocked all the same.

Let me rephrase :

I'm sending one HTTP request per second with an headless browser that doesn't even load fonts/CSS/JS crap (not even images), from my personal computer on a single IP, to academic journals and doi.org.

And I'm getting blocked.

Back in 2023, I could send 60 requests/s (or as many as my crappy ISP would allow) without issue, and be done with crawling in a matter of hours. That could qualify as DDOS. I don't think what I'm doing now is comparable.

Define "bad actor". It's public knowledge. You either abuse the infrastructure or you don't. Don't publish if you don't want people to use it.

Bing and Google have been training AI models since before 2013, Word2Vec is a shallow neural network training vector embeddings for tokens, published in 2013. Nobody took offense of language models milling their websites when it was only for indexation, now that GenAI has abused the system, suddently all AI is bad AI. It's a bit late to panic.

I don't have the resources to either pay or negociate with all involved publishers across the whole DOI ecosystem. What they are doing right now is ensuring the monopoly of Microsoft and Google over information retrieval systems, and indirectly feeding the privacy leakage around that. Not to mention, good luck finding any relevant technical info with Google these days, it second-guesses the search topic so aggressively that it has become useless for anything but finding open restaurants nearby.