Messing around with all this has also made me realize that the amount of robot traffic happening on websites like this one must be through the roof. I haven't looked at any stats, but it's gotta be going bananas.
It's not storing all this, some of it's doing in real time, and I can only imagine that the amount of electricity and bandwidth that's being take up just by robots crawling websites nowadays is crazy. At some point, it's going to be more than us.
I can confirm this. In one example that I manage (a public gitea instance from older days) I see 99.999% of the requests coming from the usual AI suspects (Amazon, Alibaba, Microsoft, Google etc.).
Noticed it because disk was getting full from several dozens of GB of webserver logs (for something that gets like 1-2 human visits a day), the motherfuckers don't even throttle their crawlers, they're firing off requests in the dozens per second, every single one of them behaves like a berserk and of course they do not respect any robots.txt at all.
Given how often you saw normal search engines in your logs and how often you now see AI crawlers, I can only guess that it's like a death-race of who downloads the whole internet first to train their shitty eliza pro+ LLMs with.
And since their hunger for new data is so immense, they have no time to be kind to webservers. Fuck webpages anyway, in their vision people will not visit webpages anymore in the future, they will just ask some shitshippiddi for everything.
Needed some time to collect all the evil cloud networks they come from, but once I had that I traffic-shaped them down to 10kb/s which translates to about one request per minute in total. Even if this only slightly increases their cost of aggro-crawling the web, it's a start..
Yes, having it farm haikus out of this thread that has 35,000 pages and seeing that it picked ones across several years of posting made me realize this.
Everyone's talking about the amount of electricity and water, etc. that it uses. But I hadn't even started to think about the second order effects of that. Because it's causing everything else, as you say, to get gobbled up over and over and over again. So that was a light bulb that went on for me after I performed this little experiment. I suppose I should be thankful that that little light bulb above my head actually had any power to use
And I think I share at least some of your sentiment about AI, I laughed several times reading your post.
It seems that computers and the internet and technology is eating everything all the time. We talk about this. Bitcoin is the internet of money. Email was one of the first ones that made it dawn on everyone and so on and so on.
I've been thinking about it for a while and I believe the technology that AI directly replaces... that it is the descendant of is the Ouija board.
AI is the digital Ouija board. I mean, think about it. The information that comes from a Ouija board is harnessed from a strange sort of community group think type of thing. It averages out the subconscious, perhaps, voices of a group of people. Stronger voices pushing the needle a little further and then the consensus of voices also seeming to take control.
So we end up with some horrible summation, some mean, some mix of ingredients that should never have gone together in the first place. Since much of what it trains on is propaganda, then much of its answers are just the same milk toast bullshit that we've been hearing all our lives dressed up into bullet lists.
And who knows? Could be evil spirits might even be able to direct the needle. Why not?
👻