Quote from: Vod on June 01, 2019, 07:42:38 PM

Quote from: akamit on June 01, 2019, 07:40:35 PM

I had assumed that the bot will go through 1 to 2601082 accounts and will check every post to find plagiarism.

That's half a billion posts. Shocked

Your msgID was 51311881 (51,311,881). That's "only" 51 million posts Tongue

Quote from: Quickseller on June 01, 2019, 07:46:58 PM

If Vod is correct about the number of posts, that would mean taking over 5,000 days assuming one post was checked per second.

You can download one page (up to 20 posts) per second. In ideal circumstances this would take just a month, in reality probably 2 months.

Quote from: suchmoon on June 01, 2019, 07:54:49 PM

Still a quadrillion pairs just to crosscheck post-to-post copies inside the forum so good luck to anyone attempting that.

I've looked at it before, finding duplicate posts wasn't that hard. The problem was the very large number of BS-posts like "good project". It's not plagiarism, it's useless, but there are far too many similar posts to manually report them all.

I think a combination of checksum with Linux commands "sort" and "uniq" can be used to find exact duplicates pretty fast. That solves the problem of a quadrillion checks as it'll use a well tested and very fast sorting algorithm. If someone can get me 51 million posts (excluding quotes and probably smileys), I'll give it a try.