Quote from: LoyceV on September 18, 2018, 07:35:59 PM

Quote from: Initscri on September 18, 2018, 06:31:59 PM

P.S: If any mods/admins aren't ok with me scraping the site, by all means let me know. I'd obviously write the bot/script in such a way that it doesn't slam the server & only send a certain amount of requests per second/minute (more or less like a Google bot). I know other users have written similar bots/scraping tools, so I thought it'd be ok. But if not, just let me know

I've recently started scraping recent. It saves the first unedited version of the post in raw HTML, excluding quotes. Your post for example looks like this:

Code:

Initscri
186520
45883661
Other / Meta / "Multiple Accounts" / Copy-pasta detection scripts/bots

Hey all,

I've been planning to write a few scripts relating to BitcoinTalk. It's been on my "developer bucket list" to write something to detect users who have multiple accounts. In order to accomplish this, and have a reliable list, I'd have to determine some logic in order to base this.

I have a few things in mind:

Index/scrape posts &:

For multiple account detection:

- Look for same address usage between posts (BTC, ETH, etc)
- Look for same account usage between posts (telegram, skype, etc)
- [other ideas here]

For copy-pasta detection:
- write a script to determine copy-pasta from accounts by matching the text of posts to similar text of other sites in order to return a probability percentage of the user copy/pasting (including src for manual analysis)
- [other ideas here]

Results would be posted here for mods to look at (if need be), or just to keep a record of such a connection. I'd also probably link to results in this topic

I wanted to post this thread in advance to see if anyone else had any other logic / ideas in mind for these scripts/bots? This will solely be when I have the time to create this (which won't be for a couple of weeks), so I thought I'd post this well in advance.

Thanks!

The first line is your Username, then userID, post number, some raw headers, and the last line is the post itself.
In compressed format, it takes about 10 MB per day. Instead of scraping the same data again, I could easily send it to you, and a few day's worth of data should be enough for you to start testing. If interested, let me know.

Not a bad idea. I'll take that into account. I probably won't be starting for a little while, but I'll send you a message in a little while if I need it.

I like the idea of scraping recent & just grabbing raw HTML to compare.

In order to minimize requests but allow multiple filtering scripts to parse the data separately, I'll probably end up scraping recent with 1 bot, caching that for a set time period (sort of like a mirror), and then using multiple other scripts to parse the data on the caching server/mirror.

What I might end up doing is creating the server that stores the cache & keeping it closed source. But I'll release the scripts that parse the data / determining abusers as open source. These scripts would connect back to the mirror server/site instead of BitcoinTalk. That way if others wish to volunteer by using some computational power to run those scripts, they can do so and it allows for others to contribute code without slamming BitcoinTalk with a massive amount of requests by testing.

Quote from: iasenko on September 18, 2018, 07:34:57 PM

I will closely follow this project. We've been waiting for such thing for a very long time. Most of the bots are using now word spinner to hide the copy-pasting, it's not easy to detect them but it's not impossible either.

Thanks! I'll try to keep this thread updated as much as I can.