Quote from: dkbit98 on February 21, 2020, 04:30:48 PM

If it is not a secret, how much data space is needed for all that millions of posts?

I'm currenly using 54 GB for loyce.club, and store 4.2 million files.

Quote

And is there a way to use some compression?

I mainly store HTML-files, so indeed, it would be great if a webbrowser would just be able to use index.html.gz to largely reduce the disk space consumption, but I just tested it and my browser doesn't get it.

Due to my lack of time it took longer than I wanted, but I now added live updates for posts per user and per topic:

Quote from: LoyceV on July 21, 2019, 05:08:59 PM

Viewing unedited/deleted posts

See http://loyce.club/archive/posts/ for all posts (Working!)
New posts are archived within seconds after being created, and instantly available.
See http://loyce.club/archive/members/ for posts made by a certain user (Working!)
Updated every 5 minutes.
See http://loyce.club/archive/topics/ for posts made in a certain topic (Working!)
Updated every 5 minutes.

How to use it

Details

Files are stored with their msgID, userID or topicID as file name. I remove the last 4 digits to create the directory name. Each directory contains up to 10,000 HTML-files. Use CTRL-F to find what you're looking for.
I don't scrape hidden boards (such as Investigations).
I don't keep post titles
I save raw HTML, including quotes
If I run out of disk space, I might create compressed archives per 10,000 posts.
Although I plan to preserve all data, I make no guarantees. Feel free to archive posts.
My current (sponsored) webhost has enough storage space for years to come.
All scrape-times use Amsterdam time (CET).
Usually, I capture at least 99.95% of all posts. Server or internet connection problems can severely reduce this.

Examples

The unedited version of this post: http://loyce.club/archive/posts/5190/51902990.html
(the layout looks better in more recent archived posts)
All posts made by me: http://loyce.club/archive/members/45/459836.html
(obviously only since I started archiving posts)
All posts made in this topic: http://loyce.club/archive/topics/516/5167469.html
(the first posts aren't shown because of the slightly different format used for my archive)