Post
Topic
Board Project Development
Re: [BETA] [NEW] beta.ninjastic.space (forum search, archive and data visualization)
by
TryNinja
on 10/09/2025, 20:57:20 UTC
When I read this thread, I thought: if someone creates a thread and after a few minutes changes its entire content, ninjastic.space will stop keeping this historical record. This could be a strategy to create a scheme for those who want to do something wrong and be difficult to detect.

I remember you saying a while ago that you would scrape posts multiple times. Will you always do this, ensuring that every so often it checks for changes, even if a post is several days, weeks, or months old?

Perhaps this is a waste of resources that is not very useful, because we know that fortunately most posts are not edited very late.
The problem is that right now I save another copy of the entire post every time something changes, even if it's a single letter. If I allow multiple unlimited rescrapes, storage usage can go quickly very fast...

I thought about storing only the post diff, but then I need to rethink how my search engine works right now. Maybe there is another solution...