Post
Topic
Board Meta
Re: Slow forum
by
PrimeNumber7
on 04/01/2020, 09:31:35 UTC
You can direct your script to review multiple pages of recent if it detects that all the posts it most recently scrapes are new, it can continue doing this until it reaches the last maximum recent page the forum provides.
I've thought about it, but haven't had the time to add it yet.
Ideally, I should switch to scraping from another server if I miss a few posts. That way, I could even cover downtime if I reboot the server. But that's for later Smiley
If you want, you can send me your code for your scraper. Adding a counter and telling your script to check the next page is something that should be very simple (< 5 mins, including testing). If I am familiar with the programming language (or if I can read the documentation necessary to help in a short time), I can give you suggested lines of code to add.