Thanks John,
Perhaps the HTTP request is triggering a catch-up which takes too long. Have you tried separating the loader from the server? One process runs Abe in an infinite loop passing --no-serve, and the web process uses --no-load (or datadir=[]). Then the web requests will not wait for data to load.
What happens if there is a catch-up triggered by request A, then request B comes in?
That stack-trace happens quite often in a row, not just once.
I'm trying your suggestion now, sounds promising to me.