I am running hyperstaked on Ubuntu 14.04.2 LTS server, without GUI.
Even for pure POW wallets, I have noticed such behaviour when a lot of transactions have been received. Happened often during solo mining for coins with very low diff.
Yeah this is pretty much the case. If you have thousands and thousands of records then it will cause a bit of lag. I don't have any good way to prevent this that I can think of.