total used free shared buff/cache available
Mem: 488 106 84 4 297 345
Swap: 3071 448 2623
ps ux|grep 8bitd
theo 7541 52.5 76.4 2219172 382296 ? SLsl 09:27 208:58 ./8bitd -daemon
theo 8632 0.0 0.1 14520 888 pts/1 S+ 16:04 0:00 grep --color=auto 8bitd
So... to summary - 8bitd resident size is 382M while your system reports 106M mem usage and 448M swap usage. I guess your provider is overbooking resources and therefore swaps aggressively. If your provider allows to temporary bump VPS parameters - try 1GB mem plan - it should give instant effect.
If it is your bare metal machine, reduce the swappiness.