total used free shared buff/cache available
Mem: 488 106 84 4 297 345
Swap: 3071 448 2623
ps ux|grep 8bitd
theo 7541 52.5 76.4 2219172 382296 ? SLsl 09:27 208:58 ./8bitd -daemon
theo 8632 0.0 0.1 14520 888 pts/1 S+ 16:04 0:00 grep --color=auto 8bitd
So... to summary - 8bitd resident size is 382M while your system reports 106M mem usage and 448M swap usage. I guess your provider is overbooking resources and therefore swaps aggressively. If your provider allows to temporary bump VPS parameters - try 1GB mem plan - it should give instant effect.
If it is your bare metal machine, reduce the swappiness.
I also get the feeling the provider is overbooking resources. I've set the swappiness to 12 now to see what that will do. If that doesn't help I'm thinking of switching provider again