Unless you are running a x86 OS or you have low RAM
and disabled the swap (or actually managed to fill swap partition), you can't get a bad_alloc for running out of physical RAM. This is all virtual address space, the underlying actual RAM isn't all that relevant.
bad_alloc appears when the OS fails to allocate memory. This is more often the case when a bogus amount of memory is requested (say unserializing a random chunck of 9 bytes as a varInt and allocating that absurd number) than actually exhausting all RAM on the system.
If you suspect it is actually how much memory you have available on your system, you should try and reduce the RAM Armory uses instead of increasing yours:
https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/BlockchainScanner.h#L96Set this value to 2 to cut the memory use in half, then
make clean
make
It's currently tuned to run at around 1GB.