Is it possible to "merge" two encrypted wallets into one and have a completely new database, if they are encrypted with the same password?
Technically it's possible, but you'd make your own standard for merging multiple wallet and managing merged wallet since :
1. Usually it's more sensible to create new wallet and send bitcoin from previous wallet to newly created wallet
2. The wallet might be created from different version of Bitcoin Core / Bitcoin Qt. There are many possible scenario such as managing 2 xprv (master private key) and combining non-HD and HD wallet.
Depending on your goal, it's likely there are simpler way to achieve your goal.
Is there any paper on how the encryption works?
AFAIK no, you need to see source code of Bitcoin Core, see link mentioned above or search about it on
https://bitcoin.stackexchange.com/