I am using bitcoin core, bitcoin-cli but I am constructing the raw transaction manually.
If you want to do that, would it be better for you to create a raw transaction without any inputs and use fundrawtransaction to add the inputs instead? If you were to specify the fee rate as well as the deduction of the fees from the outputs, then it would be far easier than to calculate the fees manually. See this:
https://bitcoincore.org/en/doc/0.18.0/rpc/rawtransactions/fundrawtransaction/.