A significant percentage of the population doesn't even understand the difference between a self-custody wallet, and an account with a service provider.

Then companies like Coinbase (and others) make it even more confusing by calling their accounts "Wallets" all over their user interface.

Then on top of that, you've got multi-sig addresses and the possibility of CoinJoin transactions.

And even if users completely understand how to "use their own wallet" AND they understand that they need to avoid input sharing, you still have the problem that a user could compromise their address very shortly after sending the bitcoins to you (either lost the private keys and don't have a backup, or leaked the private keys). In either case, those coins aren't going to make it back to the person you intend them to.

And if the input is spending a non-standard output, then there is no "address" at all. Are you going to perfectly reproduce their customized script? Or is your code going to try to convert the bytes into an address for which they don't have the private key?