A transaction can have inputs signed by multiple keys--this is the concept behind hierarchical deterministic (HD) wallets, among others.

Bitcoin inputs and outputs are not just sent to addresses, they are controlled by scripts. Outputs should generally conform to standard script formats (P2PKH, P2SH, etc.), and inputs contain instructions to unlock them. Saying that value is sent to an address is essentially shorthand for saying, "This transaction has an output with a script that can be unlocked by a signature for this public key."

What I'm getting at is, which transactions are sending funds in which direction is complicated, and you won't be able to find a way to cover all your bases. The scheme you described should cover many cases, but it might be worth it to reevaluate this way of thinking about transactions if it's really that central to your application.