A transaction can have multiple inputs and multiple outputs. There's no way to be sure that the input address you choose to refund to was the original owner of those Bitcoin.