I don't think that assumption would be unreasonable either, until he tells you explicitly denies it.  Now for your assumption to be true, FH would have to be a liar.

If you think he's a liar, fine.  But to say "I don't care what he did or says", demonstrates that you don't care whether or not your accusations are true or false.

You are the one making the accusation, the burden of proof is on you.