You transferred 500 Bitcoins to Patrick and did not require a signed contract to secure or to determine both parties obligations? Is that the way you handle your stock exchange business?
If you think an IRC log represents a legitimate contract, you have completely failed.
You need to read up on contract law then. Anything written can count as a valid contract. Heck, even a verbal or handshake agreement can be a contract, and enforceable in court. The only thing that needs to be done is the contract has to be proven. IRC logs can server that purpose, as can emails, recorded telephone conversations, etc.
The only thing an official signed contract does is makes it easier to enforce, especially when it comes to the nitty-gritty details. But from what was said in IRC, this is easily an enforceable contract in which Patrick promised to pay back interest on a loan, and has failed to do so.
Unless Patrick repays the loan, he absolutely SHOULD be called a scammer. He has an obligation to pay, because he said he would.