Then refute his claim. Give evidence that you're the only person who has owned that account. Sign old bitcoin addresses, dig up social media accounts, etc etc. Don't just complain and try to make a connection of fraud with a negative rep (albeit somewhat preemptive).