The worst they can do probably is cause BTC prices to temporarily become more volatile (by hoarding an extreme amount of coins, and releasing them all at once after prices settle - an extreme case). "Destroying" BTC (by destroying the wallet containing the hoarded coins for example) won't affect much in the long-run, as BTC is meant to be infinitely divisible - any remaining coins in circulation will take their place.
The only true way to destory Bitcoin as a whole would be to either hoard ALL 21 million coins (thereby completely removing it from circulation), or obtain and maintain >50% of the total network's hashing power. As ASICs are still rare, and with the current network speed of ~60 TH/s along with the increasing price of BTC, either option would be extremely costly endeavours, and realistically, I just don't see this happening.