If someone gains access to your CLAM wallet (and the password, if encrypted), then they also have access to the BTC, LTC, or DOGE at the addresses you imported.
A precise identification of the address as the lynchpin. IIRC, the original instructions suggested that, for maximum peace of mind, any balance in a qualifying BTC/LTC/DOGE wallet should be transferred to a newly-created BTC/LTC/DOGE wallet/address and the now-empty qualifying wallet/address used for the CLAM import. I couldn't pick any holes in that but IANAC (C=Cryptographer).
Cheers
Graham
Edit, use a more precise reference