As others have alluded to, end user security is the biggest problem IMO.  It would be nice if the bitcoin wizards could just program away the hackers and scammers and protect end users from themselves but that is easier said than done.  I do feel confident that eventually a hardware and/or software "killer app" solution will be designed to do this without the requirement of third party trust.