Sounds pretty secure, I just store mine in encrypted files and keep them in a few places. There's always a chance you make loose the USB drive and don't forget its not reccomended to use flash drives for long term storage. Pen drives degrade over time.... Apparently 
With this thought nothing would be stored on the USB drive. If it works the way I think it does all of the information I need to access the wallet is stored in the twelve word seed provided by Electrum. This way the only way I could lose the bitcoins here are through some sort of Linux vulnerability, an issue with Electrum, or something physically stealing my written down seed in real life.
Or maybe my ISP could do a man in the middle attack? Not that it's likely, but is that an issue as well? Since I'm downloading Electrum from a live internet connection. I guess thinking about it that way means that it could be my ISP, or Electrum's web server or anything inbetween...