I'm sure there could be a mechanism that allows you to swap fully charged batteries for the empties as well so charging times wouldn't be an issue if you have spares.
The problem is that you cannot simply swap your batteries as fast as you can fill your gas tank to full. Also, anyone would not want to go through the hassle of opening up your hood(or anywhere your battery is stored) and the hassle of removing and reattaching some things when you are in a long distance travel because you could have had used that time to do other more important things or rest.
Free energy is free whereas gas costs a lot of money. Doesn't really matter if it takes a little longer to 'fill up' and seems a small price to pay for free energy.
Still, the time it would take for it to be fully charged is a long time and typically charging happens at the busiest time of the day(which is in the morning). You would not need to drive to work at night, do you? (unless you are in a night shift or travels for only a kilometer or two, the long charging time is fine).