Bitcoin doesn't need a fully relational database, it just needs a key/value store. A standalone database like MySQL/Postgres would complicate installation a lot! A library equivalent like SQLite would probably be even slower than Berkeley DB which is what's used at the moment.

Right now the core Bitcoin network has run out of disk seeks, which are a finite resource. That's why some mining pools are dropping fee-paying transactions from SatoshiDice. In other words, Bitcoin has already reached a scalability limit and miners are load-shedding in order to keep the orphan rates down.

LevelDB was designed to optimize for reduced disk seeks as its primary goal, and it sacrifies many advanced database features to get that. Its design is one reason Google BigTable is such a high-performance storage system. Fortunately we don't need those advanced features, but we do need more seeks, which is why it makes sense to use it.