It also holds other people's public keys, the network object inventory, your subscriptions, and your whitelist and blacklist. So it is reasonably logical that it would also hold your address book.
I think this is flawed: the green stuff is user settings and the red comes from the network. Those should be separated.
I agree that that is the most logical way to separate them. But that isn't what people are asking for.