This is no different than trusting a third party service.
From a regulatory point of view it's completely different to trusting a third-party service that actually handles the transaction. It's also much easier to distribute (other people can run mirrors of the directory) and validate (since you can check what addresses I'm giving out, and blow the whistle on me if I get cracked or start pulling shenanigans).
Do you seriously want to broadcast all of your transactions to the whole wide world and tie them to your email address?

I also do that if I stick a Bitcoin address on my website or in my signature. (OK, I don't necessarily put my email address on my website or my Bitcoin Talk signature, but anyone who can use Google can connect the two.)
If you don't want that you shouldn't use my public directory, but you could do it with a non-public directory that fed a new Bitcoin address to each user, while keeping the transactions on-chain. I guess one decentralized way to do that would be that you'd stick an "address server" URL in your DNS, which you could run yourself, and have that feed a new address to each person who shows up.
PS. I'm not saying there isn't value to off-chain systems for a lot of use-cases, just that off-chain isn't the only way to do it, and has a lot of downsides.