The biggest problem, in my view, is routing failures. It's extremely annoying, speaking out of personal experience. When I want to make a transaction, I want to be certain it will finish, and it's almost certainly the case that at least a quarter of my lightning payments fail. You can only reduce the failure probability by opening more and more channels
This is another reason why I like custodial LN wallets (for small amounts): in my experience, they're much better connected than my own channels.