You can just install Xfce on Ubuntu and still get 5 years LTS, right? The few GB size difference for having some unused software on your system shouldn't matter compared to the size of the blockchain.
It's definitely possible. But since OP said "There has to be a simple OS that just works", there are few things i'd like to point.
1. Conflict possibility between DE or default app comes with DE. You might mention you could just remove GNOME, but there's risk you accidentally remove important application or library.
2. I have doubt there won't be any problem when you perform major update (e.g. 20.04 LTS to 22.04 LTS).