Well hell, if you want to go really extreme, it isn't really Linux, but it is Unix based.

You can install OpenBSD without GUI, it is easy to install, but I am not sure that there is much you can do with it with a default install.

I mean if you don't want a GUI, I already doubt that you want an Internet browser on it, so it might be something for you.

The system is extremely secure and very minimalistic. You can run Bitcoin Core on it, I run a Bitcoin Core node on OpenBSD without GUI, but I never really considered it as a general purpose PC. A bit too extreme for me.