It's actually pretty to easy to end up with a non-graphical system.  In the debian installer, just don't select the desktop environment task.

(https://www.debian.org/releases/stable/i386/ch06s03.html.en see 6.3.5.2. Selecting and Installing Software)

That said, if you want to boot into an xserver and still be very keyboard driven, tiling window managers like i3 (as pointed out by Anduck) can be quite wonderful.

Also, Carlton Banks nailed it when it comes to the popular linuxes.  If you really want to get into rolling your own, you could try the linux-from-scratch method (warning: not for the faint of heart).