The problem is that wxWidgets includes a bunch of "core" functionality that isn't related to GUI widgets at all.  String classes and whatnot.  The core of BitCoin was build with those utility classes.

You'll notice that Gavin's configure call disables GUI.

I have a linux box that has no windowing system on it at all; no GTK, no X, nothing.  I did something similar to Gavin and the daemon is running fine.

I'd much prefer if it could be built without wxWidgets, though.  (Or Boost, for that matter!)