I mean, he must have been aware of the fact that this is a legal grey zone, and no one takes such a risk without remaining anonymous or taking precautions like living/hosting on some caribbean island with low/almost no goverment control?
Nope, Nefario naively assumed what he was doing was in fact legal. Not that he ever bothered to ask someone who knows the law, until recently at least, with the predictable result of GLBSE closing immediately.
As for running GLBSE as a tor hidden service; Im sure its doable, but if GLBSE was a scammer dream come true, imagine what a Tor based equivalent would be like.