After a long fight, I managed to add QR code output and http callback to Jay's libpam-bitid. Now you can use your mobile phone to authenticate a ssh session. It works for sudo as well, and probably anything that can output the QR code into the terminal. It probably wouldn't work for a graphical session manager / screensaver.
I made a youtube video to show how cool it is:
https://www.youtube.com/watch?v=U6I4HuzSTM0.