Thanks to Tuxavant's suggestion, it seems that adding a contact only works if it's just the address. No protocol and no label. It also just stores it as the address, with no way to name it and also doesn't seem to be a way to delete it.

Making it work with the standard protocol and allowing renaming or at least deleting would be nice additions to the app. If I have time later I'll check out the code and see if it's something I can add myself. Probably be a day or so till I get around to it though.