Hey, i wrote my methodology in my github repo for my addresses list..
Anyways, here is what i did..
$ nl concat.txt | TMPDIR='/large/tmp/dir' LC_ALL=C sort -k2 -u | sort -n | cut -f2 > final.txt
Note that using LC_ALL=C will greatly speed up sorting!