What WinAPI are used to encrypt TLS? If you mean intercept internal Claymore functions, this need reverse engineering and I am not good at it. Also Claymore Miner is 64 bit and seems to be packed / obfuscated. As stash2coin logs show packet buffer is already encrypted in ws2_32.send (expected) and it also seems like it uses encryption for dev fee even if main worker is not.
My guess is that he is using WinHTTP library if the project is written in C++. Someone has to be foolhardy enough to implement TLS with raw sockets 
In either case you can see what APIs are being used with Dependency Walker. It's all in plaintext.I just peeked into the Claymore Dual Miner v.10.1 with IDA Pro. He doesn't pack it like most malware is packed. He uses something called VMProtect. It's a weird type "packer" -- it basically takes the assembly/machine code for the part of the executable that the author wants to obfuscate and converts it into some proprietary byte code that VMProtect invented. Then when the executable runs, the obfuscated part has to go thru VMProtect's virtual machine to get interpreted. This makes the code extremely slow when executing, but hard to reverse engineer (simply because the structure of their proprietary byte code is not documented.) The rest of the binary doesn't seem to be packed though. This btw makes me think that if the Claymore Miner wasn't packed that way it might have produced a slightly better hash rate. Just a guess though.
Oh, and as x64 binary goes, the same WinAPI assembly trampoline can be used for it as well. We'll just need to modify the machine code for it. Or, you can use WinDivert library, like this guy did with his NoFee executable.
In any case, I wouldn't mind to collaborate with you on your open source project -- as a challenge I guess. PM me if anything.