It looks like it's a non-standard authentication method, not using TLS but it is using keys on the secp256k1 curve and a form of ECDH called Noise_XK.

In order to get the session key that is generated by this algorithm which encrypts the traffic, you'll have to find a utility which allows you to replicate the Noise_XK key exchange process.