It is worth setting up if your host is going to be publicly accessible. With regards to setting up SSL, what you might need to understand is that clients usually cache the certificates of the hosts after connecting for the first time and if the certificate changes, they would flag the connection as possibly being MITM'ed.

Can you state exactly what you are experiencing with setting up SSL?