Is there any way to backup labels somehow?
One time I had to reinstall Electrum on my new computer and after importing my old seed phrase all labels disappeared, but I didn't click save backup from file menu before (not sure if that saves labels btw).
Maybe labels are also saved in some Electrum folder file.
Yup, the labels are only in the wallet file though. The backup wallet functions backs up your wallet file and exports it in .backup extension.
Either using that or just directly copying your wallet file from Electrum's data directory is fine.