It seems the way you say is safe but it's somewhat complicated. I prefer to keep all of that on my device, in my opinion that way is still said to be safe. As long as you don't do anything like opening the door for malware. Or rather do not install anything on your device.
Indeed, the way I use is complicated, but security is more guaranteed. Although the data can be stolen by others, they will not be able to open it because it has been given a password. Devices that are used without attaching anything are not necessarily safe if they are still connected to the internet.