Swig isn't a python package just like you said, so it doesn't matter if you're in an env or using global python. You should be able to use it anywhere.

Using brew isn't going to make it system-dependent. You're installing those so that you can either build armory or put qt into the package. In the end the only things going into the actual package are qt, some python packages installed in the env, python, and armory

Looks like the problem isn't where swig is installed, but that it can't find some files. I'm not sure what you're trying to do with the Lib folder from the env either...