Nerdpack. What happens to Anaconda with ~/.local
Ok, I don’t this is well documented and it has been causing me headaches for months, but here it is:
- If you are using a virtual environment, a very common one is Anaconda or miniconda. This let’s you install python packages on your machine and switch between environments.
- Another package like this is pipenv which does much the same thing.
- The problem is that there are have been many times when I get the wrong package and I can’t figure out why in Anaconda.
The answer is pretty subtle but in short:
- Anaconda is a virtual environment except that it enables USER_SITE. This means that you can override the environment and this is set as ENABLE_USER_SITE=true. The problem is that if you have a
~/.local/libthen you will silently override the conda environment with those packages.
- Now this is not normally a problem, because the only way to do this is to run
pip install --userin your bare metal environment. So first of all, don’t do this!
- But the problem is that a VIM package, I’m forgetting the name, maybe Black creates a virtual python environment and it installs everything into that .local directory. As a result, you will pick old packages there.
- This creates all kinds of strangeness, the net is that if you want to figure out what packages are loading run
python -m sitewhich is a module that manages all this complexity
Well the hack solution is just to delete the ~/.local/lib but this does mean no local or user installed packages ever.
The real solution to this thought is to completely seal up Anaconda. You can do this permanently with a configuration file
pyvenv.cfg and this works because there are some magic here about environment configuration. It is complex, but it is hard to know where to put this file. The documentation that its goes into the parent of one directory sys.executable, sys.prefix and sys.exec_prefix.
For Anaconda with a homebrew installation, this appears to be
/usr/local/Caskroom/miniconda so put it there this
pyvenv.cfg and the line to disable it is but this didn’t seem to work even adding:
This by the way is an open bug is Anaconda which is how to isolate from the so called USER_SITE which is normally ~/.local
The fixes appear to be to run python with a -s flag or to set an environment variable PYTHONNOUSERSITE to false and you can check this by looking at
site.ENABLE_USER_SITE in python for running
python -m site from the command line.
The hacks to get around this are pretty ugly:
- Don’t allow any user packages (ugh!)
- Whenever you start conda, you have to remember to set that environment variable.
- After you start Anaconda, clone a virtualenv in a special way, in short, you create python venv and then use
conda create --cloneof that environment since virtualenv shuts this off. In this way it works much like pipenv which is sensitive to the directory it is created in. However this creates a complete copy so suddenly in your git repo you will have a
./liband have to manage all those files
- Reach deep into the guts of Anaconda at
/usr/local/Caskroom/miniconda/basewhich is hard to capture when you do a new installation and defeats the purpose of using Home-brew so you can create a script that does this in
- Reach deep into the python installation and find
site.pywhich you can get run running
import site; site.__file__and hand editing the file to set
ENABLE_USER_SITE = False
As a really strange aside, this ~/.local problem does *not* happen with Home-brew Python. That’s because the USER_SITE there is set to a home-brew directory and not
~/.local that is only true with Anaconda
So net, net, an obscure bug, but widely discussed since some users are relying of this silent import of packages. Personally, I think all packages should be completely explicit as pipenv does, but glad they have that environment variable.