A standard Python environment directory structure (created by Anaconda)
# base python
/anaconda/
# user created environment
/anaconda/env/
Each python environment follow the similar structure (including anaconda's base one)
PATH/bin
(include executable binary, like conda, pip, python_intepreter, etc)
PATH/lib
/python_{version_number}
/site-package (both conda and pip will install packages to this directory)
-
Q: under conda environment, where does
pipinstall packages to?- A: within activate conda environment's corresponding
/lib/python{version}/site-package/directory
- A: within activate conda environment's corresponding
-
Q: if not using conda, where is Mac's original default python environment?
- A: no idea ...
-
Q: conda by default install environment to system directory
/anaconda(same as//anaconda). Can I change it to user's local directory?- A: tbd
Anaconda
Create/change environment
Create environment
conda create --name snowflakes pandas
conda create --name bunnies python=3 astroid babel
create Python 3 environment
$ conda create -n py35 python=3.5 anaconda
$ source activate py35
$ source deactivate
Here, the ‘py35’ is the name of the environment you want to create, and ‘anaconda’ is the meta-package that includes all of the actual Python packages comprising the Anaconda distribution. When creating a new environment and installing Anaconda, you can specify the exact package and Python versions, for example, numpy=1.7 or python=3.5.
Create Python 2 environment
$ conda create -n py27 python=2.7 anaconda
$ source activate py27
$ source deactivate
TIP: Install all the programs you will want in this environment at the same time. Installing one program at a time can lead to dependency conflicts.
Switch environment
source activate snowflakes
List all environments
conda info --envs
conda env list
Clone environment
conda create --name flowers --clone snowflakes
Remove environment
conda remove --name flowers --all
Export environment file and create env from environment file
First switch to the environment you want to export, then:
conda env export > environment.yml
conda env create -f environment.yml
Find system's python package directory:
>>> import site; site.getsitepackages()
['/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages', '/usr/lib/site-python']
/usr/lib/python2.7/site-packages seems to be important ?
if I using sudo pip install virtualenv it will install virtualenv package into /usr/bin
Use pip to manage environment
Check package information (including the installed location)
pip show pandas
- ** pip is the recommended installer **
Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state.
Update pip:
pip install -U pip setuptools
Normal Install or update a package
To install the latest version of “SomeProject”:
pip install SomeProject
To install a specific version:
pip install SomeProject==1.4
To install greater than or equal to one version and less than another:
pip install SomeProject>=1,<2
To install a version that’s “compatible” with a certain version: [4]
pip install SomeProject~=1.4.2
In this case, this means to install any version “==1.4.*” version that’s also “>=1.4.2”.
To update a package:
pip install -U SomePackage
pip install --upgrade SomeProject
Source Distributions vs Wheels
- pip can install from either Source Distributions (sdist) or Wheels, but if both are present on PyPI, pip will prefer a compatible wheel.
- Wheels are a pre-built distribution format that provides faster installation compared to Source Distributions (sdist), especially when a project contains compiled extensions.
Use pip with proxy setting
pip install --proxy user:pass@proxy:port works
pip search --proxy user:pass@proxy:port dont work
Installing to the User Site
To install packages that are isolated to the current user, use the --user flag:
pip install --user SomeProject
Requirements files
Install a list of requirements specified in a Requirements File.
pip install -r requirements.txt
Installing from a local src tree
Installing from local src in Development Mode, i.e. in such a way that the project appears to be installed, but yet is still editable from the src tree.
pip install -e <path>
You can also install normally from src
pip install <path>
Installing from local archives
Install a particular source archive file.
pip install ./downloads/SomeProject-1.0.4.tar.gz
Install from a local directory containing archives (and don’t check PyPI)
pip install --no-index --find-links=file:///local/dir/ SomeProject
pip install --no-index --find-links=/local/dir/ SomeProject
pip install --no-index --find-links=relative/dir/ SomeProject
virtualenv
$ virtualenv --no-site-packages env_name
source env_name/bin/activate
- say we create the virtual directory 'env_name':
the python site-package will be under:
env_name/lib/python2.7/site-packages/
Python Packaging Ecosystem
- A package is simply a directory with an init.py file inside it.
- distutils was created to install packages into the PYTHONPATH with little difficulty. The PYTHONPATH, also sys.path in code, is a list of locations to look for Python packages.
- if PYTHONPATH is set to /www/python:/opt/py, the search path will begin with
['', '/www/python', '/opt/py', ...].
>>> import sys
>>> sys.path
['', # the current directory, represented by a null string
'/usr/local/lib/python2.6',
'/usr/local/lib/python2.6/site-packages',
...]
>>> import mypackage # this package is at current directory
>>> mypackage.__file__
'mypackage/__init__.py'
The convention way of manually installing packages is to put them in the .../site-packages/ directory. But one may want to install Python modules into some arbitrary directory.
The most convenient way is to add a path configuration file to a directory that’s already in Python’s path, which could be the .../site-packages/ directory. Path configuration files have an extension of .pth, and each line must contain a single path that will be appended to sys.path.
Finally, sys.path is just a regular Python list, so any Python application can modify it by adding or removing entries.
-
Current python packaging tools
- Distribute
- pip, which replace easy_install
- virtualenv
Distribute is a collection of enhancements to the Python standard library module: distutils (for Python 2.3.5 and up on most platforms; 64-bit platforms require a minimum of Python 2.4) that allows you to more easily build and distribute Python packages, especially ones that have dependencies on other packages. Distribute was created because the Setuptools package is no longer maintained.
python directory structure
Linux
use default python
'/usr/lib/python2.7/site-packages', '/usr/lib/site-python'
use anaconda
use virtualenv
Mac
use default python
use anaconda
use virtualenv
Python Package
How to create Python Package (not very useful)
This is better : https://packaging.python.org/distributing/
Install Package
https://packaging.python.org/installing/
It’s important to note that the term “package” in this context is being used as a synonym for a distribution (i.e. a bundle of software to be installed), not to refer to the kind of package that you import in your Python source code (i.e. a container of modules). It is common in the Python community to refer to a distribution using the term “package”. Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or another larger software distribution like Python itself.
