PEP 370 – Per user site-packages directory
This PEP proposes a new a per user site-packages directory to allow users the local installation of Python packages in their home directory.
Rationale
Current Python versions don’t have a unified way to install packages into the home directory of a user (except for Mac Framework builds). Users are either forced to ask the system administrator to install or update a package for them or to use one of the many workarounds like Virtual Python [1], Working Env [2] or Virtual Env [3].
It’s not the goal of the PEP to replace the tools or to implement isolated installations of Python. It only implements the most common use case of an additional site-packages directory for each user.
The feature can’t be implemented using the environment variable PYTHONPATH. The env var just inserts a new directory to the beginning of sys.path but it doesn’t parse the pth files in the directory. A full blown site-packages path is required for several applications and Python eggs.
Specification
site directory (site-packages)
A directory in sys.path . In contrast to ordinary directories the pth files in the directory are processed, too.
A site directory inside the users’ home directory. A user site directory is specific to a Python version. The path contains the version number (major and minor only). Unix (including Mac OS X) ~/.local/lib/python2.6/site-packages Windows %APPDATA%/Python/Python26/site-packages
Usually the parent directory of the user site directory. It’s meant for Python version specific data like config files, docs, images and translations. Unix (including Mac) ~/.local/lib/python2.6 Windows %APPDATA%/Python/Python26
It’s located inside the user’s home directory. The user site and use config directory are inside the base directory. On some systems the directory may be shared with 3rd party apps. Unix (including Mac) ~/.local Windows %APPDATA%/Python
A directory for binaries and scripts. [10] It’s shared across Python versions and the destination directory for scripts. Unix (including Mac) ~/.local/bin Windows %APPDATA%/Python/Scripts
Windows Notes
On Windows the Application Data directory (aka APPDATA ) was chosen because it is the most designated place for application data. Microsoft recommends that software doesn’t write to USERPROFILE [5] and My Documents is not suited for application data, either. [8] The code doesn’t query the Win32 API, instead it uses the environment variable %APPDATA%.
The application data directory is part of the roaming profile. In networks with domain logins the application data may be copied from and to the a central server. This can slow down log-in and log-off. Users can keep the data on the server by e.g. setting PYTHONUSERBASE to the value “%HOMEDRIVE%%HOMEPATH%Applicata Data”. Users should consult their local administrator for more information. [13]
Unix Notes
On Unix ~/.local was chosen in favor over ~/.python because the directory is already used by several other programs in analogy to /usr/local . [7] [11]
Mac OS X Notes
On Mac OS X Python uses ~/.local directory as well. [12] Framework builds of Python include ~/Library/Python/2.6/site-packages as an additional search path.
Implementation
The site module gets a new method adduserpackage() which adds the appropriate directory to the search path. The directory is not added if it doesn’t exist when Python is started. However the location of the user site directory and user base directory is stored in an internal variable for distutils.
The user site directory is added before the system site directories but after Python’s search paths and PYTHONPATH . This setup allows the user to install a different version of a package than the system administrator but it prevents the user from accidentally overwriting a stdlib module. Stdlib modules can still be overwritten with PYTHONPATH .
For security reasons the user site directory is not added to sys.path when the effective user id or group id is not equal to the process uid / gid [9]. It’s an additional barrier against code injection into suid apps. However Python suid scripts must always use the -E and -s option or users can sneak in their own code.
The user site directory can be suppressed with a new option -s or the environment variable PYTHONNOUSERSITE . The feature can be disabled globally by setting site.ENABLE_USER_SITE to the value False . It must be set by editing site.py . It can’t be altered in sitecustomize.py or later.
The path to the user base directory can be overwritten with the environment variable PYTHONUSERBASE . The default location is used when PYTHONUSERBASE is not set or empty.
distutils.command.install (setup.py install) gets a new argument —user to install packages in the user site directory. The required directories are created on demand.
distutils.command.build_ext (setup.py build_ext) gets a new argument —user which adds the include/ and lib/ directories in the user base directory to the search paths for header files and libraries. It also adds the lib/ directory to rpath.
The site module gets two arguments —user-base and —user-site to print the path to the user base or user site directory to the standard output. The feature is intended for scripting, e.g. ./configure —prefix $(python2.5 -m site —user-base)
distutils.sysconfig will get methods to access the private variables of site. (not yet implemented)
The Windows updater needs to be updated, too. It should create a menu item which opens the user site directory in a new explorer windows.
Backwards Compatibility
Reference Implementation
A reference implementation is available in the bug tracker. [4]
PEP 250 – Using site-packages on Windows
The standard Python distribution includes a directory Lib/site-packages , which is used on Unix platforms to hold locally installed modules and packages. The site.py module distributed with Python includes support for locating other modules in the site-packages directory.
This PEP proposes that the site-packages directory should be used on the Windows platform in a similar manner.
Motivation
On Windows platforms, the default setting for sys.path does not include a directory suitable for users to install locally developed modules. The “expected” location appears to be the directory containing the Python executable itself. This is also the location where distutils (and distutils-generated installers) installs packages. Including locally developed code in the same directory as installed executables is not good practice.
Clearly, users can manipulate sys.path , either in a locally modified site.py , or in a suitable sitecustomize.py , or even via .pth files. However, there should be a standard location for such files, rather than relying on every individual site having to set their own policy.
In addition, with distutils becoming more prevalent as a means of distributing modules, the need for a standard install location for distributed modules will become more common. It would be better to define such a standard now, rather than later when more distutils-based packages exist which will need rebuilding.
It is relevant to note that prior to Python 2.1, the site-packages directory was not included in sys.path for Macintosh platforms. This has been changed in 2.1, and Macintosh includes sys.path now, leaving Windows as the only major platform with no site-specific modules directory.
Implementation
The implementation of this feature is fairly trivial. All that would be required is a change to site.py , to change the section setting sitedirs. The Python 2.1 version has:
if os.sep == '/': sitedirs = [makepath(prefix, "lib", "python" + sys.version[:3], "site-packages"), makepath(prefix, "lib", "site-python")] elif os.sep == ':': sitedirs = [makepath(prefix, "lib", "site-packages")] else: sitedirs = [prefix]
A suitable change would be to simply replace the last 4 lines with:
else: sitedirs == [prefix, makepath(prefix, "lib", "site-packages")]
Changes would also be required to distutils, to reflect this change in policy. A patch is available on Sourceforge, patch ID 445744, which implements this change. Note that the patch checks the Python version and only invokes the new behaviour for Python versions from 2.2 onwards. This is to ensure that distutils remains compatible with earlier versions of Python.
Finally, the executable code which implements the Windows installer used by the bdist_wininst command will need changing to use the new location. A separate patch is available for this, currently maintained by Thomas Heller.
Notes
- This change does not preclude packages using the current location – the change only adds a directory to sys.path , it does not remove anything.
- Both the current location ( sys.prefix ) and the new directory (site-packages) are included in sitedirs, so that .pth files will be recognised in either location.
- This proposal adds a single additional site-packages directory to sitedirs. On Unix platforms, two directories are added, one for version-independent files (Python code) and one for version-dependent code (C extensions). This is necessary on Unix, as the sitedirs include a common (across Python versions) package location, in /usr/local by default. As there is no such common location available on Windows, there is also no need for having two separate package directories.
- If users want to keep DLLs in a single location on Windows, rather than keeping them in the package directory, the DLLs subdirectory of the Python install directory is already available for that purpose. Adding an extra directory solely for DLLs should not be necessary.
Open Issues
- Comments from Unix users indicate that there may be issues with the current setup on the Unix platform. Rather than become involved in cross-platform issues, this PEP specifically limits itself to the Windows platform, leaving changes for other platforms to be covered in other PEPs.
- There could be issues with applications which embed Python. To the author’s knowledge, there should be no problem as a result of this change. There have been no comments (supportive or otherwise) from users who embed Python.
Copyright
This document has been placed in the public domain.