PEP 365 – Adding the pkg_resources module
This PEP proposes adding an enhanced version of the pkg_resources module to the standard library.
pkg_resources is a module used to find and manage Python package/version dependencies and access bundled files and resources, including those inside of zipped .egg files. Currently, pkg_resources is only available through installing the entire setuptools distribution, but it does not depend on any other part of setuptools; in effect, it comprises the entire runtime support library for Python Eggs, and is independently useful.
In addition, with one feature addition, this module could support easy bootstrap installation of several Python package management tools, including setuptools , workingenv , and zc.buildout .
Proposal
Rather than proposing to include setuptools in the standard library, this PEP proposes only that pkg_resources be added to the standard library for Python 2.6 and 3.0. pkg_resources is considerably more stable than the rest of setuptools, with virtually no new features being added in the last 12 months.
However, this PEP also proposes that a new feature be added to pkg_resources , before being added to the stdlib. Specifically, it should be possible to do something like:
python -m pkg_resources SomePackage==1.2
to request downloading and installation of SomePackage from PyPI. This feature would not be a replacement for easy_install ; instead, it would rely on SomePackage having pure-Python .egg files listed for download via the PyPI XML-RPC API, and the eggs would be placed in the $PYTHON_EGG_CACHE directory, where they would not be importable by default. (And no scripts would be installed.) However, if the download egg contains installation bootstrap code, it will be given a chance to run.
These restrictions would allow the code to be extremely simple, yet still powerful enough to support users downloading package management tools such as setuptools , workingenv and zc.buildout , simply by supplying the tool’s name on the command line.
Rationale
Many users have requested that setuptools be included in the standard library, to save users needing to go through the awkward process of bootstrapping it. However, most of the bootstrapping complexity comes from the fact that setuptools-installed code cannot use the pkg_resources runtime module unless setuptools is already installed. Thus, installing setuptools requires (in a sense) that setuptools already be installed.
Other Python package management tools, such as workingenv and zc.buildout , have similar bootstrapping issues, since they both make use of setuptools, but also want to provide users with something approaching a “one-step install”. The complexity of creating bootstrap utilities for these and any other such tools that arise in future, is greatly reduced if pkg_resources is already present, and is also able to download pre-packaged eggs from PyPI.
(It would also mean that setuptools would not need to be installed in order to simply use eggs, as opposed to building them.)
Finally, in addition to providing access to eggs built via setuptools or other packaging tools, it should be noted that since Python 2.5, the distutils install package metadata (aka PKG-INFO ) files that can be read by pkg_resources to identify what distributions are already on sys.path . In environments where Python packages are installed using system package tools (like RPM), the pkg_resources module provides an API for detecting what versions of what packages are installed, even if those packages were installed via the distutils instead of setuptools.
Implementation and Documentation
The pkg_resources implementation is maintained in the Python SVN repository under /sandbox/trunk/setuptools/ ; see pkg_resources.py and pkg_resources.txt . Documentation for the egg format(s) supported by pkg_resources can be found in doc/formats.txt . HTML versions of these documents are available at:
(These HTML versions are for setuptools 0.6; they may not reflect all of the changes found in the Subversion trunk’s .txt versions.)
Copyright
This document has been placed in the public domain.
Packaging namespace packages¶
Namespace packages allow you to split the sub-packages and modules within a single package across multiple, separate distribution packages (referred to as distributions in this document to avoid ambiguity). For example, if you have the following package structure:
mynamespace/ __init__.py subpackage_a/ __init__.py . subpackage_b/ __init__.py . module_b.py setup.py
And you use this package in your code like so:
from mynamespace import subpackage_a from mynamespace import subpackage_b
Then you can break these sub-packages into two separate distributions:
mynamespace-subpackage-a/ setup.py mynamespace/ subpackage_a/ __init__.py mynamespace-subpackage-b/ setup.py mynamespace/ subpackage_b/ __init__.py module_b.py
Each sub-package can now be separately installed, used, and versioned.
Namespace packages can be useful for a large collection of loosely-related packages (such as a large corpus of client libraries for multiple products from a single company). However, namespace packages come with several caveats and are not appropriate in all cases. A simple alternative is to use a prefix on all of your distributions such as import mynamespace_subpackage_a (you could even use import mynamespace_subpackage_a as subpackage_a to keep the import object short).
Creating a namespace package¶
There are currently three different approaches to creating namespace packages:
- Use native namespace packages. This type of namespace package is defined in PEP 420 and is available in Python 3.3 and later. This is recommended if packages in your namespace only ever need to support Python 3 and installation via pip .
- Use pkgutil-style namespace packages. This is recommended for new packages that need to support Python 2 and 3 and installation via both pip and python setup.py install .
- Use pkg_resources-style namespace packages. This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe.
While native namespace packages and pkgutil-style namespace packages are largely compatible, pkg_resources-style namespace packages are not compatible with the other methods. It’s inadvisable to use different methods in different distributions that provide packages to the same namespace.
Native namespace packages¶
Python 3.3 added implicit namespace packages from PEP 420. All that is required to create a native namespace package is that you just omit __init__.py from the namespace package directory. An example file structure:
setup.py mynamespace/ # No __init__.py here. subpackage_a/ # Sub-packages have __init__.py. __init__.py module.py
It is extremely important that every distribution that uses the namespace package omits the __init__.py or uses a pkgutil-style __init__.py . If any distribution does not, it will cause the namespace logic to fail and the other sub-packages will not be importable.
Because mynamespace doesn’t contain an __init__.py , setuptools.find_packages() won’t find the sub-package. You must use setuptools.find_namespace_packages() instead or explicitly list all packages in your setup.py . For example:
from setuptools import setup, find_namespace_packages setup( name='mynamespace-subpackage-a', . packages=find_namespace_packages(include=['mynamespace.*']) )
A complete working example of two native namespace packages can be found in the native namespace package example project.
Because native and pkgutil-style namespace packages are largely compatible, you can use native namespace packages in the distributions that only support Python 3 and pkgutil-style namespace packages in the distributions that need to support Python 2 and 3.
pkgutil-style namespace packages¶
Python 2.3 introduced the pkgutil module and the pkgutil.extend_path() function. This can be used to declare namespace packages that need to be compatible with both Python 2.3+ and Python 3. This is the recommended approach for the highest level of compatibility.
To create a pkgutil-style namespace package, you need to provide an __init__.py file for the namespace package:
setup.py mynamespace/ __init__.py # Namespace package __init__.py subpackage_a/ __init__.py # Sub-package __init__.py module.py
The __init__.py file for the namespace package needs to contain only the following:
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
Every distribution that uses the namespace package must include an identical __init__.py . If any distribution does not, it will cause the namespace logic to fail and the other sub-packages will not be importable. Any additional code in __init__.py will be inaccessible.
A complete working example of two pkgutil-style namespace packages can be found in the pkgutil namespace example project.
pkg_resources-style namespace packages¶
Setuptools provides the pkg_resources.declare_namespace function and the namespace_packages argument to setup() . Together these can be used to declare namespace packages. While this approach is no longer recommended, it is widely present in most existing namespace packages. If you are creating a new distribution within an existing namespace package that uses this method then it’s recommended to continue using this as the different methods are not cross-compatible and it’s not advisable to try to migrate an existing package.
To create a pkg_resources-style namespace package, you need to provide an __init__.py file for the namespace package:
setup.py mynamespace/ __init__.py # Namespace package __init__.py subpackage_a/ __init__.py # Sub-package __init__.py module.py
The __init__.py file for the namespace package needs to contain only the following:
__import__('pkg_resources').declare_namespace(__name__)
Every distribution that uses the namespace package must include an identical __init__.py . If any distribution does not, it will cause the namespace logic to fail and the other sub-packages will not be importable. Any additional code in __init__.py will be inaccessible.
Some older recommendations advise the following in the namespace package __init__.py :
try: __import__('pkg_resources').declare_namespace(__name__) except ImportError: __path__ = __import__('pkgutil').extend_path(__path__, __name__)
The idea behind this was that in the rare case that setuptools isn’t available packages would fall-back to the pkgutil-style packages. This isn’t advisable because pkgutil and pkg_resources-style namespace packages are not cross-compatible. If the presence of setuptools is a concern then the package should just explicitly depend on setuptools via install_requires .
Finally, every distribution must provide the namespace_packages argument to setup() in setup.py . For example:
from setuptools import find_packages, setup setup( name='mynamespace-subpackage-a', . packages=find_packages() namespace_packages=['mynamespace'] )
A complete working example of two pkg_resources-style namespace packages can be found in the pkg_resources namespace example project.
Table of Contents
- An Overview of Packaging for Python
- The Packaging Flow
- Tutorials
- Guides
- Installing packages using pip and virtual environments
- Installing stand alone command line tools
- Installing pip/setuptools/wheel with Linux Package Managers
- Installing scientific packages
- Package index mirrors and caches
- Hosting your own simple repository
- Packaging and distributing projects
- Including files in source distributions with MANIFEST.in
- Single-sourcing the package version
- Dropping support for older Python versions
- Packaging binary extensions
- Packaging namespace packages
- Creating and discovering plugins
- Using TestPyPI
- Making a PyPI-friendly README
- Publishing package distribution releases using GitHub Actions CI/CD workflows
- Tool recommendations
- Analyzing PyPI package downloads