Python setup local package

Installing Python packages locally

The process of setting up Python for your personal use and needs consists of first choosing a Python distribution and setting up the environment using modules, and second adding any custom packages to your environment locally. These two steps are discussed below.

Choosing a Python distribution and setting up the environment

As with all user-selectable system-supplied software at the HPCC, modules are used to select the Python distribution to be used and set up a user’s environment. Typically users initialize their environment when they log in by setting environment information for every application they will reference during the session. The HPCC uses the Environment Modules package as a tool to simplify shell initialization and allow users the ability to easily modify their environment during the session with module files.

For more information about how to load and maintain your software environment using modules, please refer to the user guide «Software Environment Setup».

In order to select a distribution and set the environment variables for Python you will first need to check which versions of Python are available using the «module spider python» command. This command will return a description of the software and which versions are currently installable through the modules system.

The default version of python at this time loaded by «module load intel python» or «module load gnu python» is Python 2. To load the most recent available system-installed version of python you can run «module load intel python3» or «module load gnu python3» command. You can then run the «module list» command to verify that the Python module has been successfully loaded. This can be done using one of the following sets of commands:

#To load the Intel compiled version of Python run the following:
module load intel python # For older version

# OR
module load intel python3 # for newer version

# check which is loaded
module list
python —version

# OR

#To load the GNU compiled version of R run the following:
module load gnu python # For older version

# OR
module load gnu python3 # for newer version

# check which is loaded
module list
python —version

In general, the Intel distribution of Python is more fully featured and has tools available to improve multi-threaded coding, which can be important to optimize the performance of long-running Python applications. The Gnu distribution may be more portable for moving your application to other systems, as it does not require a commercial license. Most Python code will function equally well with either distribution. You can alternatively use the Conda package manager as described below to install a custom version of Python and manage packages in user-controlled environments.

A good starting point if you don’t know how to choose from the above options is Python 3 in the Intel distribution, which can be selected using «module load intel python3».

Installing packages locally to supplement or replace a system distribution

By default, Python packages require the installation be performed by the ‘root’ user. However, most python package installers and managers will also allow the user to install the package into their HOME folder to supplement the features of these distributions.

You can alternately choose to use Anaconda or Miniconda through the Conda package manager, which will enable you to use a newer distribution of Python than the system defaults. Using the Conda package manager allows you total control over the Python setup for your code without any dependencies on system Python versions.

Below you will find instructions for installing python packages locally using the following methods:

Installing Packages Locally with pip

Pip is the PyPA (Python Packaging Authority) recommended tool for installing Python packages. As such, many Python packages can be installed using pip. The problem with pip is that if you attempt to install a package without root access, the tool will simply fail and give you no hint that you can install the package without root. In order to bypass the need for root access you can instuct pip to instead install to your HOME folder by adding the —user option as shown below:

for python 2: pip install —user

for python 3: pip3 install —user

Installing Packages Locally with easy_install

Easy_install is another commonly used tool for installing Python packages and is a supported method for the installation of many packages. Similar to pip, this tool will also fail if you attempt to install a package without root access. Unlike pip, when easy_install fails it does hint that it is possible to install without root but it does not give you the command to make it work. In order to bypass the need for root access you can instruct easy_install to instead install to your HOME folder by adding the —user option as shown below:

Installing Packages Locally from Source

Many Python packages also allow the user to install them directly from their source code. Often these packages provice a «setup.py» file for the purpose of installing these packages. Much like pip and easy_install, the additional of a —user flag is often sufficient for setup.py to install the package directly into your HOME folder. This can be done using the command shown below:

python setup.py install --user

Installing Packages using the Conda Package Manager

Some users find themselves needing to install a different version of a package that the HPCC has installed, which will then often override the version you attempt to install into your local HOME directory. Users also sometimes encounter Python packages with an immense number of dependencies that make installing them difficult. If you find yourself needing to install a complex Python package, a package version different from the one we provided or if you simply need a specific version of Python then we strongly suggest you install a copy of the Conda package manager into your HOME folder. The Conda package manager will allow you to fully control your Python environment and often makes the installation of complex Python workloads as simple as a few Conda commands. The HPCC provides documentation for the installation and usage of Conda, which can be found here: Installing a local copy of Python

High Performance Computing Center

Источник

Installing Packages¶

This section covers the basics of how to install Python packages .

It’s important to note that the term “package” in this context is being used to describe a bundle of software to be installed (i.e. as a synonym for a distribution ). It does not to refer to the kind of package that you import in your Python source code (i.e. a container of modules). It is common in the Python community to refer to a distribution using the term “package”. Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or another larger software distribution like Python itself.

Requirements for Installing Packages¶

This section describes the steps to follow before installing other Python packages.

Ensure you can run Python from the command line¶

Before you go any further, make sure you have Python and that the expected version is available from your command line. You can check this by running:

You should get some output like Python 3.6.3 . If you do not have Python, please install the latest 3.x version from python.org or refer to the Installing Python section of the Hitchhiker’s Guide to Python.

If you’re a newcomer and you get an error like this:

>>> python3 --version Traceback (most recent call last): File "", line 1, in NameError: name 'python3' is not defined 

It’s because this command and other suggested commands in this tutorial are intended to be run in a shell (also called a terminal or console). See the Python for Beginners getting started tutorial for an introduction to using your operating system’s shell and interacting with Python.

If you’re using an enhanced shell like IPython or the Jupyter notebook, you can run system commands like those in this tutorial by prefacing them with a ! character:

In [1]: import sys ! --version Python 3.6.3

It’s recommended to write rather than plain python in order to ensure that commands are run in the Python installation matching the currently running notebook (which may not be the same Python installation that the python command refers to).

Due to the way most Linux distributions are handling the Python 3 migration, Linux users using the system Python without creating a virtual environment first should replace the python command in this tutorial with python3 and the python -m pip command with python3 -m pip —user . Do not run any of the commands in this tutorial with sudo : if you get a permissions error, come back to the section on creating virtual environments, set one up, and then continue with the tutorial as written.

Ensure you can run pip from the command line¶

Additionally, you’ll need to make sure you have pip available. You can check this by running:

If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip. If you’re on Linux and installed using your OS package manager, you may have to install pip separately, see Installing pip/setuptools/wheel with Linux Package Managers .

If pip isn’t already installed, then first try to bootstrap it from the standard library:

python3 -m ensurepip --default-pip

py -m ensurepip --default-pip

If that still doesn’t allow you to run python -m pip :

• Securely Download get-pip.py1
• Run python get-pip.py . 2 This will install or upgrade pip. Additionally, it will install setuptools and wheel if they’re not installed already.

Warning Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state. You can use python get-pip.py —prefix=/usr/local/ to install in /usr/local which is designed for locally-installed software.

Ensure pip, setuptools, and wheel are up to date¶

While pip alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools and wheel projects are useful to ensure you can also install from source archives:

python3 -m pip install --upgrade pip setuptools wheel

py -m pip install --upgrade pip setuptools wheel

Optionally, create a virtual environment¶

See section below for details, but here’s the basic venv 3 command to use on a typical Linux system:

python3 -m venv tutorial_env source tutorial_env/bin/activate

py -m venv tutorial_env tutorial_env\Scripts\activate

This will create a new virtual environment in the tutorial_env subdirectory, and configure the current shell to use it as the default python environment.

Creating Virtual Environments¶

Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally. If you are looking to safely install global command line tools, see Installing stand alone command line tools .

Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python3.6/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.

Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.

In all these cases, virtual environments can help you. They have their own installation directories and they don’t share libraries with other virtual environments.

Currently, there are two common tools for creating Python virtual environments:

• venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.
• virtualenv needs to be installed separately, but supports Python 2.7+ and Python 3.3+, and pip , setuptools and wheel are always installed into created virtual environments by default (regardless of Python version).

The basic usage is like so:

python3 -m venv source /bin/activate

Источник