Python site package ubuntu

Defaulting to user installation because normal site-packages is not writeable

When installing Python packages using pip , you might encounter the following message in the console:

This message appears when pip fails to install the package in the system-wide directory.

This is not an error message as pip can still install the package, but you might have problems like unable to access the package from your terminal as follows:

This tutorial explains why this message occurs and how you can make it disappear in practice.

Why this message occurs

By default, pip tries to install your package in the system-wide directory named site-packages .

When you lack permission to write to system-wide site-packages , then pip will install it in the user-wide site-packages instead.

You can verify this by running python -m site command from the terminal. There should be multiple site-packages directory as shown below. The screenshot is from a Macbook:

The one under /Applications/Xcode.app is the system-wide directory that requires root permission to write.

The USER_SITE one is where pip installs packages when the “Defaulting to user installation” message appears.

How to solve for Windows users

If you’re using Windows, then this message likely occurs because you installed Python under C:\Program Files which requires administrator permission to modify.

To remove the message, you need to add the —user flag when installing packages:

If you want to permanently fix this, you need to uninstall Python from your computer and install it again using the default settings.

If you need to customize Python installation, then take note of the default installation path and use it when you customize the install location:

By using the location provided in Python default installation, you will have Python installed in a folder that’s writeable without admin permission.

Also, don’t forget to check the options ‘Use admin privileges’ and ‘Add python.exe to PATH’.

Try running pip install command again and you won’t receive the message this time.

How to solve for Linux users

If you’re using Linux, then you will get this message because you installed python using sudo as in:

In Linux-based OS, the system-wide packages directory is also known as dist-packages , and you need root permission to access this directory.

Again, you can verify this using the python -m site command:

When pip fails to install to the dist-packages directory, the message appears to notify you that it’s installing to the site-packages directory instead.

To remove the message, you need to add the —user flag when installing packages:

This command tells pip to skip the dist-packages directory and go to site-packages directly.

If you want to install to dist-packages , then you need to add sudo to the install command:

But you might get a warning message in the console as follows:

   If you want to permanently fix this message, then you need to install Python without using sudo . This means you can’t use apt-get either.

You can get Python in Linux using Homebrew, or you can download the source and compile it yourself.

Follow the steps below to compile Python manually. Note that you can get the full path to Python .tgz file from here:

The make command might take a while to complete. Once done, you should now have python3 and pip3 available from your user directory.

Use the which command to verify the installation:

 Alright. Now you can install packages without using sudo command:

Because you installed Python and pip without using sudo , the default installation will use the site-packages directory.

Keep in mind that installing Python from source like this enables Python only for the current user. You need to repeat the process for other users.

How to solve for Mac users

If you’re using a Mac, this message appears when you use Python and pip that are provided by the system.

It’s recommended to install your own Python version to avoid permission problems and future changes from the macOS itself.

You can install Python using Homebrew as follows:

Once installed, you should open a new Terminal window for the changes to take affect.

You can see the active python3 and pip3 versions using the which command as follows:

 Once both python3 and pip3 are using the Homebrew-sourced version, you’re good to go.

Install any packages using pip install and you won’t receive the error message.

Conclusion

Now you’ve learned how to solve the “Defaulting to user installation” message that occurs when installing Python packages.

I hope this tutorial is helpful. Until next time! 👋

Take your skills to the next level ⚡️

I’m sending out an occasional email with the latest tutorials on programming, web development, and statistics. Drop your email in the box below and I’ll send new stuff straight into your inbox!

About

Hello! This website is dedicated to help you learn tech and data science skills with its step-by-step, beginner-friendly tutorials.
Learn statistics, JavaScript and other programming languages using clear examples written for people.

Search

Type the keyword below and hit enter

Tags

Click to see all tutorials tagged with:

Источник

How does python find packages?

I just ran into a situation where I compiled and installed Python 2.7.9 from source on Ubuntu, but Python could not find the packages I had previously installed. This naturally raises the question — how does Python know where to find packages when you call import ? This post applies specifically to Python 2.7.9, but I’m guessing Python 3x works very similarly.

In this post I first describe how Python finds packages, and then I’ll finish with the discovery I made regarding the default Python that ships with Ubuntu and how it differs from vanilla Python in how it finds packages.

sys.path

Python imports work by searching the directories listed in sys.path .

Using my default Ubuntu 14.04 Python:

> import sys > print '\n'.join(sys.path) /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages

So Python will find any packages that have been installed to those locations.

How sys.path gets populated

As the docs explain, sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

You can read more about sys.path in the Python docs.

Assuming your PYTHONPATH environment variable is not set, sys.path will consist of the current working directory plus any manipulations made to it by the site module.

The site module is automatically imported when you start Python, you can read more about how it manipulates your sys.path in the Python docs.

You can manipulate sys.path

You can manipulate sys.path during a Python session and this will change how Python finds modules. For example:

import sys, os # This won't work - there is no hi module import hi Traceback (most recent call last): File "", line 1, in module> ImportError: No module named hi # Create a hi module in your home directory. home_dir = os.path.expanduser("~") my_module_file = os.path.join(home_dir, "hi.py") with open(my_module_file, 'w') as f: f.write('print "hi"\n') f.write('a=10\n') # Add the home directory to sys.path sys.path.append(home_dir) # Now this works, and prints hi! import hi print hi.a 

The module __file__ attribute

When you import a module, you usually can check the __file__ attribute of the module to see where the module is in your filesystem:

> import numpy > numpy.__file__ '/usr/local/lib/python2.7/dist-packages/numpy/__init__.pyc' 

The file attribute is not present for C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file.

So, for example this doesn’t work:

> import sys > sys.__file__ Traceback (most recent call last): File "", line 1, in module> AttributeError: 'module' object has no attribute '__file__' 

It makes sense that the sys module is statically linked to the interpreter — it is essentially part of the interpreter!

The imp module

Python exposes the entire import system through the imp module. That’s pretty cool that all of this stuff is exposed for us to abuse, if we wanted to.

imp.find_module can be used to find a module:

> import imp > imp.find_module('numpy') (None, '/usr/local/lib/python2.7/dist-packages/numpy', ('', '', 5)) 

You can also import and arbitrary Python source as a module using imp.load_source . This is the same example before, except imports our module using imp instead of by manipulating sys.path :

import sys, os, imp # Create a hi module in your home directory. home_dir = os.path.expanduser("~") my_module_file = os.path.join(home_dir, "hi.py") with open(my_module_file, 'w') as f: f.write('print "hi"\n') f.write('a=10\n') # Load the hi module using imp hi = imp.load_source('hi', my_module_file) # Now this works, and prints hi! import hi print hi.a # a is 10! print type(hi) # it's a module! 

Passing ‘hi’ to imp.load_source simply sets the __name__ attribute of the module.

Ubuntu Python

Now back to the issue of missing packages after installing a new version of Python compiled from source. By comparing the sys.path from both the Ubuntu Python, which resides at /usr/bin/python , and the newly installed Python, which resides at /usr/local/bin/python , I could sort things out:

Ubuntu Python ( /usr/bin/python ):

>>> import sys >>> print '\n'.join(sys.path) /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages

Python compiled from source ( /usr/local/bin/python )

>>> import sys >>> print '\n'.join(sys.path) /usr/local/lib/python27.zip /usr/local/lib/python2.7 /usr/local/lib/python2.7/plat-linux2 /usr/local/lib/python2.7/lib-tk /usr/local/lib/python2.7/lib-old /usr/local/lib/python2.7/lib-dynload /usr/local/lib/python2.7/site-packages

Turns out what mattered for me was dist-packages vs. site-packages . Using Ubuntu’s Python, my packages were installed to /usr/local/lib/python2.7/dist-packages , whereas the new Python I installed expects packages to be installed to /usr/local/lib/python2.7/site-packages . I just had to manipulate the PYTHONPATH environment variable to point to dist-packages in order to gain access to the previously installed packaged with the newly installed version of Python.

How did Ubuntu manipulate the sys.path ?

So how does the Ubuntu distribution of Python know to use /usr/local/lib/python2.7/dist-packages in sys.path ? It’s hardcoded into their site module! First, find where the site module code lives:

> import site > site.__file__ '/usr/lib/python2.7/site.pyc' 

Here is an excerpt from Ubuntu Python’s site.py , which I peeked by opening /usr/lib/python2.7/site.py in a text editor. First, a comment at the top:

For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python /dist-packages, Debian addons install into /usr//python /dist-packages. /usr/lib/python /site-packages is not used.

OK so there you have it. They explain how the Debian distribution of Python is different.

And now, for the code that implementes this change:

def getsitepackages(): """Returns a list containing all global site-packages directories (and possibly site-python). For each directory present in the global ``PREFIXES``, this function will find its `site-packages` subdirectory depending on the system environment, and will return a list of full paths. """ sitepackages = [] seen = set() for prefix in PREFIXES: if not prefix or prefix in seen: continue seen.add(prefix) if sys.platform in ('os2emx', 'riscos'): sitepackages.append(os.path.join(prefix, "Lib", "site-packages")) elif os.sep == '/': sitepackages.append(os.path.join(prefix, "local/lib", "python" + sys.version[:3], "dist-packages")) sitepackages.append(os.path.join(prefix, "lib", "python" + sys.version[:3], "dist-packages")) else: sitepackages.append(prefix) sitepackages.append(os.path.join(prefix, "lib", "site-packages")) if sys.platform == "darwin": # for framework builds *only* we add the standard Apple # locations. from sysconfig import get_config_var framework = get_config_var("PYTHONFRAMEWORK") if framework: sitepackages.append( os.path.join("/Library", framework, sys.version[:3], "site-packages")) return sitepackages 

It’s all there, if you are crazy enough to dig this deep.

© Lee Mendelowitz – Built with Pure Theme for Pelican

Источник