Python import path environment

How To Set Python Module Search Path To Find Modules

When you want to import a python module library in your python source code, you need first to make the python module library importable by adding the module package path in the PYTHONPATH system environment variable. You can also add the python module package path to the python module search path at runtime in python source code. This example will show you how to do it.

1. Add Python Module Package Path In System Environment Variable PYTHONPATH.

Suppose your python module is saved in folder /tmp. We will add /tmp folder in the PYTHONPATH environment variable value.

Open a terminal and go to the user home directory use cd ~ command.

$ ls -al . -rw-r--r--@ 1 zhaosong staff 1176 Apr 30 09:15 .bash_profile .

2. Display Python Library Search Path In Python Source Code.

Run into python interactive console in a terminal.

$ python3 Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>

>>> import sys >>> for path in sys.path: . print(path) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa

3. Append Directory To Python Library Search Path.

1. Python sys.path.append function can append directory to the end of python library search directory.

>>> sys.path.append('/abc') >>> >>> >>> for line in sys.path: . print(line) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa /abc

4. Append Exist Module Library Directory To Python Library Search Directory.

1. Python module’s __file__ attribute returns the module file saved directory. You can append that directory to the python library search path as below.

>>> import sys, os # os.__file__ will return the os module directory. >>> sys.path.append(os.__file__) >>> >>> >>> for line in sys.path: . print(line) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa /abc /Users/zhaosong/anaconda3/lib/python3.6/os.py

5. Question & Answer.

5.1 How can I add another python source directory in the Python search path for a large python project.

1. My team just handle an old python project from another team, the python project is so large, there are a lot of python source files in the python project. Our development environment is Linux and no IDE, only in the command line. And when I run the python script with the command python abc.py it prompts the error ImportError: no module named com.test_module. And there is a lot of such kind of errors in other python scripts. All the old python project files are saved in a folder like /codebase/old_python_project. And there are a lot of subfolders in the project base folder. How can I make python search all the modules in the project folder and it’s subfolders to fix the import error? Thanks.
2. You can add your existing python project folder in the PYTHONPATH system environment variable to fix your issue. You can also use the function sys.path.append(‘/codebase/old_python_project’) in your python script source code and then can import the python modules. If you want to add all the project subfolders in the python module search path, you can reverse loop your project folder and when you reach it’s subfolder then you can call the sys.path.append() function to add the subfolder to the python module search path, you can try it.

import os, sys def reverse_add_python_module_search_path(module_dir): files_array = [] files_array = os.listdir(module_dir) for file in files_array: if os.path.isdir(file): sys.path.append(file) reverse_add_python_module_search_path(file)

Источник

How to import local modules with Python

Importing files for local development in Python can be cumbersome. In this article, I summarize some possibilities for the Python developer.

TL; DR: I recommend using python -m to run a Python file, in order to add the current working directory to sys.path and enable relative imports.

Definitions

Script: Python file meant to be run with the command line interface.

Module: Python file meant to be imported.

Package: directory containing modules/packages.

Example

Consider the following project structure:

project ├── e.py └── src ├── a │ └── b.py └── c └── d.py 

Say that in b.py , we want to use d.py :

Let’s try to run python from the project directory:

~/Documents/code/project$ python3 src/a/b.py Traceback (most recent call last): File "/home/qfortier/Documents/code/project/src/a/b.py", line 4, in import src.c.d ModuleNotFoundError: No module named 'src' 

What happened? When Python tries to import a module, it looks at the paths in sys.path (including, but not limited to, PYTHONPATH):

~/Documents/code/project$ python3 src/a/b.py ['/home/qfortier/Documents/code/project/src/a', '/usr/lib/python38.zip', '/usr/lib/python3.8', . ] 

We see that the directory containing b.py (the script we run) is added to sys.path . But d.py is not reachable from the directory a , hence the above error.

1st solution: add root to sys.path

We can add the path to the root of the project:

~/Documents/code/project$ python3 src/a/b.py ['/home/qfortier/Documents/code/project/src/a', . '.'] 

The added path ‘.’ refers to the current working directory (from which Python was run) ~/Documents/code/project. Python can indeed import c.d from this directory.

This solution works regardless of the directory used to run Python:

~/Documents/code/project$ cd ../.. ~/Documents$ python3 code/project/src/a/b.py ['/home/qfortier/Documents/code/project/src/a', . 'code/project'] 

It should also work on a different computer, with a different filesystem or OS, thanks to Pathlib.
However, modifying sys.path at the beginning of every file is tedious and hard to maintain. A better solution would be to use a context.py file modifying sys.path and imported by every other file, but this is still unsatisfying.
Another possibility is to add the root directory to PYTHONPATH (before running the script). This is done automatically by poetry , for example.

Relative import

Relative imports were introduced in PEP 328 as a way to improve maintainability and avoid very long import statements. They must use the from .module import something syntax:

~/Documents/code/project$ python3 src/a/b.py Traceback (most recent call last): File "src/a/b.py", line 4, in from ..c import d ImportError: attempted relative import with no known parent package 

This is not working! Indeed, relative imports rely on the __name__ or __package__ variable (which is __main__ and None for a script, respectively).
If we import b.py from another file, everything is fine:

~/Documents/code/project$ python3 e.py Name: src.a.b Package: src.a 

Note: to go up several directories in relative imports, use additional dots: from . c import d would go 2 directories up, for example.

2nd solution: run as a module

It is unfortunate that scripts can’t use relative imports. PEP 338 overcomes this limitation by adding the -m option. With this option, a script is run as if it was imported as a module.

~/Documents/code/project$ python3 -m src.a.b # note that we use . and not / here Name: __main__ Package: src.a 

python3 -m src.a.b acts as if a Python file containing import src.a.b was run in the current directory, with two notable differences:

• the __package__ variable is filled, enabling relative imports (see PEP 366)
• the directory from which Python was run (here: ~/Documents/code/project) is added to sys.path

Since I don’t see any drawback to this, I recommend always using python3 -m to run a script.

Run as a module on Visual Code

The default launch configuration in Visual Code runs Python files as scripts (without -m ). This GitHub issue explains how to add -m automatically:

• add the “Command Variable” extension to Visual Code
• set the following Python launch configuration in your settings.json:

3rd solution : modify PYTHONPATH

With Visual Code, you can automatically add the root directory to your project by adding a line to the launch configuration :

Alternatively, you can add a .env file in the root directory :

4rd solution (outdated): install in editable mode

pip install -e . installs a package in editable mode, which can then be imported anywhere on the computer. In practice, this is essentially a symbolic link to your package. Therefore, any modification of the package is reflected on the code importing it. It requires a setup.py at the root of your package.

Note: according to PEP 517, this is no longer recommended: Python packages should rely on a toml file (see Poetry) and not on setup.py anymore.

References

Updated: May 9, 2021

Источник