Multiprocessing Manager Example in Python
You can share objects among processes using a manager.
In this tutorial you will discover how to use managers to share access to centralized Python objects.
What Is a Manager
A manager in the multiprocessing module provides a way to create Python objects that can be shared easily between processes.
Managers provide a way to create data which can be shared between different processes, including sharing over a network between processes running on different machines. A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.
— multiprocessing — Process-based parallelism
A manager creates a server process that hosts a centralized version of objects that can be shared among multiple processes.
The objects are not shared directly. Instead, the manager creates a proxy object for each object that it manages and the proxy objects are shared among processes.
The proxy objects are used and operate just like the original objects, except that they serialize data, synchronize and coordinate with the centralized version of the object hosted in the manager server process.
A proxy is an object which refers to a shared object which lives (presumably) in a different process. […] A proxy object has methods which invoke corresponding methods of its referent (although not every method of the referent will necessarily be available through the proxy).
— multiprocessing — Process-based parallelism
This makes managers a process-safe and preferred way to share simple data structures like lists and dicts among processes.
They are also a preferred way to share concurrency primitives among processes, specifically among workers in a multiprocessing pool.
You can learn more about multiprocessing managers in the tutorial:
Next, let’s look at how we might create and use a manager.
Run your loops using all CPUs, download my FREE book to learn how.
How to Create and Use a Manager
A multiprocessing.Manager provides a way to create a centralized version of a Python object hosted on a server process.
Once created, it returns proxy objects that allow other processes to interact with the centralized objects automatically behind the scenes.
The multiprocessing.Manager provides the full multiprocessing API, allowing Python objects and concurrency primitives to be shared among processes.
This includes Python objects we may want to share, such as:
It include shared ctypes for primitive values, such as:
It includes concurrency primitives for synchronizing and coordinating processes, such as:
It even includes patterns and process-safe data structures, such as:
First, we must create a manager instance.
Multiprocessing Manager to Share an Object with Processes
You can use a Manager to host a centralized Python object that can be shared with multiple processes that is both process-safe and changes to the object are propagated and made available to all processes seamlessly.
In this tutorial you will discover how to use a Manager to share an ad hoc Python object with multiple processes.
Need Manager to Share Queue
A manager in the multiprocessing module provides a way to create Python objects that can be shared easily between processes.
Managers provide a way to create data which can be shared between different processes, including sharing over a network between processes running on different machines. A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.
— multiprocessing — Process-based parallelism
A manager creates a server process that hosts a centralized version of objects that can be shared among multiple processes.
The objects are not shared directly. Instead, the manager creates a proxy object for each object that it manages and the proxy objects are shared among processes.
The proxy objects are used and operate just like the original objects, except that they serialize data, synchronize and coordinate with the centralized version of the object hosted in the manager server process.
A proxy is an object which refers to a shared object which lives (presumably) in a different process. […] A proxy object has methods which invoke corresponding methods of its referent (although not every method of the referent will necessarily be available through the proxy).
— multiprocessing — Process-based parallelism
This makes managers a process-safe and preferred way to share Python objects among processes.
You can learn more about multiprocessing managers in the tutorial:
When using multiprocessing, we may need to share an arbitrary Python object with child processes.
It may be a data structure or similar object where the internal state of the object may need to be updated from multiple child processes concurrently, although in a process-safe manner in order to avoid race conditions.
How can we use a multiprocessing Manager to share a Python object with child processes?
Run your loops using all CPUs, download my FREE book to learn how.
How to Use a Manager to Share an Object with Processes
We can use a multiprocessing manager to share an ad hoc Python object with multiple processes.
This requires a few steps:
- Defining a custom Manager class that extends BaseManager.
- Registering the Python class with the custom manager.
- Creating an instance of the custom manager.
- Creating an instance of the Python object from the manager instance.
Let’s take a closer look at each step in turn.
Step 1. Define a Custom Manager
We must define a custom manager in order to use the manager to create and manage ad hoc Python objects.
The custom class does not need to override the constructor or any methods, nor define any functionality.