Python thread join with timeout

Let’s say that you want to start a thread and run some background processing there but also you want to stop the program when you press CTRL+C or set a time out. (if you want to close all threads scroll down to the end)

Just close the thread and don’t care what happens
Wait for the background thread to finish the current task and then return from it

Note

Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event . (from python documentation)

If we manage to close the main-thread all the daemon threads will be forced to close.

Using daemon threads for tasks that can be interrupted

1. Using try/except for KeyboardInterrupt

Notice that we don’t use join() , because is blocking, we can’t detect when CTRL+C has been pressed when we are blocked. Because of this we use is_alive() to check if the thread finished or not.

#!/usr/bin/env python3 import sys import threading from time import sleep def process(): """Long lasting operation""" sleep(300) print("finished work") def main(argv): t = threading.Thread(target=process) t.daemon = True t.start() try: while t.is_alive(): print("Waiting for background thread to finish") sleep(1) print("Thread finished task, exiting") except KeyboardInterrupt: print("Closing main-thread.This will also close the background thread because is set as daemon.") return 0 if __name__ == "__main__": sys.exit(main(sys.argv))

2. Using signal.signal to exit main thread

SIGINT is the signal for CTRL+C and the signal_handler is the function to be executed when the SIGINT is detected.

#!/usr/bin/env python3 import signal import sys import threading from time import sleep def process(): """Long lasting operation""" sleep(300) print("finished work") def signal_handler(signal, frame): print("Closing main-thread.This will also close the background thread because is set as daemon.") sys.exit(0) def main(argv): signal.signal(signal.SIGINT, signal_handler) t = threading.Thread(target=process) t.daemon = True t.start() while t.is_alive(): print("Waiting for background thread to finish") sleep(1) print("Thread finished task, exiting") return 0 if __name__ == "__main__": sys.exit(main(sys.argv))

3. Using join() to add a timeout and then close all threads (can’t catch CTRL+C)

The call to join() is blocking, we can’t catch CTRL+C, so we will see another example how to add timeout and catch CTRL+C. You can use this if in your context you don’t care about CTRL+C.

Don’t use join() on a daemon thread without timeout, by definition daemon threads are infinite loops but nobody stops you to mark a thread as daemon that is not an infinite loop.

#!/usr/bin/env python3 import sys import threading from time import sleep TIMEOUT = 10 def process(): """Long lasting operation""" sleep(12) print("finished work") def main(argv): """ This example has timeout using the join() method. The join() is blocking, meaning that an SIGINT event(CTRL+C) can't be detected. :param argv: :return: """ t = threading.Thread(target=process) t.daemon = True t.start() print(f"Waiting seconds for background thread to finish.") t.join(TIMEOUT) if t.is_alive(): print("Background thread timed out. Closing all threads") return -1 else: print("Background thread finished processing. Closing all threads") return 0 if __name__ == "__main__": sys.exit(main(sys.argv))

4. Using is_alive() for timeout and catching CTRL+C

#!/usr/bin/env python3 import sys import threading from time import sleep TIMEOUT = 10 def process(): """Long lasting operation""" sleep(12) print("finished work") def main(argv): """ This example combines timeout and KeyboardInterrupt. If detected KeyboardInterrupt all threads are closed. The background thread is close automatically because is set as daemon. :param argv: :return: """ t = threading.Thread(target=process) t.daemon = True t.start() print(f"Waiting seconds for background thread to finish.") try: loop_thread(t) except KeyboardInterrupt: print("KeyboardInterrupt detected closing all threads") def loop_thread(t): i = 1 while i  TIMEOUT: if t.is_alive(): print("Background thread processing, please wait.") sleep(1) i += 1 else: print("Background thread finished processing. Closing all threads") if i == TIMEOUT + 1: print("Timeout occurred.") if __name__ == "__main__": sys.exit(main(sys.argv))

Closing an non daemon thread gracefully (waiting for thread to finish work)

1. Using Event() to stop the thread gracefully

As stated in the python doc, this example uses an Event() which will be set when CTRL+C is pressed. The background thread will check this flag every time it gets a new item to process, if set will not get any new items and just return.

This thread is not marked as daemon meaning that the main-thread can’t close if this thread hasn’t finished executing.

If we do one time background job we can just have the code in the thread target function, if we want the thread to loop over some items we can use a queue.Queue()

#!/usr/bin/env python3 import queue import sys import threading from time import sleep stop = threading.Event() work_queue = queue.Queue() work_queue.put('Item 1') work_queue.put('Item 2') def process(): """Long lasting operation""" while not stop.isSet() and not work_queue.empty(): sleep(5) item = work_queue.get() # process item print(f"Done task: ") work_queue.task_done() if stop.isSet(): print("KeyboardInterrupt detected, closing background thread. ") def main(argv): t = threading.Thread(target=process) t.start() try: while t.is_alive(): print("Waiting for background thread to finish") sleep(1) except KeyboardInterrupt: stop.set() print("Closing main-thread.Please wait for background thread to finish the current item.") return 0 work_queue.join() print("Thread finished all tasks, exiting") return 0 if __name__ == "__main__": sys.exit(main(sys.argv))

Forcefully closing all threads os._exit(), even those that are not a daemon thread

#!/usr/bin/env python3 import os import queue import sys import threading from time import sleep work_queue = queue.Queue() work_queue.put('Item 1') work_queue.put('Item 2') def process(): """Long lasting operation""" while not work_queue.empty(): sleep(5) item = work_queue.get() # process item print(f"Done task: ") work_queue.task_done() def main(argv): t = threading.Thread(target=process) t.start() try: while t.is_alive(): print("Waiting for background thread to finish") sleep(1) except KeyboardInterrupt: print("Closing main-thread.Please wait for background thread to finish the current item.") if threading.activeCount() > 1: os._exit(getattr(os, "_exitcode", 0)) else: sys.exit(getattr(os, "_exitcode", 0)) work_queue.join() print("Thread finished all tasks, exiting") return 0 if __name__ == "__main__": sys.exit(main(sys.argv))

You could also look into concurrent.futures depending on the situation the code could be easier to write using concurrent.futures.

Consider use of timeout with threads all the time in production , if code shouldn’t take more than 10 seconds use timeout, is always better to see an error in a log file that threaded timed out than having the issue hidden for probably hours and then you start debugging, there is no need to lose time and money.

When to use threads? When you wait on I/O. If you do cpu intensive work look into subprocess module

If you want to learn more about threads and concurrency in python you can watch this youtube video by python core developer Raymond Hettinger.

Источник

How to Join a Thread in Python

You can join a thread by calling the Thread.join() function.

In this tutorial you will discover how to join threads in Python.

Need to Join a Thread

A thread is a thread of execution in a computer program.

Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create additional threads in our program in order to execute code concurrently.

Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.

You can learn more about Python threads in the guude:

In concurrent programming, we may need to wait until another thread has finished running. This may be for many reasons, such as:

The current thread needs a result from the target thread.
A resource is shared between the current and target threads.
The current thread has no other work to complete.

The join() method provides a way for one thread to block until another thread has finished.

How can we use the join() method to join a thread in Python?

Run your loops using all CPUs, download my FREE book to learn how.

How to Join a Thread

A thread can be joined in Python by calling the Thread.join() method.

This has the effect of blocking the current thread until the target thread that has been joined has terminated.

The target thread that is being joined may terminate for a number of reasons, such as:

Finishes executing it’s target function.
Finishes executing it’s run() method if it extends the Thread class.
Raised an error or exception.

Once the target thread has finished, the join() method will return and the current thread can continue to execute.

The join() method requires that you have a threading.Thread instance for the thread you wish to join.

This means that if you created the thread, you may need to keep a reference to the threading.Thread instance. Alternatively, you can use the threading.enumerate() function to enumerate through all active threads and locate the thread you wish to join by name.

The join() method also takes a “timeout” argument that specifies how long the current thread is willing to wait for the target thread to terminate, in seconds.

Once the timeout has expired and the target thread has not terminated, the join() thread will return.

Источник

Timeout Function in Python 3

Have you ever implemented a function that has to stop its execution after certain period of time? It is not as easy as it sounds, is it? I had to develop such functionality for a customer who had a requirement that one activity shouldn’t take more than 180 seconds to execute. There are two approaches to achieve this behavior. The first one is to use threads and the second one, to use processes. The second one is better in my opinion but let me first start with the threads.

Implementing timeout function with thread:

In order to implement the timeout function, we need one thread to execute the function and another to watch the time that it takes. When the time is over only the second thread knows it. If we could simply kill the function thread everything would work as expected but since they share the same execution context, we can’t. Threads can’t be killed so what we can do is to signal the other thread that it should stop. The drawback of this approach is that it can’t be used in all the cases. For example if we are using an external library inside that function, the execution might be stuck in a code that we don’t have access to. In this case we can’t guarantee that the function will stop exactly after the given period. But in most of the cases this approach is enough. In the first thread (the one that executes the function) we have to make regular checks if the time is over. We can use the Event object from the threading module in Python 3 to send a signal from one thread to another. Here is an example:

Источник