Can threads create sub-threads in Python?
I realize python threads are subject to the interpreter global lock, but my application involves access to a server, so the multi threading is to avoid serialized connections which would take too long.
Yes, spawning threads from within another thread is OK. Just remember that there exists a main Python thread, that governs the others.
I don’t see why spawning new threads from a thread would be an issue, actually the first thread you spawn is already a sub thread of the main thread. Now I think you are confusing threads and processes given your 2nd question, every thread is subordinated to the process that spawns them, and though for the OS it does not really matter which thread «owns» which, on a design perspective, it is always better to have subthreads owned by the thread which started them
@asettouf The second question comes from observing htop with tree view. The threads are all branches of the python script. And I believe this affects what threads get killed if i kill another. (I’ve just edited to clarify what I mean)
1 Answer 1
So regarding your questions:
- Q1: It is not a problem to start «subthreads» from a thread
- Q2: It is actually an interesting question, my instinct would say «no», but getting a proof sounds better to me
So I created a quick test as below (I would use a gist but I can’t access such things from where I am):
from threading import Thread import time def sub_worker(id): print("SubWorker started from thread", id) while True: print("Subworking. ") time.sleep(5) def worker(id): print("Worker started from thread", id) count = 1 while count < 5: print("Working. ") tmp_thread = Thread(target=sub_worker, args=[count]) tmp_thread.start() count +=1 time.sleep(1) raise EnvironmentError("Tired of working") main = Thread(target=worker, args=[0]) main.start() Which gives us the output (as expected an error in the parent thread does not stop the "children"):
Worker started from thread 0 Working. SubWorker started from thread 1 Subworking. Working. SubWorker started from thread 2 Subworking. Working. SubWorker started from thread 3 Subworking. Working. SubWorker started from thread 4 Subworking. Exception in thread Thread-1: Traceback (most recent call last): File "C:\Temp\tt\Tools\Anaconda3.4.3.1\lib\threading.py", line 916, in _bootstrap_inner self.run() File "C:\Temp\tt\Tools\Anaconda3.4.3.1\lib\threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "C:/Temp/tt/Tools/PyCharmWorkspace/xml_parse/test.py", line 18, in worker raise EnvironmentError("Tired of working") OSError: Tired of working Subworking. Subworking. Subworking. Subworking. Subworking. Subworking. Subworking. I think that htop shows this hierarchy might be due to the fact that threads are treated as processes by the Linux kernel. And since a call to fork is made it can shows this hierarchy. With the concept of threads, I do not believe that a hierarchy makes so much sense as each of them will share the same resources (memory, file descriptors . etc)
Spawning a thread from a thread
The problem is that proc_q1 appears to be called properly adding ids the t2_q, but the t2 threads are not picking up the tasks from the queue before the exit_flag is set to True . Why doesn't proc_q2 ever get called?
import urllib.request import threading from queue import Queue from xml.etree import ElementTree as ET from urllib.parse import urlparse class Scanner(object): @staticmethod def proc_q1(thread_id, q1, q2): while not exit_flag: t1_lck.acquire() if not q1.empty(): new_id = q1.get() t1_lck.release() # add id to t2_queue for threads2 to pick up t2_lck.acquire() t2_q.put(new_id) t2_lck.release() print('proc_q1 - new_id: <>'.format(new_id)) else: t1_lck.release() @staticmethod def proc_q2(thread_id, q): t2_lck.acquire() if not q.empty(): new_id = q.get() t2_lck.release() print('proc_q2 - new_id: <>'.format(new_id)) else: t2_lck.release() class MyT1(threading.Thread): def __init__(self, thread_id, q1, q2): threading.Thread.__init__(self) self.thread_id = thread_id self.q1 = q1 self.q2 = q2 def run(self): print('Starting t1-thread <>'.format(self.thread_id)) Scanner.proc_q1(thread_id=self.thread_id, q1=self.q1, q2=self.q2) print('Exiting t1-thread <>'.format(self.thread_id)) class MyT2(threading.Thread): def __init__(self, thread_id, q): threading.Thread.__init__(self) self.thread_id = thread_id self.q = q def run(self): print('Starting t2-thread <>'.format(self.thread_id)) Scanner.proc_q2(thread_id=self.thread_id, q=self.q) print('Exiting t2-thread <>'.format(self.thread_id)) # Create new threads thread_pool_size = 5 t1_list = list() t2_list = list() t1_q = Queue(2 * thread_pool_size) t2_q = Queue(2 * thread_pool_size) t1_lck = threading.Lock() t2_lck = threading.Lock() exit_flag = False for i in range(thread_pool_size): t1 = MyT1(thread_id=i, q1=t1_q, q2 = t2_q) t1.start() t1_list.append(t1) t2 = MyT2(thread_id=i, q=t2_q) t2.start() t2_list.append(t2) # put data on the queue t1_lck.acquire() for i in range(10): t1_q.put(i) t1_lck.release() # Wait for empty queue while not (t1_q.empty() and t2_q.empty()): pass exit_flag = True # wait for threads2 to finish for t in t2_list: t.join() # wait for threads1 to finish for t in t1_list: t.join() print("Done") And here's the output (the script doesn't exit correctly)
Starting t1-thread 0 Starting t2-thread 0 Exiting t2-thread 0 Starting t1-thread 1 Starting t2-thread 1 Exiting t2-thread 1 Starting t1-thread 2 Starting t2-thread 2 Starting t1-thread 3 Exiting t2-thread 2 Starting t2-thread 3 Starting t1-thread 4 Exiting t2-thread 3 Starting t2-thread 4 Exiting t2-thread 4 proc_q1 - new_id: 0 proc_q1 - new_id: 1 proc_q1 - new_id: 2 proc_q1 - new_id: 3 proc_q1 - new_id: 5 proc_q1 - new_id: 4 proc_q1 - new_id: 6 proc_q1 - new_id: 7 proc_q1 - new_id: 8 proc_q1 - new_id: 9 Python How to start Thread inside another one?
I'm trying to start multiple thread, and the start time of the next thread will depend on what It'll happend in the first and etc. So I found some post in Stackoverflow like this one in the answer he talk about the Event class: Python threading - How to repeatedly execute a function in a separate thread? So I try to do in that way :
from threading import Thread, Event import time class MyThread3(Thread): def __init__(self, event): Thread.__init__(self) self.stopped = event def run(self): while self.stopped.wait(0.5): print("The Third thread is running..") class MyThread2(Thread): def __init__(self, event): Thread.__init__(self) self.stopped = event def run(self): time.sleep(1) my_event2.clear() time.sleep(3) my_event.set() time.sleep(2) my_event2.set() class MyThread1(Thread): def __init__(self, event): Thread.__init__(self) self.stopped = event def run(self): while not self.stopped.wait(0.5): print("Thread is running..") my_event = Event() my_event2 = Event() thread1 = MyThread1(my_event) thread2 = MyThread2(my_event) thread3 = MyThread3(my_event2) thread1.start() thread2.start() thread3.start() thread1.join() thread2.join() thread3.join() If I put while not in the Thread3 it start at the same time as the Thread1, so why if I put the opposit and if the state of the Event change and it doesn't start ? We cannot change an Event inside a Thread ? How can I start a thread from another Thread ?
Creating Threads within a Thread in Python
I am using the threading libary and want to have one thread that will call several threads. The background to this program is that I have a camera which captures Image and makes them available in a class on a TCP-SocketServer. Thus I need one thread that runs the camera capturing and a second thread that runs the TCPServer, but within this Thread there are several Threads for each incoming connection. This last thread means I need a thread that can create threads on its own. Unfortunately this did not work. I managed to break down the immense code into a small snippet which represents the problem:
import threading def adder(x,res,i): res[i] = res[i] + x*i; def creator(a,threads,results): results = [] for i in range(0,a): results.append(0) threads.append(threading.Thread(target=adder,args=(a,results,i))) threads[i].start() for i in range(0,len(threads)): threads[i].join() return results; threads = []; results = []; mainThread = threading.Thread(target=creator,args=([5,threads,results])) mainThread.start() mainThread.join() for i in range(0,len(results)): print results[i] print threads[i] In the function creator which is called as a thread there should be several threads created with the funciton adder . However the results are empty, why is that so? This is the same problem that occurs in my larger program.
Python threads have no such method ( .terminate() ) - regardless of whether they've been started. See here: stackoverflow.com/questions/323972/…
Python threads have no .stop() method either - your edited code should be raising an exception about that now. See the link I already gave: Python supports no builtin way to force threads to stop.
Strangely it worked. However I now replaced the thread to one that will just have a simple task and wait for all the threads to join , however the results are still empty and I get now the notification that NoneType has no function called join
Calling a method in thread from another thread, python
How can I achieve communication between threads? I have one thread in which I do some stuff, then I need to call a method from an object that lives in the main program thread and this method should be executed in the main process:
class Foo(): def help(self): pass class MyThread(threading.Thread): def __init__(self, connection, parser, queue=DEFAULT_QUEUE_NAME): threading.Thread.__init__(self) def run(self): # do some work # here I need to call method help() from Foo() # but I need to call it in main process bar = Foo() my_work_thread = MyThread() my_work_thread.run() Both threads have access to the same memory, so bar in the separate thread is identical to bar in the main thread
Hard to tell, what you wanna do. What does help() do? Why you need to call it from the main process? Does help() depend ons some variables available only in the main process? Normally you can share variables between threads, just remember to make them thread-safe (locking)
1 Answer 1
There are many possibilities how to do it, one is using 2 queues:
from time import sleep import threading, queue class Foo(): def help(self): print('Running help') return 42 class MyThread(threading.Thread): def __init__(self, q_main, q_worker): self.queue_main = q_main self.queue_worker = q_worker threading.Thread.__init__(self) def run(self): while True: sleep(1) self.queue_main.put('run help') item = self.queue_worker.get() # waits for item from main thread print('Received ', item) queue_to_main, queue_to_worker = queue.Queue(), queue.Queue( ) bar = Foo() my_work_thread = MyThread(queue_to_main, queue_to_worker) my_work_thread.start() while True: i = queue_to_main.get() if i == "run help": rv = Foo().help() queue_to_worker.put(rv) Running help Received 42 Running help Received 42 Running help Received 42 . etc Great thanks! But, the only thing that embarrasses me: a while loop in the main process. Do we can to use something like subscribers? E.g.: when something appends to queue, then run function on_queue_append() ?
@Q-bart There aren't any subscribers in Python queue, but you can use get_nowait() just to peek if something is in the queue: docs.python.org/3/library/queue.html#queue.Queue.get