Hello everyone! I was curious about something I noticed while staring at htop on my machine while working on this project. I noticed that performing that work was taking a fair bit longer than I had anticipated on longer videos and, until I get a better understanding of how task-graphs are being ex…

To summarize that post, it’s basically that: Python GIL is not released for the threaded operations of reading video files. One question that I think would be more appropriate in this thread is this: Is there a way to tell generally when something can benefit from threading vs processing? It’s b…

Since numpy often releases the GIL, threading should accordingly lead to performance gains. But in your case, there is also a python for-loop (looping the function hog() over the frames in each chunk) which probably interferes with the full release of the GIL, and which is probably why the multi-pr…

Got it, so that would be why altering how hog() in that case works so it can operate on entire chunks at once instead of on a frame by frame basis would maybe allow threading to be used. Is it a general rule that threads are faster than processes overall?

Exactly. At least, that’s how I see it. :smile: As far as I understand it, multi-processing generally incurs an overhead when processes communicate with each other in order to share data. This overhead is typically absent in multi-threading, as different threads are able to share/access the same …

Understanding How Dask is Executing Processes vs Threads

Dask Array

ParticularMiner April 30, 2022, 4:51pm 2

@jmdelahanty

See this post.

Topic		Replies	Views
Memory management in multi process vs thread-based parallelism Distributed	3	583	December 17, 2021
How does Dask avoid GIL in multi threading? Distributed	2	829	May 22, 2023
Is there a way to use threads / processes exclusively for a code block delayed	1	442	August 22, 2022
Why is 'processes' executing my code sequentially? Dask DataFrame	2	163	September 6, 2024
One output time vs multiple output time Deploying Dask delayed	1	277	April 19, 2022

Understanding How Dask is Executing Processes vs Threads

Related topics