While experimenting I saw some stuff, probably because I lack information/understanding, but I want to log these anyway:
There were many warnings, I wanted to get rid of them using Cluster(silence_logs=logging.ERROR) but it did not. So I had to use warnings.filterwarnings("ignore").
When viewing the profile tab, the bars show “threading” although I use processes with futures (none mention processes).
When a complete run finishes, the web server dies, so you cannot go back to another tab and examine the result. Perhaps there is an option for that that I didn’t see?
The webserver that provides the above visualization webapp (Dashboard), default serving at port 8787. I think it is based on a Bokeh Server. Naturally, this webserver closes when my process finishes (e.g. after many hours of execution where I naturally do not sit in front of that computer to examine the graphs). When it finishes, then you cannot browse the results.
I though about this but I was expecting a single mention of “processpoolexecutor” which spawns those threads. But there is only “threadpoolexecutor” and other thread stuff. I’m sure it works in processes as they consume the core, but anyway, it seemed misleading.
I had a look of it, but could not see any option for keeping the server still alive after the processes finishes, if I’m not missing something. You would probably not need these after your code is finished, but the Dashboard is of tremendous help during multi-process design and coding.
