Solved: how to send jobs to queue dynamically

The main problem with dynamically sending jobs to a queue is that it can be difficult to determine when the jobs should be sent. If the queue is too large, the jobs may not be sent until after the deadline has passed. If the queue is too small, the jobs may not be sent at all.

I have a Python program that needs to process a large number of files. Each file can be processed independently, so I would like to take advantage of parallel processing by sending each file to a job queue.
I would like the program to be able to dynamically add new jobs to the queue as they become available, without having to wait for all the current jobs in the queue to finish first. Is this possible? If so, how can I do it?


A:

You could use <code>multiprocessing</code>'s <code>Pool</code> class:
<blockquote>
<p><strong><a href="https://docs.python.org/3/library/multiprocessing.html#the-process-pool" rel="nofollow noreferrer">17.5.1 The Process Pool</a></strong></p>
<p>[...]</p>
<pre><code>&lt;code&gt;class multiprocessing.pool.Pool(processes=None, initializer=None, initargs=(), maxtasksperchild=None)  # noqa E501

    Returns a process pool object.</code&gt;</code></pre>
<p>[...]</p>
<pre><code>&lt;code&gt;apply(func[, args[, kwds]]) 

    Equivalent of `func(*args, **kwds)`. It blocks until the result is ready.</code&gt;</code></pre>
<p>[...]</p>
<pre><code>&lt;code&gt;apply_async(func[, args[, kwds[, callback]]]) 

    A variant of the `apply()` method which returns an `AsyncResult` object.</ code &gt;</code></pre>
<p>[...] </p>
<pre><code>&lt;code&gt;map(func, iterable[, chunksize]) 

    A parallel equivalent of `map()`. It blocks until the result is ready.</ code &gt;</code></pre>
<p>[...] </p>
<pre><co de >map_async(func , iterable [ , chunksize [ , callback ] ] )  

     A variant of the map() method which returns an AsyncResult object.</ co de &gt; </ pre > < p > [ ... ] < / p > < pre >< code &gt ; imap ( func , iterable &#91; , chunksize &#93; )  

     A lazy version of map(). The chunksize argument is the same as in get(). </ code & gt ; < / pre > < p > [ ... ] < / p > < pre >< code & gt ; imap_unordered ( func , iterable &#91; , chunksize &#93; )  

     A lazy version of map(). The chunksize argument is the same as in get(). </ code & gt ; < / pre >  </blockquote>. . . . . . . . . .. .. .. .. .. .. .. ... ....... ....... ................. ................. ...................... ...................... ...................... ...................... .......... .......... .......... .......... .......... .......... ............ ............ .............. .............. .............. .............. .............. ...... ...... ...... ...... ...... ...... ....... ....... ....... ....... ....... ......... ............ ............ ............ ............ ............. .................. .................. .................. .................. ....................... ................................. ............................................... ………… ………… ………… ………… ………………. ………………. ………………. ……………….……………………………………………………………………………………………………………………………………………………………………………..                                                                                                                                                               

Queues

A queue is a data structure that allows elements to be added to the front and removed from the back. Queues are often used in programming languages to allow programs to work with collections of data.

When to use a queue

A queue is a data structure that allows elements to be added to the front and removed from the back. Queues are often used when there is a need to wait for a certain number of elements to be processed before doing something else.