Proposal: Control.Concurrent.Async

[wren ng thornton]

On 6/9/12 5:28 PM, Simon Marlow wrote:
> All good points. I don't have many use cases for the waitAny* family yet
> so I'd been holding off refining the API until I had a clearer picture
> of what would be useful for clients. There are lots of possibilities
> here.

The way I typically use it is as follows: The main thread spawns off a 
bunch of heterogeneous workers--- the kind that interact with one 
another in an event-loop like manner (not of the fire-and-forget nor the 
web-server kind). But, the workers may fail/terminate for various 
reasons--- including thrown exceptions, but also more benign things like 
expected failure modes (encoded in normal ADTs or passed along via 
TChans/TVars/etc), and temporary failures that just require some 
'manual' help to get unstuck. After firing them off to do the work, the 
main thread's job is to monitor the workers in order to (a) handle any 
of the failure modes it can, (b) ensure clean and safe shutdown in the 
event of irrecoverable errors.

A related but somewhat different style of using waitAny* parallelism is 
for things like grid managers. Again, there are a bunch of heterogeneous 
worker threads. But this time, rather than focusing on errorful 
termination as above, the focus here is on proper termination. In 
particular, once some job A has finished, we can now start jobs B, C, 
and D which depend on the output of A. In the small scale, you can just 
have A finish by firing off the other jobs; but in the large scale that 
doesn't work. You need to (a) be able to specify the job dependencies 
independently of the work that job is doing, (b) be able to handle 
failures and restarts as above, (c) perform load balancing[1], and so on.



[1] For a proper grid manager this is "real" load balancing by deciding 
which machines a particular job should be run on. But even within a 
single OS process, we can know that certain tasks require more or less 
memory or other resources, and so can alter how aggressively to fire off 
jobs in order to keep from overloading the machine. It's this extreme 
heterogeneity of jobs which is a core part of being a "grid manager" 
IMO; when the jobs are relatively homogeneous (even if doing different 
things) then a lot of these complications go away.

-- 
Live well,
~wren