[Haskell-cafe] Re: Parallel combinator, performance advice

[ChrisK]

Neil Mitchell wrote:
> Sorry, accidentally sent before finishing the response! I also noticed
> you sent this directly to me, not to -cafe, was that intentional?

The mail/news gateway makes it look like that, but I also sent to the mailing list.

> You mean something like:
>
> parallel_ xs =
>    sem <- createSemapore (length xs)
>    enqueue [x >> signalSemapore sem | x <- xs]
>    waitOnSemaphore sem
>
> I thought of something like this, but then the thread that called
> parallel_ is blocked, which means if you fire off N threads you only
> get N-1 executing. If you have nested calls to parallel, then you end
> up with thread exhaustion. Is there a way to avoid that problem?
>
> Thanks
>
> Neil

Your parallel_ does not return until all operations are finished.

> parallel_ (x:xs) = do
>     ys <- mapM idempotent xs
>     mapM_ addParallel ys
>     sequence_ $ x : reverse ys

By the way, there is no obvious reason to insert "reverse" there.

What I meant was something like:
>
> para [] = return ()
> para [x] = x
> para xs = do
>   q <- newQSemN 0
>   let wrap x = finally x (signalQSemN q 1)
>       go [y] n = wrap x >> waitQSemN q (succ n)
>       go (y:ys) n = addParallel (wrap y) >> go ys $! succ n
>   go xs 0

This is nearly identical to your code, and avoid creating the MVar for each
operation.  I use "finally" to ensure the count is correct, but if a worker
threads dies then bas things will happen.  You can replace finally with (>>) if
speed is important.

This is also lazy since the length of the list is not forced early.