<br><font size=2 face="sans-serif">Hello,</font>
<br>
<br><font size=2 face="sans-serif"> Just writing to let people know
the resolution of this problem...</font>
<br>
<br><font size=2 face="sans-serif"> After much frustration and toil,
we realized there was a bug in GHC's handle abstraction over sockets. </font>
<br>
<br><font size=2 face="sans-serif"> We resolved our immediate problem
by having our code deal directly with the sockets, and we filed a bug report,
#2703, which has just been (partially fixed) by Simon Marlow. </font>
<br>
<br><font size=2 face="sans-serif">thanks,</font>
<br><font size=2 face="sans-serif"> Jeff</font>
<br>
<br><tt><font size=2>Simon Marlow <simonmarhaskell@gmail.com> wrote
on 10/10/2008 09:23:31 AM:<br>
<br>
> Jeff Polakow wrote:<br>
> <br>
> > Don Stewart <dons@galois.com> wrote on 10/09/2008 02:56:02
PM:<br>
> > <br>
> > > jeff.polakow:<br>
> > > > We have a server that accepts messages
over a socket, spawning <br>
> > threads to<br>
> > > > process them. Processing these messages
may cause other, outgoing<br>
> > > > connections, to be spawned. Under
sufficient load, the main <br>
> > server loop<br>
> > > > (i.e. the call to accept, followed
by a forkIO), becomes <br>
> > nonresponsive.<br>
> > > ><br>
> > > > A smaller distilled testcase reveals
that when sufficient socket <br>
> > activity<br>
> > > > is occurring, an incoming connection
may not be responded to <br>
> > until other<br>
> > > > connections have been cleared out
of the way, despite the fact <br>
> > that these<br>
> > > > other connections are being handled
by separate threads. One <br>
> > issue that<br>
> > > > we've been trying to figure out
is where this behavior arises <br>
> > from-- the<br>
> > > > GHC rts, the Network library, the
underlying C libraries.<br>
> > > ><br>
> > > > Have other GHC users doing applications
with large amounts of<br>
> > > socket usage<br>
> > > > observed similar behavior and managed
to trace back where it <br>
> > originates<br>
> > > > from? Are there any particular architectural
solutions that <br>
> > people have<br>
> > > > found to work well for these situations?<br>
> > ><br>
> > > Hey Jeff,<br>
> > ><br>
> > > Can you say which GHC you used, and whether you used
the threaded<br>
> > > runtime or non-threaded runtime?<br>
> > ><br>
> > Oops, forgot about that...<br>
> > <br>
> > We used both ghc-6.8.3 and ghc-6.10.rc1 and we used the threaded
<br>
> > runtime. We are running on a 64 bit linux machine using openSUSE
10.<br>
> <br>
> The scheduler doesn't have a concept of priorities, so the accepting
thread <br>
> will get the same share of the CPU as the other threads. Another
issue is <br>
> that the accepting thread has to be woken up by the IO manager thread
when <br>
> a new connection is available, so we might have to wait for the IO
manager <br>
> thread to run too. But I wouldn't expect to see overly long
delays. Maybe <br>
> you could try network-alt which does its own IO multiplexing.<br>
> <br>
> If you have multiple cores, you might want to try fixing the thread
<br>
> affinity - e.g. put all the worker threads on one core, and the accepting
<br>
> thread on the other core. You can do this using GHC.Conc.forkOnIO,
with <br>
> the +RTS -qm -qw options.<br>
> <br>
> Other than that, I'm not sure what to try right now. We're hoping
to get <br>
> some better profiling for parallel/concurrent programs in the future,
but <br>
> it's not ready yet.<br>
> <br>
> Cheers,<br>
> Simon<br>
</font></tt>
<br>
<span style="font-family:'Arial',sans-serif; font-size:8pt; color:#000000">---<br>
<br>
This e-mail may contain confidential and/or privileged information. If you <br>
are not the intended recipient (or have received this e-mail in error) <br>
please notify the sender immediately and destroy this e-mail. Any <br>
unauthorized copying, disclosure or distribution of the material in this <br>
e-mail is strictly forbidden.</span>