On 30 January 2012 14:22, Rob Stewart <span dir="ltr"><<a href="mailto:robstewart57@gmail.com" target="_blank">robstewart57@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
I'm experiencing the "accept: resource exhausted (Too many open<br>
files)" exception when trying to use sockets in my Haskell program.<br>
<br>
The situation:<br>
- Around a dozen Linux machines running my Haskell program,<br>
transmitting thousands of messages to each other, sometimes within a<br>
small period of time.<br></blockquote><div><br></div><div>...</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
$ ulimit -n<br>
1024<br>
<br></blockquote><div><br></div><div>This is not an OS limit, this is your freely chosen limit. You should not run with this few file descriptors on a server. Increasing this by 50x is entirely reasonable. However, having too many open TCP connections is not a good thing either. 1024 was an upper limit way way back on the i386 linux architecture for code using the select() system call, that is why it is still a common default.</div>
<div><br></div><div>There are a few ways to get out of this situation.</div><div><br></div><div>1. Reuse your TCP connections. Maybe you could even use HTTP. An HTTP library might do reusing of connections for you.</div>
<div><br></div><div>2. Since you are blocking in getContents, there is a probability that it is the senders that are being lazy in sendAll. They opened the TCP connection, but now they are not sending everything in sendAll, so your receiver is having lots of threads that are blocked on reading. Try to be strict when *sending* so you do not have too many ongoing TCP connections. </div>
<div><br></div><div><div>3. On the receiver side, to be robust, you could limit the number of threads that are allowed to do an accept() to the number of file descriptors you have free. You can also block on a semaphore whenever accept returns out of resources, and signal that semaphore after every close.</div>
</div><div><br></div><div><br></div><div>Alexander</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Indeed, when I experience the "accept: resource exhausted (Too many<br>
open files)" exception, I check the number of open sockets, which<br>
exceeds 1024, by looking at the contents of the directory:<br>
ls -lah /proc/<prod_id>/fd<br>
<br>
It is within the getContents function that, once the lazy bytestring<br>
is fully received, the socket is shutdown <a href="http://goo.gl/B6XcV" target="_blank">http://goo.gl/B6XcV</a> :<br>
shutdown sock ShutdownReceive<br>
<br>
There seems to be no way of limiting the number of permitted<br>
connection requests from remote nodes. What I am perhaps looking for<br>
is a mailbox implementation on top of sockets, or another way to avoid<br>
this error. I am looking to scale up to 100's of nodes, where the<br>
possibility of more than 1024 simultaneous socket connections to one<br>
node is increased. Merely increasing the ulimit feels like a temporary<br>
measure. Part of the dilemma is that the `connect' call in `sendMsg'<br>
does not throw an error, despite the fact that it does indeed cause an<br>
error on the receiving node, by pushing the number of open connections<br>
to the same socket on the master node, beyond the 1024 limit permitted<br>
by the OS.<br>
<br>
Am I missing something? One would have thought such a problem occurs<br>
frequently with Haskell web servers and the like.. ?<br>
<span><font color="#888888"><br>
--<br>
Rob Stewart<br>
<br>
_______________________________________________<br>
Haskell-Cafe mailing list<br>
<a href="mailto:Haskell-Cafe@haskell.org" target="_blank">Haskell-Cafe@haskell.org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/haskell-cafe" target="_blank">http://www.haskell.org/mailman/listinfo/haskell-cafe</a><br>
</font></span></blockquote></div><br>