<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Feb 11, 2010, at 1:57 PM, Bardur Arantsson wrote:</div><blockquote type="cite"><div><font class="Apple-style-span" color="#000000"><br></font><blockquote type="cite"> 2. the remote client has terminated the connection as far as it is<br></blockquote><blockquote type="cite">concerned but not notified the server -- when you try to send data it will<br></blockquote><blockquote type="cite">reject it, and send/write/sendfile/etc will raise sigPIPE.<br></blockquote><blockquote type="cite">Looking at your debug output, we are seeing the sigPIPE / Broken Pipe error<br></blockquote><blockquote type="cite">most of the time. But then there is the case where we get stuck on the<br></blockquote><blockquote type="cite">threadWaitWrite.<br></blockquote><blockquote type="cite">threadWaitWrite is ultimately implemented by passing the file descriptor to<br></blockquote><blockquote type="cite">the list of write descriptors in a call to select(). It seems, however, that<br></blockquote><blockquote type="cite">select() is not waking up just because calling write() on a file descriptor<br></blockquote><blockquote type="cite">*would* cause sigPIPE.<br></blockquote><br>That's what I expect select() with an "errfd" FDSET would do.<br></div></blockquote><div><br></div><div>Nope. The expectfds are only trigger in esoteric conditions. For TCP sockets, I think it only occurs if there is out-of-band data available to be read via recv() with the MSG_OOB flag.</div><div><font class="Apple-style-span" face="Arial, 'Liberation Sans', 'DejaVu Sans', sans-serif" size="4"><span class="Apple-style-span" style="border-collapse: collapse; font-size: 14px; line-height: 18px;"><br></span></font></div><div><font class="Apple-style-span" face="Arial, 'Liberation Sans', 'DejaVu Sans', sans-serif" size="4"><span class="Apple-style-span" style="border-collapse: collapse; font-size: 14px; line-height: 18px;"><a href="http://uw714doc.sco.com/en/SDK_netapi/sockC.OoBdata.html">http://uw714doc.sco.com/en/SDK_netapi/sockC.OoBdata.html</a></span></font></div><br><blockquote type="cite"><div><blockquote type="cite">The easiest way to confirm this case is probably to write a small, pure C<br></blockquote><blockquote type="cite">program and see what really happens.<br></blockquote><blockquote type="cite">If this is the case, then it means the only way to tell if the client has<br></blockquote><blockquote type="cite">abruptly dropped the connection is to actually try sending the data and see<br></blockquote><blockquote type="cite">if the sending function calls sigPIPE. And that means doing some sort of<br></blockquote><blockquote type="cite">polling/timeout?<br></blockquote><br>Correct, but the trouble is deciding how often to poll and/or how long the timeout should be.<br><br>I don't see any easy answer to that. That's why my suggested "solution" is to simply punt it to the OS (by using portable mode) and suck up the extra overhead of the portable solution. Hopefully the new GHC I/O manager will make it possible to have a proper solution.<br></div></blockquote><div><br></div><div>The whole point of the sendfile library is to use sendfile(), so not using sendfile() seems like the wrong solution. I am also not convinced that the new GHC I/O manager will do anything new to make it possible to have a proper solution. I believe we would be seeing the same error even in pure C, so we need to know the work around that works in pure C as well. I am not convinced we are punting to the OS by using portable mode either (more below).</div><div><br></div><blockquote type="cite"><div><blockquote type="cite">I do not have a good explanation as to why the portable version does not<br></blockquote><blockquote type="cite">fail. Except maybe it is just so slow that it does not ever fill up the<br></blockquote><blockquote type="cite">buffer, and hence does not get stuck in threadWaitWrite?<br></blockquote><br>The portable version doesn't call threadWaitWrite. It simply turns the Socket into a handle (which causes it to become blocking) and so the kernel is tasked with handling all the gritty details.<br></div></blockquote><div><br></div><div>The portable version does not directly call threadWaitWrite, but it still calls it.</div><div><br></div><div>Data.ByteString.Char8.hPutStr calls</div><div>Data.ByteString.hPut which calls</div><div>Data.ByteString.hPutBuf which calls</div><div>System.IO.hPutBuf which calls</div><div>GHC.IO.Handle.Text.hPutBuf which calls</div><div>GHC.IO.Handle.bufWrite.Text which calls</div><div>GHC.IO.Device.write which calls</div><div>GHC.IO.FD.fdWrite which calls</div><div>GHC.IO.FD.writeRawBufferPtr which calls</div><div><br></div><div>which is defined as:</div><div><br></div><div><div>writeRawBufferPtr :: String -> FD -> Ptr Word8 -> Int -> CSize -> IO CInt</div><div>writeRawBufferPtr loc !fd buf off len</div><div> | isNonBlocking fd = unsafe_write -- unsafe is ok, it can't block</div><div> | otherwise = do r <- unsafe_fdReady (fdFD fd) 1 0 0</div><div> if r /= 0 </div><div> then write</div><div> else do threadWaitWrite (fromIntegral (fdFD fd)); write</div><div> where</div><div> do_write call = fromIntegral `fmap`</div><div> throwErrnoIfMinus1RetryMayBlock loc call</div><div> (threadWaitWrite (fromIntegral (fdFD fd)))</div><div> write = if threaded then safe_write else unsafe_write</div><div> unsafe_write = do_write (c_write (fdFD fd) (buf `plusPtr` off) len)</div><div> safe_write = do_write (c_safe_write (fdFD fd) (buf `plusPtr` off) len)</div><div><br></div><div>According to the following test program, I expect that 'isNonBlocking fd' will be 'True'. So it seems like the portable solution should be vulnerable to the same condition. Perhaps the portable version is just so slow that the OS buffers never fill up so EAGAIN is never raised?</div><div><br></div><div>-------------------------------------------------------------------------------------------------------</div><div><br></div><div><div>{-# LANGUAGE RecordWildCards #-}</div><div>module Main where</div><div><br></div><div>import Control.Concurrent (forkIO)</div><div>import Control.Monad (forever)</div><div>import Network (PortID(PortNumber), Socket, listenOn)</div><div>import Network.Socket (accept, socketToHandle)</div><div>import System.IO</div><div>import qualified GHC.IO.FD as FD</div><div>import GHC.IO.Handle.Internals (withHandle, flushWriteBuffer)</div><div>import GHC.IO.Handle.Types (Handle__(..), HandleType(..))</div><div>import qualified GHC.IO.FD as FD</div><div>import System.Posix.Types (Fd(..))</div><div>import System.IO.Error</div><div>import GHC.IO.Exception</div><div>import Data.Typeable (cast)</div><div>import GHC.IO.Handle.Internals (wantWritableHandle)</div><div><br></div><div>main =</div><div> listen (PortNumber (toEnum 2525)) $ \s -> </div><div> do h <- socketToHandle s ReadWriteMode</div><div> wantWritableHandle "main" h $ \h_ -> showBlocking h_</div><div> </div><div> </div><div>showBlocking :: Handle__ -> IO () </div><div>showBlocking h_@Handle__{..} =</div><div> case cast haDevice of</div><div> Nothing -> return ()</div><div> Just fd -> case (FD.fdIsNonBlocking fd) of</div><div> 1 -> putStrLn "is NonBlocking"</div><div> 0 -> putStrLn "is not NonBlocking"</div><div> </div><div>listen :: PortID -> (Socket -> IO ()) -> IO () </div><div>listen port handler =</div><div> do socket <- listenOn port</div><div> forever $ do (s,sa) <- accept socket</div><div> forkIO $ handler s</div><div><br></div><div>-------------------------------------------------------------------------------------------------------</div><div><br></div><div><br></div></div></div><br><blockquote type="cite"><div><blockquote type="cite">Any way, the fundamental question is:<br></blockquote><blockquote type="cite"> When your write buffer is full, and you call select() on that file<br></blockquote><blockquote type="cite">descriptor, will select() return in the case where calling write() again<br></blockquote><blockquote type="cite">would raise sigPIPE?<br></blockquote><br>I believe so, *if* you give it the FD in the exceptfds FD_SET parameter. Let's face it, any other behavior doesn't make any sense since it's the equivalent of forcing all timeout handling onto the user, just like threadWaitWrite currently does. I've written my fair share of networking code in various languages (including C/C++) and I've never seen this problem of "missing wakeups" before.<br></div></blockquote><div><br></div><div>All my reading seems to indicate that exceptfds won't help anything -- it is a seldom used feature and doesn't do what people wish it actually did. For example, see answer 1 on this page:</div><div><br></div><div><a href="http://stackoverflow.com/questions/1342712/nix-select-and-exceptfds-errorfds-semantics">http://stackoverflow.com/questions/1342712/nix-select-and-exceptfds-errorfds-semantics</a></div><div><br></div><div>There is some evidence that when you are doing select() on a readfds, and the connection is closed, select() will indicate that the fds is ready to be read, but when you read it, you get 0-bytes. That indicates that a disconnect has happened. However, if you are only doing read()/recv(), I expect that only happens in the event of a proper disconnect, because if you are just listening for packets, there is no way to tell the difference between the sender just not saying anything, and the sender dying:</div><div><br></div><div><a href="http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html#advanced">http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html#advanced</a></div><div><div><a href="http://channel9.msdn.com/forums/TechOff/434466-TcpClient-Test-for-Disconnected/">http://channel9.msdn.com/forums/TechOff/434466-TcpClient-Test-for-Disconnected/</a></div><div><a href="http://bytes.com/topic/python/answers/40278-detecting-shutdown-remot">http://bytes.com/topic/python/answers/40278-detecting-shutdown-remot</a></div></div><div><br></div><div>But I can not find any clear information on what happens when you do select() on a write socket, and the remote end abruptly disconnects. Consider first the case that does not use select() at all:</div><div><br></div><div>1. write() to a non-blocking socket. That copies the data into the OS buffers, and then returns successfully. </div><div>2. But then the OS tries to send the data, and the connection has been reset. It can't notify you that the write() failed, because it that call to write() already returned.</div><div>3. you try to do a second write(), that is when you get sigPIPE.</div><div><br></div><div>Now let's say you do:</div><div><br></div><div>write()</div><div>select()</div><div>write () </div><div><br></div><div>I believe that select() will not wakeup if it is just monitoring the ability to write to the socket and the remote end abruptly drops the connection. (I am certain that under linux, if another thread explicitly closes the socket, that does not cause select() to wake up either.) However, there is some evidence that if you monitor the socket for both reads and writes, that when the first write fails, select will wakeup and tell you that there is data available to read(). If you read() the data, you will find out that there are 0-bytes available, meaning the connection was closed. Alternatively, if you tried the second write, then you would get sigPIPE. </div><div><br></div><div><a href="http://www.developerweb.net/forum/showthread.php?t=2956">http://www.developerweb.net/forum/showthread.php?t=2956</a></div><div><a href="http://stackoverflow.com/questions/180095/how-to-handle-a-broken-pipe-sigpipe-in-python">http://stackoverflow.com/questions/180095/how-to-handle-a-broken-pipe-sigpipe-in-python</a></div><div><br></div><div>The tricky part is that if you wake up for a read() and there is data available, we don't want to actually read it in then sendfile function, (because that data is destined for somewhere else). So, you need to just peek at the data with out actually reading it to see if there is at least 1 byte available.</div><div><br></div><div>If this method of detection is correct, then what we need is a threadWaitReadWrite, that will notify us if the socket can be read or written. The IO manager does not currently provide a function like that.. but we could fake it like this: (untested):</div><div><br></div><div><div>import Control.Concurrent</div><div>import Control.Concurrent.MVar</div><div>import System.Posix.Types</div><div><br></div><div>data RW = Read | Write</div><div><br></div><div>threadWaitReadWrite :: Fd -> IO RW</div><div>threadWaitReadWrite fd =</div><div> do m <- newEmptyMVar</div><div> rid <- forkIO $ threadWaitRead fd >> putMVar m Read</div><div> wid <- forkIO $ threadWaitWrite fd >> putMVar m Write</div><div> r <- takeMVar m</div><div> killThread rid</div><div> killThread wid</div><div> return r</div><div><br></div><div>Of course, in the case where the client disconnects because someone turns off the power or pulls the ethernet cable, we have no way of knowing what is going on -- so there is still the possibility that dead connections will be left open for a long time.</div><div><br></div><div>And, there is also the concern that even the portable version may have this issue. My research indicates that it should. In fact, any application which tries to send data over the network could be vulnerable to this bug. So, I am a little disturbed as to why the portable version does not appear to have issues..</div><div><br></div><div>- jeremy</div></div></div></body></html>