Personal tools

Iteratee I/O

From HaskellWiki

(Difference between revisions)
Jump to: navigation, search
(Updated a link)
(Described iteratees)
Line 1: Line 1:
{{Stub}}
+
Iteratee I/O is a way to avoid the problems that can occur with lazy I/O. They work by making the I/O actions explicit, making their behavior easier to reason about.
   
Iteratee I/O is a way to avoid the problems that can occur with lazy I/O.
+
== The problem with lazy I/O ==
   
Discussions elsewhere on this wiki include:
+
As a beginner, you probably used Haskell's lazy I/O through the <code>System.IO</code> module. However, while it is good enough for simple programs, its unpredictability makes it unsuitable for practical use.
   
* [[Enumerator and iteratee]]
+
For example, a common beginner mistake is to close a file before one has finished reading it:
* [[Iteratee]]
 
   
Essays by Oleg:
+
<haskell>
  +
wrong = do
  +
fileData <- withFile "test.txt" ReadMode hGetContents
  +
putStr fileData
  +
</haskell>
  +
  +
The problem is <code>withFile</code> closes the handle before <code>fileData</code> is forced. The correct way is to pass all the code to <code>withFile</code>:
  +
  +
<haskell>
  +
right = withFile "test.txt" ReadMode $ \handle -> do
  +
fileData <- hGetContents handle
  +
putStr fileData
  +
</haskell>
  +
  +
Here, the data is consumed before <code>withFile</code> finishes.
  +
  +
Although this is easily fixed, the type system does not enforce the correct solution. Even worse, if you use the former code, it won't even raise an error &ndash; it will just fail silently and return an empty string. Many years passed before a satisfactory solution to the ''streaming data problem'' was found.
  +
  +
== How iteratees work ==
  +
  +
When you "step" an iteratee, it reads a chunk of data, updates the internal state and returns a new iteratee along with the data it read. Because an iteratee is simply a function with state, many iteratees can be composed together to form a pipeline.
  +
  +
Some implementations also provide a resource management layer that releases resources automatically when they are no longer needed. This is very useful in a server, where sockets and file handles are scarce.
  +
  +
== Implementations ==
  +
  +
; [http://hackage.haskell.org/package/iteratee iteratee] : The original iteratee library, by Oleg Kiselyov.
  +
; [http://hackage.haskell.org/package/iterIO iterIO] : Yet another implementation.
  +
; [http://hackage.haskell.org/package/enumerator enumerator] : Used in Snap. It does not use any extensions, so it will work with most Haskell compilers.
  +
; [http://hackage.haskell.org/package/pipes pipes] : A more recent implementation, which strives to be more elegant than existing libraries.
  +
; [http://hackage.haskell.org/package/pipes-core pipes-core] : Fork of pipes which adds resource finalization.
  +
; [http://hackage.haskell.org/package/conduit conduit] : Popular implementation designed with practical use in mind, created by the author of Yesod. Recently heavily influenced by pipes.
  +
; [http://hackage.haskell.org/package/liboleg liboleg] : An evolving collection of Oleg Kiselyov's Haskell modules (depends on the package unix and will therefore not compile on Windows systems).
  +
  +
== Essays by Oleg ==
 
 
 
* Oleg's writings: [http://okmij.org/ftp/Streams.html#iteratee Incremental multi-level input processing with left-fold enumerator: predictable, high-performance, safe, and elegant]
 
* Oleg's writings: [http://okmij.org/ftp/Streams.html#iteratee Incremental multi-level input processing with left-fold enumerator: predictable, high-performance, safe, and elegant]
Line 13: Line 13:
 
* [http://okmij.org/ftp/Haskell/Iteratee/IterateeM.hs A general library by Oleg]
 
* [http://okmij.org/ftp/Haskell/Iteratee/IterateeM.hs A general library by Oleg]
   
Other discussions:
+
== Other discussions ==
   
 
* [http://themonadreader.wordpress.com/2010/05/12/issue-16/ The Monad.Reader Issue 16]; see the section "Iteratee: Teaching an Old Fold New Tricks" by John W. Lato
 
* [http://themonadreader.wordpress.com/2010/05/12/issue-16/ The Monad.Reader Issue 16]; see the section "Iteratee: Teaching an Old Fold New Tricks" by John W. Lato
Line 21: Line 21:
 
* [http://www.scs.stanford.edu/11au-cs240h/notes/iteratee.html Stanford CS240h lecture on iteratee]
 
* [http://www.scs.stanford.edu/11au-cs240h/notes/iteratee.html Stanford CS240h lecture on iteratee]
   
Uses of Iteratee I/O:
+
== Users of Iteratee I/O ==
* [http://snapframework.com Snap] : The Snap web framework
+
  +
* [http://snapframework.com Snap]: The Snap web framework
 
* [http://hackage.haskell.org/package/yaml yaml]: Low-level binding to the libyaml C library]
 
* [http://hackage.haskell.org/package/yaml yaml]: Low-level binding to the libyaml C library]
 
* [http://hackage.haskell.org/package/usb-0.4 usb 0.4]: Communicate with USB devices
 
* [http://hackage.haskell.org/package/usb-0.4 usb 0.4]: Communicate with USB devices
Line 28: Line 28:
 
* [http://hackage.haskell.org/package/wai WAI]: a Web Application Interface for haskell web frameworks (used by [http://www.yesodweb.com Yesod]).
 
* [http://hackage.haskell.org/package/wai WAI]: a Web Application Interface for haskell web frameworks (used by [http://www.yesodweb.com Yesod]).
   
Packages:
+
== See also ==
* [http://hackage.haskell.org/package/iterIO iterIO] (note, that this will not compile on Windows, as it depends on the package unix)
+
* [http://hackage.haskell.org/package/enumerator enumerator] [http://www.haskell.org/pipermail/haskell-cafe/2010-August/082324.html ANNOUNCE: enumerator, an alternative iteratee package] (this package does not use any extensions, so it will work with most Haskell compilers)
+
* [[Enumerator and iteratee]]
* [http://hackage.haskell.org/package/attoparsec-iteratee attoparsec-iteratee]: An adapter to convert attoparsec Parsers into blazing-fast Iteratees
+
* [[Iteratee]]
* [http://hackage.haskell.org/package/iteratee iteratee]: Iteratee-based I/O
 
* [http://hackage.haskell.org/package/iteratee-parsec iteratee-parsec]: Package allowing parsec parser initeratee
 
* [http://hackage.haskell.org/package/liboleg liboleg]: An evolving collection of Oleg Kiselyov's Haskell modules (depends on the package unix and will therefore not compile on Windows systems)
 
* See also the [http://hackage.haskell.org/packages/archive/pkg-list.html#cat:enumerator enumerator category on Hackage]
 
   
 
[[Category:Idioms]]
 
[[Category:Idioms]]

Revision as of 02:20, 12 July 2012

Iteratee I/O is a way to avoid the problems that can occur with lazy I/O. They work by making the I/O actions explicit, making their behavior easier to reason about.

Contents

1 The problem with lazy I/O

As a beginner, you probably used Haskell's lazy I/O through the System.IO module. However, while it is good enough for simple programs, its unpredictability makes it unsuitable for practical use.

For example, a common beginner mistake is to close a file before one has finished reading it:

wrong = do
    fileData <- withFile "test.txt" ReadMode hGetContents
    putStr fileData

The problem is withFile closes the handle before fileData is forced. The correct way is to pass all the code to withFile:

right = withFile "test.txt" ReadMode $ \handle -> do
    fileData <- hGetContents handle
    putStr fileData

Here, the data is consumed before withFile finishes.

Although this is easily fixed, the type system does not enforce the correct solution. Even worse, if you use the former code, it won't even raise an error – it will just fail silently and return an empty string. Many years passed before a satisfactory solution to the streaming data problem was found.

2 How iteratees work

When you "step" an iteratee, it reads a chunk of data, updates the internal state and returns a new iteratee along with the data it read. Because an iteratee is simply a function with state, many iteratees can be composed together to form a pipeline.

Some implementations also provide a resource management layer that releases resources automatically when they are no longer needed. This is very useful in a server, where sockets and file handles are scarce.

3 Implementations

iteratee 
The original iteratee library, by Oleg Kiselyov.
iterIO 
Yet another implementation.
enumerator 
Used in Snap. It does not use any extensions, so it will work with most Haskell compilers.
pipes 
A more recent implementation, which strives to be more elegant than existing libraries.
pipes-core 
Fork of pipes which adds resource finalization.
conduit 
Popular implementation designed with practical use in mind, created by the author of Yesod. Recently heavily influenced by pipes.
liboleg 
An evolving collection of Oleg Kiselyov's Haskell modules (depends on the package unix and will therefore not compile on Windows systems).

4 Essays by Oleg

5 Other discussions

6 Users of Iteratee I/O

  • Snap: The Snap web framework
  • yaml: Low-level binding to the libyaml C library]
  • usb 0.4: Communicate with USB devices
  • sstable: SSTables in Haskell
  • WAI: a Web Application Interface for haskell web frameworks (used by Yesod).

7 See also