[Haskell-cafe] two problems with Data.Binary and Data.ByteString

[Krasimir Angelov]

I had the same problem (stack overflow). The solution was to change
the >>= operator in the Get monad. Currently it is:

  m >>= k   = Get (\s -> let (a, s') = unGet m s
                           in unGet (k a) s')

but I changed it to:

    m >>= k   = Get (\s -> case unGet m s of
                             (a, s') -> unGet (k a) s')

It seems that the bind operator is lazy and this caused the stack overflow.

I have also another problem. Every Int and Word is stored as 64-bit
value and this expands the output file a lot. I have a lot of integers
and most of them are < 128 but not all of them. I changed the
serialization so that the Int and Word are serialized in a variable
number of bytes. Without this change the binary serialization was even
worse than the textual serialization that we had before. The file was
almost full with zeros.

I just haven't time to prepare a patch and to send it for review but
if other people have the same problem I will do it.


Best Regars,
   Krasimir




On Wed, Aug 13, 2008 at 1:13 AM, Tim Newsham <newsham at lava.net> wrote:
> I have a program that read in and populated a large data structure and
> then saved it out with Data.Binary and Data.ByteString.Lazy.Char8:
>
>   saveState db = B.writeFile stateFile =<<
>       encode <$> atomically (readTVar db)
>
> when I go to read this in later I get a stack overflow:
>
> loadState db = do
>    d <- decode <$> B.readFile stateFile
>    atomically $ writeTVar db d
>
>  Stack space overflow: current size 8388608 bytes.
>  Use `+RTS -Ksize' to increase it.
>
> or from ghci:
>
>    d <- liftM decode
>      (Data.ByteString.Lazy.Char8.readFile
>         "savedState.bin") :: IO InstrsDb
>
>    fromList *** Exception: stack overflow
>
> The data type I'm storing is a Map (of maps):
>
>   type DailyDb = M.Map Date Daily
>   type InstrsDb = M.Map String DailyDb
>
> What's going on here?  Why is the system capable of building and saving
> the data but not in reading and umarhsalling it?  What is the proper way
> to track down where the exception is happening?  Any debugging tips?
>
> I also noticed another issue while testing.  If my program loads
> the data at startup by calling loadState then all later calls to
> saveState give an error:
>
>  Log: savedState.bin: openFile: resource busy (file is locked)
>
> this does not occur if the program wasnt loaded.  My best guess here
> is that B.readFile isnt completing and closing the file for some
> reason.  Is there a good way to force this?
>
> Tim Newsham
> http://www.thenewsh.com/~newsham/
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>