[GHC] #2148: x86_64 code use several GiB of memory generates: internal error: ASSERTION FAILED: file sm/Storage.c, line 1126

[GHC]

#2148: x86_64 code use several GiB of memory generates: internal error: ASSERTION
FAILED: file sm/Storage.c, line 1126
-------------------------------+--------------------------------------------
    Reporter:  twhitehead      |       Owner:          
        Type:  bug             |      Status:  new     
    Priority:  normal          |   Component:  Compiler
     Version:  6.8.2           |    Severity:  normal  
    Keywords:                  |    Testcase:          
Architecture:  x86_64 (amd64)  |          Os:  Linux   
-------------------------------+--------------------------------------------
 I have some fairly simple code using !ByteStrings to parses fasta files
 (http://en.wikipedia.org/wiki/Fasta_format) and generate all the M-length
 suffixes.

 When I set it to hold enough M-length suffixes is memory that several GiB
 of RAM are consumed, it either segment faults or loops infinitely (the
 pared down code I've attached infinitely loops with the given parameters,
 while my original code segment faulted).  I hooked into the attached
 version while it was infinitely looping one with gdb and did a backtrace.

 {{{
 #0  0x0000000000460ef4 in free_list_push_forwards ()
 #1  0x0000000000461357 in freeGroup ()
 #2  0x000000000046222f in GarbageCollect ()
 #3  0x000000000045d33f in scheduleDoGC ()
 #4  0x000000000045dd36 in scheduleWaitThread ()
 #5  0x000000000045aa01 in real_main ()
 #6  0x000000000045aad3 in main ()
 }}}

 Disassembling the free_list_push_forwards routine revealed that it was
 infinitely stuck in the while loop because the tail of the passed block
 descriptor (list) was circular and the passed block descriptor had a
 larger block count than any of the block descriptors in its circular tail.
 I also noted that the block descriptors composing the tail where not non-
 decreasing by block count up to the circular point (I don't know if this
 is a problem though).

 When I compile with -debug, I get the following (first two lines is
 program output):

 {{{
 Starting new sorted block at 139609412 (/tmp/PChunk3790.tmp)
 Starting new sorted block at 255642977 (/tmp/PChunk3791.tmp)
 Bug-debug: internal error: ASSERTION FAILED: file sm/Storage.c, line 1126

     (GHC version 6.8.2 for x86_64_unknown_linux)
     Please report this as a GHC bug:
 http://www.haskell.org/ghc/reportabug
 }}}

 I've attached the Bug.hs code (which is the pared down version of my
 code).  A gzipped version of the input file (~3GiB) can be found at
 ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/nr.gz.  To generate the above
 output, I compiled with "ghc -debug -O2 --make Bug" and with "./Bug-debug
 nr 8 67108864" (i.e., generate blocks of 67108864 8-character suffixes
 from the fasta file nr).

 From what I've seen so far, the problem seem linked to the complexity of
 the code and the amount of memory used.  I've never had any problems as
 long as I stay under 4GiB of RAM, and the more I pared down the code, the
 more memory I had to make it use before it would start to demonstrate the
 problem (with the parameters above the pared down code uses just over
 16GiB).  I could possibly pare the code down further.

 I do not know at all what I'm talking about, but from the little I've seen
 so far, I wonder if it is related to the 64bit memory space (e.g., having
 enough blocks that it can actually loops a 32bit counter tracking them,
 etc.).

 NOTE:  I compiled ghc from source from
 http://www.haskell.org/ghc/dist/stable/dist/ghc-6.8.2-src.tar.bz2.

-- 
Ticket URL: <http://hackage.haskell.org/trac/ghc/ticket/2148>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler