Haskell ghci ocupying too much memory

String has much more overhead than most people realize when they first start using it. For each byte in a simple latin-1 input file:

• There's 4 bytes (maybe 8, I'm not sure) for the Char# storing its full unicode codepoint.
• There's an 8 byte pointer from the Char constructor to the Char#.
• There's 16 bytes of pointer from the : constructor to the Char and the rest of the String.

All told that's a 28x blowup for simple text. 28*21MB ~= 600MB. If you're ever storing both the file's contents and the parsed contents, double that again, to get 1.2GB -- pretty close to all of the 1.5GB you actually observe.

This is (part of) what makes ByteString and Text such critical parts of production Haskell programs. They have much less overhead, because they use arrays instead of linked lists and, in the case of Text, UTF8 or something like it instead of UCS32. Typically there will be only a couple 8-byte words of overhead per file (as opposed to the accounting above which was per byte) compared to the file's actual contents. You'll want to use one of those -- or an existing CSV library based on one of those.

Since you’re seeing the same issue with Text, this looks like excessive laziness, also called a “space leak” (although it’s more like a clog).

splitNewline and splitComma have similar issues, so I’ll go over splitComma as an example.

For each character of the input, splitComma allocates a cons cell ((:)). In the third clause, this is (h :< head solved) : tail solved. The head of this cell is a thunk, h :< head solved, whose closure holds references to both h and solved. So solved will be retained until both the head and tail of the list are evaluated.

That is, using head and tail is equivalent to a lazy pattern match on solved:

splitComma isInside (h :< t) = (h :< h') : t' 
  where ~(h' : t') = splitComma (if isInside then h /= '\"' else h == '\"') t

It would be better to make the pattern match strict, so that evaluating the result forces solved to be evaluated first. This can be done using case:

splitComma isInside (h :< t) = 
  case solved of
    h' : t' -> (h :< h') : t'
    _ -> error "splitComma: empty list"
  where solved = splitComma (if isInside then h /= '\"' else h == '\"') t

Or a bang pattern (!⟨pattern⟩):

splitComma isInside (h :< t) = (h :< h') : t'
  where !(h' : t') = splitComma (if isInside then h /= '\"' else h == '\"') t

However, it’s inefficient in the first place to rebuild the input character-by-character like this. When the head of that cell is evaluated, pattern (: