The 2005 GHC survey

It took me about half a day to write some code to generate HTML summaries of the results and comments, and over two days to read through and make sense of all the comments, but it's been an enlightening experience. We certainly couldn't have predicted some of these results.

There are three views of the results. First, here's the raw data, with percentage breakdowns for the checkbox-type questions, and pages listing all the comments (without attribution):

http://www.haskell.org/ghc/survey2005/

Second, here's a super-quick summary of what we've learned:

• Most of you seem to have a generally positive experience with GHC, which is encouraging!

• The #1 complaint was the compilation speed and memory usage of GHC.

• Next most unpopular was performance and size of the generated code.

• The most-requested feature, after performance improvements, is some kind of debugger.

• Portability and the difficulty of compiling GHC was raised a lot.

• API and feature changes breaking code between releases was also a common gripe.

• Many people mentioned the quality of error messages as something that could be improved.

• Many people want improvements to documentation, especially library docs.

• Lots of requests for an IDE and better GUI support.

• There is some concern over the complexity of GHC affecting its maintainability and portability, and how difficult it is to contribute.

Third, here's a more detailed summary, with excerpts of some of your comments. The huge amount of data has made it quite difficult to compile this, and I'm certain I've missed some important points, but I hope I've captured the general feel of the community here.

Overall Opinions

First of all, it was clear that lots of people completing the survey were trying out GHC for the first time. Several respondents were students downloading GHC to use for coursework. Several others mentioned Darcs or Pugs specifically as reasons for needing GHC.

In the overall opinions section, the range of comments was overwhelming. Many of them fell into other sections (such as suggestions for improvements), so I'll discuss them there. The following sections cover the topics that were most often commented on.

The majority of comments were positive about GHC. Here's a few examples (if GHC were a paperback, here's what we would put on the back cover):

• "Given the resources behind it, the level of support and quality is awesome!"
• "... our research would be virtually impossible without it."
• "A truly great piece of software."
• "All the people I know who are good Haskell programmers swear by it."
• "GHC is honking huge but very nice and impressive."
• "It's darn sexy."
• "It has got really stable and reliable over last few years/major versions"
• "I find it fantastic that such great software is completely free."
• "Outstanding ("world class") academic effort. I wish there were more funding to make it better."

• "Great! Big. Slow."
• "Big, fast, complex."
• "Powerful, slow, improving."
• "mature, maintained, fast"

Performance of GHC

• "... too slow for practical use... "
• "... it's very slow when compiling with optimizations... "

It is true that performance, both of the generated code and GHC itself, has not been at the top of our agenda recently, although we do try to ensure that things don't get worse with each release. As it happens though, we were planning to have a thorough investigation into performance issues soon, and performance of GHC itself is something we'll try to improve (stay tuned for an announcement of a forthocoming GHC performance-fest).

Performance of compiled code

About 2/3 of those comments wanted faster or smaller code.

• "... there are a lot of other people who would rather use Haskell over Ocaml, SML, Scheme, Lisp if only writing efficient code wasn't so hard. "
• "I was pleasantly surprised by the speed of the generated code (altough a bit large)."
• "... it performs extremely poorly when it comes to strings."

Bootstrapping and portability

An oft-mentioned concern was the difficulty of bootstrapping GHC on a new platform:

• "It's sad that it's not easier to bootstrap on new platforms..."
• "Documentation of bootstrapping the compiler is lacking and the process is difficult for someone new to the language."
• "bootstrapping from no haskell compiler was very difficult. I am an experienced linux system administrator."
• "Would be nice if cross compiling was easier..."
• "I would love to see GHC require only a c or c++ compiler to build."

The conclusion of that disccusion on the mailing list was that it would be possible to improve the bootstrap experience, at least for Unix users:

http://www.haskell.org//pipermail/glasgow-haskell-users/2005-May/008456.html

Related to this is the question of GHC's portability, which several people raised as a concern:

• "could be easier to port to other architectures -- at least for non-optimized ghc"
• "The lack of portability is an issue for adoption at work."
• "My #1 complaint is portability. From my own experience trying (and finally succeeding) to make it work on AIX, and failing to make it work on ARM, it is one of the most difficult-to-port Unix apps I've seen in a long time."
• "I do wish the dynamic linker was more portable..."

• "I've always found building GHC from source to be an exercise in self-flagellation..."
• "... A more sane build system, so I can build it without getting a hernia :)"

API stability, and backwards-compatibility

The question of stability was raised quite often.

• "I also feel that GHC is a bit of a moving target, new releases often change a lot of language features and libraries. More stability would be comforting."
• "it's a bit hard to keep compatibility with software made for earlier version as language extensions tends to change a bit every time."
• "I am sometimes annoyed at how difficult it is to compile programs that worked in previous versions of GHC with newer versions."
• "... API breakage between versions is quite frustrating!"

Backwards-compatibility problems arise because we need to make changes to experimental features or libraries, and want to do so without accumulating too much cruft. However, in the future we will strive to clearly indicate which features and libraries are considered experimental.

Many people thought that GHC's rapid acquisition of new features was a positive thing:

• "I am specifically interested (and impressed) by the constant adding of new features for extending Haskell"
• "Reading the feature list of a next GHC release feels like unpacking a Christmas present :)"

Error Reporting

• "My only complaint involves the quality of the compilation errors."
• "... warnings and error message are sometimes hard to understand."
• "Improve error messages - more concrete locations. It's usually unclear where the conflicting type comes from, ..."
• "... I wish type classes gave more helpful error messages."
• "My only complaint is the quality of the error messages, even though they have been greatly improved since earlier versions."

We urge everyone to report concrete examples of unhelpful error messages (and the code that produced them) - this is the best way we can improve the situation.

Documentation

Lack of documentation is a frequent source of frustration. However, it is clear that some respondents wanted more general Haskell documentation: well-written introductory material and lots of code samples were requested.

The other areas of documentation that people found lacking were:

• Library documentation. Haddock isn't enough, actual docs are required. Also, some kind of "gentle introduction" to the libraries would be nice.

• Template Haskell documentation was mentioned several times

Support and community

Plenty of people were enthusiastic about the support they had received from the GHC developers and the community in general.

• "Support is good: there are always people willing to help you on the mailing lists."
• "Very responsive/helpful implementers."
• "Support from the mailing lists and the maintainers is always excellent."
• "... a very responsive user community ..."

Accessibility of the code

Concerns were raised over the accessibility of the code, mainly its size and complexity, and the fact that this makes it difficult to contribute. We are sensitive to this - GHC relies heavily on external contributors, and we always need more.

• "... accessibility of its internals is not so easy."
• "The type system it implements is so complex that only a few people in the world can understand it."
• "what I don't like about GHC is that it's practically impossible to get a good grip on the application itself."

The conclusion here is that we should try to make the project more accessible to contributors. We'll discuss how exactly to do this separately, and ideas are welcome.

Platforms

Raw comments: http://www.haskell.org/ghc/survey2005/platform_comments.html

Generally, folk are happy that they can use GHC on multiple platforms. There were several requests for a GHC port to a platform we don't currently support, with the following being the most requested:

• x86-64
• .NET
• Cygwin

As for .NET, there have been several attempts, and there is at least one on-going project to generate .NET code. Simon PJ posted a long message on haskell-cafe recently explaining the issues behind compiling to .NET and why we haven't done it yet:

http://www.haskell.org//pipermail/haskell-cafe/2005-January/008244.html

The Cygwin port is latent, waiting for someone to pick it up. Volunteers are welcome!

There was the occasional sentiment expressed that the Windows platform isn't given the first-class status it deserves by the developers. The survey stats show that nearly half our users use GHC on Windows, so it's clearly important. We've started to address this with nightly builds and snapshots on Windows.

Features

Raw comments: http://www.haskell.org/ghc/survey2005/features_comments.html

The options on the survey could have been better here. Lots of people clearly left the boxes as "didn't answer" when they meant "never use". There should have been an option for "don't know what it is". The options "never use" and "nice to have" are not mutually exclusive.

Several people commented that they weren't aware that GHC supported all these features, and they learned something by filling in the survey :-)

We can produce a rough indication of the popularity of features, by giving "nice to have" 1 point and "essential to me" 2 points. The top 5 are:

1. ghci (interactive version)
2. ghc -O (code optimisation capabilities)
3. Foreign language interface
4. ghc --make (built-in make)
5. Packages

The bottom 5:

34. Rewrite rules
35. Generating or compiling External Core
36. Rebindable syntax
37. Implicit parameters
38. Linear implicit parameters

Suggestions for Improvements

Raw comments: http://www.haskell.org/ghc/survey2005/things_to_improve_comments.html

Here I'll list the features that were requested, in rough order of popularity. Some have already been mentioned: better performance, better error messages, easier bootstrapping, making GHC easier to build and hack on, better documentation, better backwards compatibility.

• About 25 responses commented on the lack of good debugging support. The kind of debugging support that people wanted varied, from "tracing"-style to "gdb"-style. We don't have any concrete plans in this direction at the moment, but it's clearly an important issue.

• Shared libraries were requested 17 times. The reasons were twofold: to be able to create plugins for existing software (Apache and Tcl/Tk, for example), and to reduce binary sizes. GHC does have fledgeling support for shared libraries: the MacOS X port uses shared libraries, and we have had support for Windows DLLs in the past, but that withered due to lack of time. We hope to support shared libraries on more platforms in the future.

• Several people asked for an IDE (13), and (often in the same sentence) better or easier access to GUI libraries.

• Parallel Haskell was mentioned 8 times.
  • "When do we get real thread-level parallelism and a parallel GC?"
  • "... would like to see work on parallel computation..."
  • "parallel Haskell is a big wish and up to now a disappontment."
  • "True Paralellism. I belive this to be the killer-app for Haskell."

  For those that don't know, we're currently working on native support for shared-memory parallelism in GHC, and we plan for this to be a part of the next major release.

• A better record system was mentioned 8 times, although the suggestions ranged from simple support for re-using field labels to a fully-fledged integrated record/module system.

• The GHC API was mentioned 6 times. This is an API that allows an external tool programmatic access to GHC's facilities for compiling and executing code, and access to GHC's abstract syntax. This is implemented and working, and will be part of GHC 6.6.

• Optimistic evaluation was mentioned a few times. We currently have no plans to inocrporate the optimistic evaluation branch, due to its significant complexity, and the fact that performance improvements tended to be inconsistent.

• Others:
  • Better recursive modules (4).
  • Views (4).
  • Compile to the JVM (3).
  • Ability to call GMP via the FFI, or link with other libraries that use GMP (3).
  • Searching for functions (2).
  • ghc --make should not link if nothing changed (3).
  • Unicode support for Haskell source (2), for I/O (3).
  • Better module system (2).
  • "An extensible exception type (better: extensible types in general). Using Dynamic is clumsy..."
  • "I'd like to be able to unwrap existential types like I can in Hugs, without getting the "my brain just exploded" error message."
  • "Global bindings of some sort..."
  • "... a more versatile and comprehensive pre-processing system like OCaml's CamlP4."
  • "Make ext-core work. Allow .hcr inputs to be re-optimized."

Development model

Raw comments: http://www.haskell.org/ghc/survey2005/devmodel_comments.html

Several people didn't understand this question. Clearly we should have been more explicit rather than using the open-source term "development model". For this reason, I don't think we can draw much from the numerical breakdown for this question.

However, the comments form those that did understand the question were generally positive.

• "Given how few people work at GHC and how complex it is, it is close to a miracle how fully-featured and reliable it is. "
• "... it is surprising how much is being achieved with so limited resources - Great Work!"
• "We need to do a better job of marketing/communicating the value of GHC to non-academic communities."

• "A bit worried about the number of people who knows ghc inside out.."
• "I have a feeling that there aren't enough people working on ghc."
• "I wish you had more manpower."
• "... I worry that not enough people are deeply involved in hacking GHC every day."

• "But it does seem to rely heavily on the continued participation of various people named Simon..."
• "... I'm worried about busses in Cambridge, and the effect they may have on Simon and Simon.... :)"
• "Not great, because we still have too much reliance on The Simons."
• "One of the problem is the high bus factor on Simon Marlow and Simon Peyton-Jones, though."

• "I think it would be easier for me to contribute if VC were in darcs instead of CVS."
• "Could use a better, less centralized, version control system like darcs."

The difficulty of getting to the stage where one can contribute effectively was raised several times (see "accessibility of the code").

Releases

Raw comments: http://www.haskell.org/ghc/survey2005/release_freq_comments.html
http://www.haskell.org/ghc/survey2005/stability_comments.html

We asked whether releases are made often enough, and of those that had an opinion, 91% said that releases were frequent enough.

A few people thought releases were too frequent; in many cases this tied in with the problems raised about backwards compatibility. This despite the fact that there was more than a year between the last two major releases, and no code-breaking changes were made in the interrim.

It's not possible to have both perfect backwards compatbility and also introduce new features and evolve the libraries. Keeping backwards compatibility also imposes a maintenance overhead and bloats the codebase. So whatever situation we adopt will be a compromise; what we can take from the survey is that the community would certainly not like major releases to happen *more* often, especially if they break code and/or build systems. If anything, we could lean towards producing more stable releases of each line. The number of patchlevel releases is generally driven by two factors: the number of bug reports against the most recent release, traded off against the difficulty of merging fixes from the trunk. The longer a stable branch remains active, the harder it gets to fix bugs in both the stable branch and the head, which eventually necessitates making a new major release.

People are happy with the stability of releases (96% of those who had an opinion thought that releases were stable enough), with several people commenting that they thought stability had improved over the last few releases.

Thanks!

The GHC Team would like to express its sincere thanks to everyone who responded to the survey. It's provided us with a wealth of information on how people use GHC, what aspects of GHC they enjoy and find most important, and how they would like to see GHC evolve.

We'll certainly be taking your opinions into account as we decide where to go next, but as always we like to think of GHC as belonging to the community, so to a some extent the direction of GHC is decided by who gets involved and how they contribute. We're always keen to help people get started with hacking GHC, so if you're interested please join cvs-ghc@haskell.org, find something to work on, post your questions, and let us know how we can make things more accessible.

Extra thanks to Tomasz Zielonka for implementing the survey and collecting the responses.

Cheers! -- The GHC Team.