The 2005 GHC surveyThanks to everyone who responded to the survey! We've had just over 600 responses (as of a week or so ago), which is more than we'd hoped for.
It took me about half a day to write some code to generate HTML summaries of the results and comments, and over two days to read through and make sense of all the comments, but it's been an enlightening experience. We certainly couldn't have predicted some of these results.
There are three views of the results. First, here's the raw data, with percentage breakdowns for the checkbox-type questions, and pages listing all the comments (without attribution):
Second, here's a super-quick summary of what we've learned:
Third, here's a more detailed summary, with excerpts of some of your comments. The huge amount of data has made it quite difficult to compile this, and I'm certain I've missed some important points, but I hope I've captured the general feel of the community here.
Overall OpinionsRaw comments: http://www.haskell.org/ghc/survey2005/overall_comments
First of all, it was clear that lots of people completing the survey were trying out GHC for the first time. Several respondents were students downloading GHC to use for coursework. Several others mentioned Darcs or Pugs specifically as reasons for needing GHC.
In the overall opinions section, the range of comments was overwhelming. Many of them fell into other sections (such as suggestions for improvements), so I'll discuss them there. The following sections cover the topics that were most often commented on.
The majority of comments were positive about GHC. Here's a few examples (if GHC were a paperback, here's what we would put on the back cover):
Performance of GHCOf the people that commented on the performance of GHC itself (about 65) , about 80% of the comments were negative. This includes responses to the "Suggestions for improvement" question that suggested improving compile times.
It is true that performance, both of the generated code and GHC itself, has not been at the top of our agenda recently, although we do try to ensure that things don't get worse with each release. As it happens though, we were planning to have a thorough investigation into performance issues soon, and performance of GHC itself is something we'll try to improve (stay tuned for an announcement of a forthocoming GHC performance-fest).
Performance of compiled codeNearly as many people commented on the performance of compiled code as on the performance of GHC itself (in some cases it wasn't clear which, but the two are linked in any case).
About 2/3 of those comments wanted faster or smaller code.
Bootstrapping and portability
An oft-mentioned concern was the difficulty of bootstrapping GHC on a new platform:
The conclusion of that disccusion on the mailing list was that it would be possible to improve the bootstrap experience, at least for Unix users:
Related to this is the question of GHC's portability, which several people raised as a concern:
API stability, and backwards-compatibility
The question of stability was raised quite often.
Backwards-compatibility problems arise because we need to make changes to experimental features or libraries, and want to do so without accumulating too much cruft. However, in the future we will strive to clearly indicate which features and libraries are considered experimental.
Many people thought that GHC's rapid acquisition of new features was a positive thing:
Error ReportingTo our surprise, many people thought that GHC's error reporting was lacking in various ways. Of those that commented on the error messages, about 70% reported problems:
We urge everyone to report concrete examples of unhelpful error messages (and the code that produced them) - this is the best way we can improve the situation.
Lack of documentation is a frequent source of frustration. However, it is clear that some respondents wanted more general Haskell documentation: well-written introductory material and lots of code samples were requested.
The other areas of documentation that people found lacking were:
Support and community
Plenty of people were enthusiastic about the support they had received from the GHC developers and the community in general.
Accessibility of the codeSeveral people identified the problem that GHC needs to be both a research vehicle as well as a get-the-job-done professional product, and these goals are opposing. The research-vehicle aspect means that features are lobbed in all the time, making the implementation ever more complex, and affecting things like maintainability and portability.
Concerns were raised over the accessibility of the code, mainly its size and complexity, and the fact that this makes it difficult to contribute. We are sensitive to this - GHC relies heavily on external contributors, and we always need more.
The conclusion here is that we should try to make the project more accessible to contributors. We'll discuss how exactly to do this separately, and ideas are welcome.
Raw comments: http://www.haskell.org/ghc/survey2005/platform_comments
Generally, folk are happy that they can use GHC on multiple platforms. There were several requests for a GHC port to a platform we don't currently support, with the following being the most requested:
As for .NET, there have been several attempts, and there is at least one on-going project to generate .NET code. Simon PJ posted a long message on haskell-cafe recently explaining the issues behind compiling to .NET and why we haven't done it yet:
The Cygwin port is latent, waiting for someone to pick it up. Volunteers are welcome!
There was the occasional sentiment expressed that the Windows platform isn't given the first-class status it deserves by the developers. The survey stats show that nearly half our users use GHC on Windows, so it's clearly important. We've started to address this with nightly builds and snapshots on Windows.
Raw comments: http://www.haskell.org/ghc/survey2005/features_comments
The options on the survey could have been better here. Lots of people clearly left the boxes as "didn't answer" when they meant "never use". There should have been an option for "don't know what it is". The options "never use" and "nice to have" are not mutually exclusive.
Several people commented that they weren't aware that GHC supported all these features, and they learned something by filling in the survey :-)
We can produce a rough indication of the popularity of features, by giving "nice to have" 1 point and "essential to me" 2 points. The top 5 are:
The bottom 5:
Suggestions for Improvements
Here I'll list the features that were requested, in rough order of popularity. Some have already been mentioned: better performance, better error messages, easier bootstrapping, making GHC easier to build and hack on, better documentation, better backwards compatibility.
Raw comments: http://www.haskell.org/ghc/survey2005/devmodel_comments
Several people didn't understand this question. Clearly we should have been more explicit rather than using the open-source term "development model". For this reason, I don't think we can draw much from the numerical breakdown for this question.
However, the comments form those that did understand the question were generally positive.
The difficulty of getting to the stage where one can contribute effectively was raised several times (see "accessibility of the code").
We asked whether releases are made often enough, and of those that had an opinion, 91% said that releases were frequent enough.
A few people thought releases were too frequent; in many cases this tied in with the problems raised about backwards compatibility. This despite the fact that there was more than a year between the last two major releases, and no code-breaking changes were made in the interrim.
It's not possible to have both perfect backwards compatbility and also introduce new features and evolve the libraries. Keeping backwards compatibility also imposes a maintenance overhead and bloats the codebase. So whatever situation we adopt will be a compromise; what we can take from the survey is that the community would certainly not like major releases to happen *more* often, especially if they break code and/or build systems. If anything, we could lean towards producing more stable releases of each line. The number of patchlevel releases is generally driven by two factors: the number of bug reports against the most recent release, traded off against the difficulty of merging fixes from the trunk. The longer a stable branch remains active, the harder it gets to fix bugs in both the stable branch and the head, which eventually necessitates making a new major release.
People are happy with the stability of releases (96% of those who had an opinion thought that releases were stable enough), with several people commenting that they thought stability had improved over the last few releases.
The GHC Team would like to express its sincere thanks to everyone who responded to the survey. It's provided us with a wealth of information on how people use GHC, what aspects of GHC they enjoy and find most important, and how they would like to see GHC evolve.
We'll certainly be taking your opinions into account as we decide where to go next, but as always we like to think of GHC as belonging to the community, so to a some extent the direction of GHC is decided by who gets involved and how they contribute. We're always keen to help people get started with hacking GHC, so if you're interested please join firstname.lastname@example.org, find something to work on, post your questions, and let us know how we can make things more accessible.
Extra thanks to Tomasz Zielonka for implementing the survey and collecting the responses.
Cheers! -- The GHC Team.