Personal tools

Haskell program coverage

From HaskellWiki

(Difference between revisions)
Jump to: navigation, search
m (Wiki Style on headlines as per guidelines, fix bulleted list formats)
 
(11 intermediate revisions by 3 users not shown)
Line 35: Line 35:
 
and we gladly acknowledge the contribution of the original authors.
 
and we gladly acknowledge the contribution of the original authors.
   
The latest version is version 0.3, (and a DARCS repo coming soon,
+
The latest version is version 0.4 and can be found at:
as well as tarballs of the various tools).
+
http://projects.unsafePerformIO.com/hpc
   
 
== Examples ==
 
== Examples ==
Line 55: Line 55:
 
showRecip.p
 
showRecip.p
   
=== Example of HTML output from hpc-source ===
+
=== Example of HTML output from hpc-markup ===
   
 
[[Image:hpcexample.gif]]
 
[[Image:hpcexample.gif]]
Line 62: Line 62:
 
or <font style="background: red">always False</font>.
 
or <font style="background: red">always False</font>.
   
== Hpc Toolkit ==
+
=== Example of HTML Summary from hpc-markup ===
  +
  +
This is an example of the table that provides the summary of coverage,
  +
with links the the individually marked-up files.
  +
  +
<table width="100%" border=1>
  +
<tr><th rowspan=2>module</th><th colspan=3>Top Level Definitions</th><th colspan=3>Alternatives</th><th colspan=3>Expressions</th></tr><tr><th>%</th><th colspan=2>covered / total</th><th>%</th><th colspan=2>covered / total</th><th>%</th><th colspan=2>covered / total</th></tr><tr>
  +
<td>&nbsp;&nbsp;<tt>module CSG</tt></td>
  +
<td align="right">100 %</td><td>0/0</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="100%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">100 %</td><td>0/0</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="100%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">100 %</td><td>0/0</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="100%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Construct</tt></td>
  +
<td align="right">48 %</td><td>17/35</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="48%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">52 %</td><td>25/48</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="52%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">60 %</td><td>381/635</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="60%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Data</tt></td>
  +
<td align="right">24 %</td><td>6/25</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="24%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">13 %</td><td>11/81</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="13%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">39 %</td><td>254/646</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="39%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Eval</tt></td>
  +
<td align="right">70 %</td><td>22/31</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="70%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">60 %</td><td>65/108</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="60%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">57 %</td><td>361/628</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="57%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Geometry</tt></td>
  +
<td align="right">75 %</td><td>42/56</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="75%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">69 %</td><td>45/65</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="69%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">70 %</td><td>300/427</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="70%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Illumination</tt></td>
  +
<td align="right">61 %</td><td>11/18</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="61%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">49 %</td><td>46/93</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="49%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">46 %</td><td>279/600</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="46%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Intersections</tt></td>
  +
<td align="right">63 %</td><td>14/22</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="63%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">38 %</td><td>83/213</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="38%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">38 %</td><td>382/1001</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="38%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Interval</tt></td>
  +
<td align="right">47 %</td><td>8/17</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="47%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">41 %</td><td>16/39</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="41%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">41 %</td><td>69/165</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="41%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Main</tt></td>
  +
<td align="right">100 %</td><td>1/1</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="100%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">100 %</td><td>1/1</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="100%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">100 %</td><td>6/6</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="100%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Misc</tt></td>
  +
<td align="right">0 %</td><td>0/1</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="0%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">0 %</td><td>0/1</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="0%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">0 %</td><td>0/10</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="0%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Parse</tt></td>
  +
<td align="right">80 %</td><td>16/20</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="80%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">68 %</td><td>26/38</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="68%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">72 %</td><td>192/264</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="72%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Primitives</tt></td>
  +
<td align="right">16 %</td><td>1/6</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="16%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">16 %</td><td>1/6</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="16%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">20 %</td><td>5/24</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="20%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
<tr>
  +
<td>&nbsp;&nbsp;<tt>module Surface</tt></td>
  +
<td align="right">36 %</td><td>4/11</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="36%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">24 %</td><td>13/53</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="24%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table><td align="right">18 %</td><td>43/231</td><td width=100><table cellpadding=0 cellspacing=0 width="100" style="background-color: #f25913"><tr><td><table cellpadding=0 cellspacing=0 width="18%"><tr><td height=12 style="background-color: #60de51"></td></tr></table></td></tr></table></tr>
  +
  +
</table>
  +
  +
== Hpc toolkit ==
   
 
The Hpc Toolkit has three parts
 
The Hpc Toolkit has three parts
Line 69: Line 69:
 
* An open common file format, uses by all the tools.
 
* An open common file format, uses by all the tools.
   
=== Hpc Tools ===
+
=== Hpc tools ===
   
There are currently three tools provided by hpc, as well as a new option for GHC 6.6.1.
+
There are currently three tools provided by hpc, as well as a new option for GHC 6.7.
   
 
Tools
 
Tools
Line 83: Line 83:
 
* -fhpc - a new GHC option, that automatically instruments Haskell programs on the fly, using a similar algorithm to hpc-trans.
 
* -fhpc - a new GHC option, that automatically instruments Haskell programs on the fly, using a similar algorithm to hpc-trans.
   
*: This option has three variations
+
=== Hpc scripts ===
** -fhpc - gather coverage information.
 
** -fhpc-count - gather coverage information, with accurage usage count.
 
** -fhpc-trace - gather coverage information dynamically, for replay.
 
   
=== Hpc Scripts ===
+
* hpc-build
  +
* hpc-run
   
<code>hpc</code> - driver script to make using hpc-trans easy.
 
* hpc build
 
* hpc run
 
* hpc report
 
* hpc markup
 
   
+
=== Hpc file formats ===
=== Hpc File Formats ===
 
   
 
There are two file formats used by Hpc externally, and one internally
 
There are two file formats used by Hpc externally, and one internally
Line 100: Line 95:
 
* tix - Coverage information generated by the execution of an instrumented program.
 
* tix - Coverage information generated by the execution of an instrumented program.
 
* mix - Information about each tick-box inside a module, including style of tick box, and location in the source.
 
* mix - Information about each tick-box inside a module, including style of tick box, and location in the source.
* pix - used only by hpc-trans, stores a list
+
* pix - used only by hpc-trans, stores a list
  +
  +
=== Example ===
  +
  +
Suppose you have run your tests and collected the coverage results in a .tix file (you don't have to do anything to collect the results - hpc will keep adding coverage information to the .tix file).
  +
  +
Now you want to see how good your testing was:
  +
  +
hpc report GenDesc.exe --hpcdir=..\src\.hpc --srcdir=..\src --per-module --include=Utils.Substitution
  +
  +
<nowiki>-----<module Utils.Substitution>-----</nowiki>
  +
65% expressions used (59/90)
  +
33% boolean coverage (1/3)
  +
0% guards (0/1), 1 always True
  +
50% 'if' conditions (1/2), 1 always True
  +
100% qualifiers (0/0)
  +
62% alternatives used (5/8)
  +
66% local declarations used (2/3)
  +
71% top-level declarations used (5/7)
  +
  +
But you know that some of the source statements are assertions and should never be executed. First create a draft overlay that provides 100% coverage.
  +
  +
hpc draft --hpcdir=..\src\.hpc --srcdir=..\src GenDesc.exe.tix > myDraft.txt
  +
  +
You can then create a .tix file from the overlay.
  +
  +
hpc overlay --hpcdir=..\src\.hpc --srcdir=..\src myDraft.txt > myDraft.tix
  +
  +
And combine these two .tix files.
  +
  +
hpc combine GenDesc.exe.tix myDraft2.tix --union > idem.tix
  +
  +
Now you get a report that doesn't include assertions e.g. calls to assertM and error (although I'm basing this on observation - I couldn't find any documentation on this).
  +
  +
hpc report idem --hpcdir=..\src\.hpc --srcdir=..\src --per-module
  +
--include=Utils.Substitution
  +
  +
<nowiki>-----<module Utils.Substitution>-----</nowiki>
  +
91% expressions used (82/90)
  +
33% boolean coverage (1/3)
  +
0% guards (0/1), 1 always True
  +
50% 'if' conditions (1/2), 1 always True
  +
100% qualifiers (0/0)
  +
87% alternatives used (7/8)
  +
66% local declarations used (2/3)
  +
71% top-level declarations used (5/7)
  +
  +
But you still know that some of the functions should not be tested; they are there just in case you need to make some performance improvements in the future.
  +
  +
Create an overlay file using the draft overlay as a template.
  +
  +
module "Utils.Substitution" {
  +
tick function "getFreesShallow" [future];
  +
tick function "getFreesUsingPadrec" [future];
  +
}
  +
  +
Create a .tix file from it.
  +
  +
hpc overlay --hpcdir=..\src\.hpc --srcdir=..\src myAnnot.txt > myAnnot.tix
  +
  +
And then combine it with the automatically generated exclusions .tix file.
  +
  +
hpc combine idem.tix myAnnot.tix --union > excludeFuture.tix
  +
  +
Now our report gives us (almost) what we want: 100% coverage.
  +
  +
hpc report excludeFuture --hpcdir=..\src\.hpc --srcdir=..\src --per-module
  +
--include=Utils.Substitution
  +
  +
<nowiki>-----<module Utils.Substitution>-----</nowiki>
  +
100% expressions used (90/90)
  +
33% boolean coverage (1/3)
  +
0% guards (0/1), 1 always True
  +
50% 'if' conditions (1/2), 1 always True
  +
100% qualifiers (0/0)
  +
100% alternatives used (8/8)
  +
100% local declarations used (3/3)
  +
100% top-level declarations used (7/7)
  +
  +
Sadly there is no documentation on how to exclude boolean coverage.
  +
  +
== Hpc quirks ==
  +
  +
Hpc (at least with 6.10.1) is not happy with literate haskell or with cpp. Even if your file contains no cpp, hpc will give spurious results if you use the cpp option on the command line. One way round both of these problems is to pre-process the source by hand with something like
  +
  +
<nowiki>ghc -E -optP-P -cpp Foo.hs</nowiki>
  +
  +
and then remove the line
  +
  +
<nowiki>{-# LINE 1 "Foo.hs" #-}</nowiki>
  +
  +
Don't use strip on your executables. I'm guessing hpc stashes information in the executable which strip removes.
   
 
----
 
----

Latest revision as of 15:51, 15 July 2009

Contents

[edit] 1 What is hpc?

Hpc is a tool-kit to record and display Haskell program coverage. Hpc includes tools that instrument Haskell programs to record program coverage, run instrumented programs, and display the coverage information obtained.

Hpc works by applying a source-to-source transformation; this transformation also generates as a by-product a program-index file (.pix) and module-index files (.mix). The transformed program is compiled with a library; in addition to its usual run-time behaviour the program generates a coverage record in a program-ticks file (.tix). If the program is run more than once, coverage data is accumulated to reflect all runs.

Hpc provides coverage information of two kinds: source coverage and boolean-control coverage. Source coverage is the extent to which every part of the program was used, measured at three different levels: declarations (both top-level and local), alternatives (among several equations or case branches) and expressions (at every level). Boolean coverage is the extent to which each of the values True and False is obtained in every syntactic boolean context (ie. guard, condition, qualifier).

Hpc displays both kinds of information in two different ways: textual reports with summary statistics (hpc-report) and sources with colour mark-up (hpc-source).

[edit] 2 Downloading

This version of hpc is available under a BSD-style license for free use by all sectors of the Haskell community. The hpc-trans tool was based on components from the nhc98 compiler or from the hat tracing system, and we gladly acknowledge the contribution of the original authors.

The latest version is version 0.4 and can be found at:

  http://projects.unsafePerformIO.com/hpc

[edit] 3 Examples

[edit] 3.1 Example textual output from hpc-report

-----<module Main>-----

 67% expressions used (72/106)
 14% boolean coverage (1/7)
     16% guards (1/6), 2 always True, 2 always False, 1 unevaluated
      0% 'if' conditions (0/1), 1 always True
    100% qualifiers (0/0)
 42% alternatives used (3/7)
 88% local declarations used (8/9)
 80% top-level declarations used (4/5)
unused declarations:
    position
    showRecip.p

[edit] 3.2 Example of HTML output from hpc-markup

Hpcexample.gif

The HTML output highlights parts of the program never evaluated; it also highlights boolean conditions for which recorded evaluations are always True or always False.

[edit] 3.3 Example of HTML Summary from hpc-markup

This is an example of the table that provides the summary of coverage, with links the the individually marked-up files.

moduleTop Level DefinitionsAlternativesExpressions
%covered / total%covered / total%covered / total
  module CSG 100 %0/0
100 %0/0
100 %0/0
  module Construct 48 %17/35
52 %25/48
60 %381/635
  module Data 24 %6/25
13 %11/81
39 %254/646
  module Eval 70 %22/31
60 %65/108
57 %361/628
  module Geometry 75 %42/56
69 %45/65
70 %300/427
  module Illumination 61 %11/18
49 %46/93
46 %279/600
  module Intersections 63 %14/22
38 %83/213
38 %382/1001
  module Interval 47 %8/17
41 %16/39
41 %69/165
  module Main 100 %1/1
100 %1/1
100 %6/6
  module Misc 0 %0/1
0 %0/1
0 %0/10
  module Parse 80 %16/20
68 %26/38
72 %192/264
  module Primitives 16 %1/6
16 %1/6
20 %5/24
  module Surface 36 %4/11
24 %13/53
18 %43/231

[edit] 4 Hpc toolkit

The Hpc Toolkit has three parts

  • A set of tools for instrumenting Haskell, and interpreting the results on a coverage run.
  • A set of scripts to make using these tools easier.
  • An open common file format, uses by all the tools.

[edit] 4.1 Hpc tools

There are currently three tools provided by hpc, as well as a new option for GHC 6.7.

Tools

  • hpc-trans - translates Haskell into instrumented Haskell
  • hpc-report - Read the output of a coverage run, report a summary
  • hpc-markup - Read the output of a coverage run, markup the source

Compiler Options

  • -fhpc - a new GHC option, that automatically instruments Haskell programs on the fly, using a similar algorithm to hpc-trans.

[edit] 4.2 Hpc scripts

  • hpc-build
  • hpc-run


[edit] 4.3 Hpc file formats

There are two file formats used by Hpc externally, and one internally by hpc-trans.

  • tix - Coverage information generated by the execution of an instrumented program.
  • mix - Information about each tick-box inside a module, including style of tick box, and location in the source.
  • pix - used only by hpc-trans, stores a list

[edit] 4.4 Example

Suppose you have run your tests and collected the coverage results in a .tix file (you don't have to do anything to collect the results - hpc will keep adding coverage information to the .tix file).

Now you want to see how good your testing was:

hpc report GenDesc.exe --hpcdir=..\src\.hpc --srcdir=..\src --per-module --include=Utils.Substitution

-----<module Utils.Substitution>-----

65% expressions used (59/90)
33% boolean coverage (1/3)
      0% guards (0/1), 1 always True
     50% 'if' conditions (1/2), 1 always True
    100% qualifiers (0/0)
62% alternatives used (5/8)
66% local declarations used (2/3)
71% top-level declarations used (5/7)

But you know that some of the source statements are assertions and should never be executed. First create a draft overlay that provides 100% coverage.

hpc draft --hpcdir=..\src\.hpc --srcdir=..\src GenDesc.exe.tix > myDraft.txt

You can then create a .tix file from the overlay.

hpc overlay --hpcdir=..\src\.hpc --srcdir=..\src myDraft.txt > myDraft.tix

And combine these two .tix files.

hpc combine GenDesc.exe.tix myDraft2.tix --union > idem.tix

Now you get a report that doesn't include assertions e.g. calls to assertM and error (although I'm basing this on observation - I couldn't find any documentation on this).

hpc report idem --hpcdir=..\src\.hpc --srcdir=..\src --per-module
--include=Utils.Substitution

-----<module Utils.Substitution>-----

91% expressions used (82/90)
33% boolean coverage (1/3)
      0% guards (0/1), 1 always True
     50% 'if' conditions (1/2), 1 always True
    100% qualifiers (0/0)
87% alternatives used (7/8)
66% local declarations used (2/3)
71% top-level declarations used (5/7)

But you still know that some of the functions should not be tested; they are there just in case you need to make some performance improvements in the future.

Create an overlay file using the draft overlay as a template.

module "Utils.Substitution" {
  tick function "getFreesShallow" [future];
  tick function "getFreesUsingPadrec" [future];
}

Create a .tix file from it.

hpc overlay --hpcdir=..\src\.hpc --srcdir=..\src myAnnot.txt > myAnnot.tix

And then combine it with the automatically generated exclusions .tix file.

hpc combine idem.tix myAnnot.tix --union > excludeFuture.tix

Now our report gives us (almost) what we want: 100% coverage.

hpc report excludeFuture --hpcdir=..\src\.hpc --srcdir=..\src --per-module
--include=Utils.Substitution

-----<module Utils.Substitution>-----

100% expressions used (90/90)
 33% boolean coverage (1/3)
       0% guards (0/1), 1 always True
      50% 'if' conditions (1/2), 1 always True
     100% qualifiers (0/0)
100% alternatives used (8/8)
100% local declarations used (3/3)
100% top-level declarations used (7/7)

Sadly there is no documentation on how to exclude boolean coverage.

[edit] 5 Hpc quirks

Hpc (at least with 6.10.1) is not happy with literate haskell or with cpp. Even if your file contains no cpp, hpc will give spurious results if you use the cpp option on the command line. One way round both of these problems is to pre-process the source by hand with something like

ghc -E -optP-P -cpp Foo.hs

and then remove the line

{-# LINE 1 "Foo.hs" #-}

Don't use strip on your executables. I'm guessing hpc stashes information in the executable which strip removes.


We hope you find this tool-kit useful. If you have any comments or feedback, please feel free to email us.

Andy Gill (andy@galois.com) Colin Runciman (colin@cs.york.ac.uk)