Code, photos, books, and anything else

Subscribe to my RSS feed

Using the new NCover

Introduction

I’ve been using NCover for some time, specifically the free version, to gather test coverage statistics when I run my unit tests. I even wrote about it some time ago in my series on .NET tooling and build automation. That’s worked well since, but the free version has not been updated, and is not compatible with .NET 4. As a result, I decided to take a look at the newer commercial version.

To start, I’ll briefly introduce NCover and provide an overview of the latest version. Later on I’ll show how to integrate it into an automated build using NAnt.

The New(er) NCover

NCover was originally a free tool; a few years ago, it became a commercial tool, starting at v2. The current release is v3.4.8.

The free version of NCover is just a console program that gathers coverage data and outputs an XML file. NCoverExplorer was created by Grant Drake to supplement the NCover console application with a GUI and report-generation capability. When NCover went commercial, NCoverExplorer became part of the product. This unified version works much better than the free version; the separate programs were somewhat clunky to get working together, but they are now nicely integrated.

The new NCover distribution has the following programs:

  • NCover.Console – Handles the running of unit tests and gathering of coverage data.
  • NCover.Explorer – Provides a GUI frontend to NCover and NCover.Reporting.
  • NCover.Reporting – Handles generation of reports from coverage data.

I’ll show you around the new product, NCoverExplorer particularly, since that’s the GUI. But I’ll show how to use the other components as well.

When you open NCoverExplorer, you’ll see something like the following:

The main NCoverExplorer window

(you can click any of the images in this post for the full size version)

To create a project file – which contains the settings necessary to gather your coverage data – click on Project Actions->New Project. That will bring up another window, shown below. There are a lot of options available, though not all are necessary. I’ll point out the more useful ones; I’ve filled out the fields necessary to collect coverage data on one of my projects.

The first screen has the basic settings: the test runner, arguments for the same, your build folder, coverage file, and trends file.

The first project settings pane

In the Output Settings section, you can choose the types of data to be collected, where source data (.pdb files) can be found, and log file location; there are several other options here, but they are less prevalent. There is the option to output HTML reports automatically by specifying an output path, but this capability is deprecated and will be removed in NCover 4.

Setting project's output settings

Here is a useful option to include and exclude data at various levels, including assemblies, types, and files. Clicking the Add button will allow you to navigate to and select your assemblies to be included or excluded; the rest require manual typing.

Configuring coverage exclusions

The last options section is titled NCover Path; you need to provide the path to the NCover console application, since NCoverExplorer is just the GUI, and delegates to NCover. You’ll also need to check the first checkbox, labeled Register the NCover profiler assembly before running, for NCover to work properly.

Minor extra settings related to the NCover console

That’s the last of the configuration. You can save the project file by clicking Project Actions->Save Project, and can return to the configuration view at any time via Project Actions->Project Options. Below is what my NCover project file looks like:

[code lang=”xml”] IIStoolsnunitChecklist.Tests.dll /noshadowtoolsnunitnunit-console.execoverage.trendSymbol, Branch, CyclomaticComplexity, MethodVisitscoverage.nccovFalseW3SVCFalseChecklist.TestsFalsereportsNCovercoverage.logVerboseTrueChecklistFalseFalseRegistry, SymbolServer, BuildPath, ExecutingDir01toolsncoverncover.console.exeChecklist01a6f7cd-0f0b-4736-8e1d-2e4a58aa2144NoneBranchCoverageViewFalse80CyclomaticComplexityViewFalse20MethodCoverageViewFalse95SymbolCoverageViewFalse95NameSequencePointCoveragePercentageTrueFalseAeLookupSvcTrue[/code]

I’ll point out that while NCover adds paths as absolute paths, the file can be edited to convert them to relative paths (as shown above), relative to the location of the project file. NCoverExplorer will respect those edits. This makes it possible to check NCover project files into source control for use on other machines, without needing to adjust the paths in the project file.

To run your tests and get the coverage data, just click the Run Coverage button; you’ll get an output pane similar to the following:

Running tests, gathering coverage data

Clearly I don’t have a lot of tests so far in this project, but it’s not a big project – yet, anyway. But I check with NCover regularly to make sure I don’t have any untested code.

After the tests have run and NCover has completed, there will be two panes at the top half of the window: one is a tree view of your project and the other is a source view, which highlights covered and uncovered code. You can close the output pane from earlier to maximize viewing space.

Viewing coverage results

One noticeable new addition over the free NCover is the addition of the trends view, accessed by clicking Trends & Statistics. It shows the trends in your coverage data over time, assuming you specified a trends data file earlier on. This file is appended each time you generate coverage data, collecting the historical data for later comparison.

Note also the display of types of coverage; besides symbol coverage, there are also branch coverage and method coverage metrics. These are somewhat new in NCover, and provide additional detail over just the overall coverage data. NCover also gathers and displays data on your code’s cyclomatic complexity.

Viewing coverage trends

The interactive coverage data explorer is a nice tool, and provides quick access to the coverage results. Sometimes you still want a static representation of the data, perhaps for reporting purposes. This can be provided by the NCoverReporting facility; this is a console program that is delegated to by NCoverExplorer, and specializes in generating reports based off coverage data.

Via the Generate Reports button, you can generate a variety of reports, as shown below. Some reports can be generated in either HTML or XML, while others are HTML only. You can specify the output directory for the reports, which can be an absolute path or one relative to the location of the NCover project file. Make your selections, then hit the Generate Report button.

Choosing report generation options

There is a listing on the NCover website showing which report types are available; there are 24 report types available in Complete, a subset of which are also available in Classic. Below is a screenshot of a summary report; it suggests that I may be missing a test or two:

Screenshot of summary report

As a final point of interest, NCoverExplorer can provide you scripts to run NCover on your assemblies based on the settings you provided earlier. Click on the Script Helper button to see a new dialog. You can choose from the command-line version, or variants that can be run by MSBuild or NAnt. This is quite helpful for integrating NCover into automated build systems.

Getting appropriate code for console use

That wraps up the look at the latest version of NCover and its capabilities. Now I’ll show how to use it in an automated build via NAnt, same as before.

Automation Integration

Updating NCover

As I’ve made clear in my previous tooling articles, I like to have build-related tools kept in the same repository as the source code, so I can run a build after a single checkout. My previous article on NCover had me adding NCover and NCoverExplorer to my project repository as separate projects, which they were at the time. I also had a separate folder for NAnt tasks for NCoverExplorer, for some reason.

I deleted the later two folders from my tools, and updated the contents of the NCover folder, removing older unused files and adding new ones where appropriate. NCover isn’t distributed as a binary archive, just an installer. You can run the installer, and extract the files from the installation directory – NCoverExplorer and NCoverReporting are bundled with NCover. They seem to work fine as standalone tools.

With the new NCover files in place, and the old ones gone, it’s now time to look at running the tools via NAnt. Just as the tooling setup has changed, so has the usage of those tools.

Running NCover via NAnt

In my previous article on NCover, I showed a test coverage target – creatively named ncover – that I use in my NAnt build files. NCover’s command-line interface has changed from the free version, so the coverage collection and report generation processes need to be updated. Following is the start of the revised ncover target:

[code lang=”xml”]

[/code]

The first line loads the NCover tasks from the project’s tools directory; the result is the ability to gather coverage data and generate reports. The second line defines the NCover output directory – NCover will create the directory if it does not exist.

Next is the ncover task:

[code lang=”xml”][/code]

This is the quick and dirty way to generate coverage data in XML form. It is much the same from my previous article, though some of the attribute names have been changed, others have been added, and there is no assemblies element anymore. I’ve added some remarks on the attributes below; full documentation is available.

  • program – the location of the NCover console – here it is found in the project’s tools directory.
  • testRunnerExe – the program being profiled – in this case, NUnit.
  • testRunnerArgs – the arguments to pass to the program being profiled – need to provide at least the test assembly name.
    • /noshadow does not seem to be required, but its use is recommended to ensure NCover works properly.
    • Note: if you are compiling for .NET 4, NUnit doesn’t support running such assemblies straight out of the box…you can make it work by giving an additional /framework=4.0.30319 parameter.
  • excludeAssemblies – a regular expression to exclude assemblies from being included in coverage data…not much point in profiling the tests assembly, so I exclude it here.
    • You can do a variety of inclusions and exclusions, for attributes, assemblies, files, types, and even methods. Some of the include/exclude permutations are only in Complete.
  • workingDirectory – the directory where your compiled assemblies are located.
  • logLevel – the level of logging NCover should do; the options are None (default), Normal, and Verbose.
  • logFile – path to log file for NCover.
  • coverageFile – path to coverage data file.
  • appendTrendTo – path to coverage trends file.
  • registerProfiler – temporarily registers the profiler on the system…seems to be required for the profiler to work correctly.

I will point out that there is the option to output a full set of HTML reports from this task by using the htmlReportDir attribute; however, this capability is deprecated and will be removed in NCover 4. It is better to use the specialized reporting facility, which will be shown next.

At this point, the coverage data has been collected in XML form; there is another step required to transform that data into readable reports, achieved with the following:

[code lang=”xml”][/code]

The task used to be ncoverexplorer, but now it is ncoverreporting, since that is the component that now handles the report generation, as mentioned earlier. There are full details for this task as well; only a fraction of the options are shown here. It can be more complex than the ncover one, particularly since it involves child elements as well as attributes. Again, some remarks follow:

  • program – the path to the NCoverReporting console application.
  • projectName – the name of the current project, which will be shown on the resulting output.
  • outputPath – the output path for the generated report(s).
  • buildId – the build ID you want shown on the report, can be any string; the current build version is a good candidate.
  • coverageDataPaths – the location of the coverage data file to use for the report; can pull in multiple files with additional include elements, if you tend to generate coverage data for different parts of your project.
  • reports – the reports you want to generate. The summary is the quick and dirty option.

I don’t use it here, but it is possible to report on whether satisfactory coverage levels are met, at the assembly, namespace, and class levels. There is also the option to fail the overall build if the satisfactory coverage level is not met. This is a Complete edition feature, and is covered in the NCoverReporting documentation.

You can generate multiple reports at the same time, based off the same coverage data, by adding another report to the reports list. The types of reports you can generate are based on the NCover edition you are using, as indicated before. Having used the free NCover for some time, I’m used to just getting the summary report, which will highlight any glaring coverage gaps, and then drilling down via NCoverExplorer. It’s worth experimenting with the different report options to determine which ones are most useful to you.

Below is the complete ncover target:

[code lang=”xml”]

  <property name="dir.reports.ncover" value="${dir.reports}NCover" />

  <ncover program="${dir.tools}ncoverNCover.Console.exe"
               testRunnerExe="${dir.tools}nunitnunit-console.exe"
               testRunnerArgs="${file.assembly.test} /noshadow /framework=4.0.30319"
               excludeAssemblies=".*Tests"
               workingDirectory="${dir.build}"
               logLevel="Normal"
               logFile="${dir.reports.ncover}coverage.log"
               coverageFile="${dir.reports.ncover}coverage.xml"
               appendTrendTo="${dir.reports.ncover}coverage.trend"
               registerProfiler="true" />

  <ncoverreporting program="${dir.tools}ncoverNCover.Reporting.exe"
                             projectName="${project::get-name()}"
                             outputPath="${dir.reports.ncover}"
                             buildId="${project.fullversion}">
        <coverageDataPaths>
              <include name="${dir.reports.ncover}coverage.xml" />
        </coverageDataPaths>
        <reports>
              <report reportType="Summary" format="Html" />
        </reports>
  </ncoverreporting>

[/code]

With the new ncover target in place, test coverage can be collected and a report generated with a single command, resulting in something like the following:

[[ncover target output]]

I’ll stop to point out a minor annoyance. Other tools I’ve written about, namely FxCop, NDepend, and Sandcastle Help File Builder, allow you to pass the project files they generate to their respective console tools, saving the need to have the necessary configuration in two places. NCover doesn’t seem to have such a feature, which requires the use of the (largish) NAnt target I showed above. Admittedly it’s not all that much, but the repetition is slightly irksome. I’ve made the suggestion to NCover to add the capability – I imagine I’m not the only one who would appreciate having that option!

Conclusion

The latest version of NCover is a worthwhile upgrade from the free version. In fact, it is required if you want to keep using NCover to analyze .NET 4 assemblies. But the improvements are numerous, the integration is better, and the overall experience simplified.

For many users, the Classic edition will be enough; it provides the GUI, build integration, and a number of useful reports. NCover Complete adds a few more features, along with many more report options. For those used to the free NCover, Classic is a step up – you lose nothing in the way of features – while Complete is more of a jump.

The licensing scheme is simple: you buy the software license (the software can be downloaded) and can optionally buy a subscription, which provides access to customer support as well as future upgrades. Even if you don’t buy a subscription, or if you let it lapse, the license is perpetual, so you can still use the software in the future. The subscription costs less than the license, so paying a yearly subscription fee costs less than buying a new license whenever an upgrade is released.

Related Reading

Author:

Format:

Category:
Programming

Tags:

1 Comment
Comments closed

Comments

#1

Shaun Wilde

June 22, 2010 @ 9:19 pm

PartCover is free and now handles .NET4

Leave a Comment

Apologies, commenting is now disabled for this post.