This entry is going to document how we separated our unit tests from our integration tests using MbUnit, NAnt, and TestDriven.NET. In detail!
At work, we've recently kicked into full-on Test Driven Development. We're loving it. Our tool stack as it relates to TDD is as follows:
As we wrote tests, we distinguished between unit tests, which exercise a particular class or small group of classes, and integration tests, which cross application layers and may require a locally installed copy of the system under test to execute properly. Our objective was to have unit tests run during continuous integration and during the TDD process, but not have the integration tests run unless we specifically request it.
MbUnit supports a [FixtureCategory(string category)] attribute on a TestFixture (i.e. class). It seemed reasonable that each TestFixture should contain either unit tests or integration tests, but not both, so we made sure that was true and applied FixtureCategory attributes to every TestFixture in our test assemblies. For example:
C#:
-
[TestFixture]
-
[FixtureCategory("unit")]
-
public class FooTest
-
{
-
[Test]
-
public void MeaningOfLifeReturns42()
-
{
-
-
int answer = foo.MeaningOfLife();
-
Assert.AreEqual(42, answer);
-
}
-
}
The next step was to get our build script capable of targeting the type of test we want to run. This can be accomplished by specifying the /filter-category:category flag in the call to mbunit.cons.exe. The relevant targets in my NAnt file ended up looking like this:
XML:
-
<target name="unit-test" depends="compile" description="runs automated unit tests">
-
<property name="test.category" value="unit" />
-
<call target="category-test" />
-
</target>
-
-
<target name="integration-test" depends="compile" description="runs automated integration tests">
-
<property name="test.category" value="integration" />
-
<call target="category-test" />
-
</target>
-
-
<target name="category-test">
-
<if test="${not(property::exists('test.category'))}">
-
<fail message="the category-test target cannot be run without setting the test.category property." />
-
</if>
-
<property name="test.dir" value="${build.dir}-${test.category}-test" />
-
<mkdir dir="${test.dir}"/>
-
<exec program="tools\msbuild\msbuild">
-
<arg value="src\${project.name}.proj" />
-
<arg value="/t:AppTests" />
-
<arg value="/p:OutputPath=${test.dir}\" />
-
<arg value="/p:Configuration=${project.config}" />
-
</exec>
-
<exec program="tools\mbunit\mbunit.cons.exe" failonerror="false" workingdir="${test.dir}">
-
<arg value="${test.dir}\FooTest.dll" />
-
<arg value="${test.dir}\BarTest.dll" />
-
<arg value="/report-type:Xml" />
-
<arg value="/report-folder:.." />
-
<arg value="/report-name-format:mbunit-service-${test.category}-results" />
-
<arg value="/filter-category:${test.category}" />
-
</exec>
-
</target>
That worked nicely, and I made sure that the targets called under CruiseControl .NET include "unit-test" and do not include "integration-test".
The next problem was a little more daunting, though. We had mapped a keyboard shortcut to telling the TestDriven.NET plugin to run tests. (We chose F6.) This functionality isi vital to our TDD process. We can hit F6 inside of a test method to run that particular test, inside of a fixture to run all tests in the fixture, inside of a namespace to run all tests in that namespace, and finally, on a project in the Visual Studio Solution Explorer to run all tests in that assembly. It's vital to the TDD process that these test runs be fast. I really wanted only unit tests to be run, and not integration tests.
After a little googling and reflectoring later, I knew that TestDriven .NET looks at a registry key to find registered TestRunners, and picks the appropriate TestRunner for the context that has the lowest priority. After installing TestDriven .NET and MbUnit the registry key [HKEY_CURRENT_USER\Software\MutantDesign\TestDriven.NET\TestRunners] had the following child keys:
CODE:
-
[HKEY_CURRENT_USER\Software\MutantDesign\TestDriven.NET\TestRunners\AdHoc]
-
@="40"
-
"AssemblyPath"="C:\\Program Files\\TestDriven.NET 2.0\\AdHoc\\TestDriven.AdHoc.dll"
-
"TypeName"="TestDriven.AdHoc.TestRunner.AdHocTestRunner"
-
-
[HKEY_CURRENT_USER\Software\MutantDesign\TestDriven.NET\TestRunners\Custom]
-
@="0"
-
"AssemblyPath"="C:\\Program Files\\TestDriven.NET 2.0\\AdHoc\\TestDriven.AdHoc.dll"
-
"TypeName"="TestDriven.AdHoc.TestRunner.CustomTestRunner"
-
"TargetFrameworkAssemblyName"="TestDriven.Framework, Version=2.0.0.0"
-
-
[HKEY_CURRENT_USER\Software\MutantDesign\TestDriven.NET\TestRunners\MbUnit]
-
@="10"
-
"AssemblyPath"="C:\\Program Files\\MbUnit\\MbUnit.AddIn.dll"
-
"TypeName"="MbUnit.AddIn.MbUnitTestRunner"
-
"TargetFrameworkAssemblyName"="MbUnit.Framework"
-
"Application"="C:\\Program Files\\MbUnit\\MbUnit.GUI.exe"
-
-
[HKEY_CURRENT_USER\Software\MutantDesign\TestDriven.NET\TestRunners\NUnit]
-
@="10"
-
"AssemblyPath"="C:\\Program Files\\TestDriven.NET 2.0\\NUnit\\nunit.addin.dll"
-
"TypeName"="NUnit.AddInRunner.NUnitTestRunner"
-
"TargetFrameworkAssemblyName"="nunit.framework"
-
"Application"="C:\\Program Files\\TestDriven.NET 2.0\\NUnit\\nunit-gui.exe"
-
-
[HKEY_CURRENT_USER\Software\MutantDesign\TestDriven.NET\TestRunners\NUnit_VSTS]
-
@="20"
-
"AssemblyPath"="C:\\Program Files\\TestDriven.NET 2.0\\NUnit\\nunit.addin.dll"
-
"TypeName"="NUnit.AddInRunner.NUnitTestRunner"
-
"TargetFrameworkAssemblyName"="Microsoft.VisualStudio.QualityTools.UnitTestFramework"
-
-
[HKEY_CURRENT_USER\Software\MutantDesign\TestDriven.NET\TestRunners\VisualStudioTestTools]
-
"AssemblyPath"="C:\\Program Files\\TestDriven.NET 2.0\\VisualStudioTestTools\\TestDriven.VisualStudioTestTools.dll"
-
"TypeName"="TestDriven.VisualStudioTestTools.VsttTestRunner"
-
"TargetFrameworkAssemblyName"="Microsoft.VisualStudio.QualityTools.UnitTestFramework"
I set out to create an assembly that could be pointed at from this registry key and contained an implementation of ITestRunner that only ran test fixtures that had a FixtureCategory attribute equal to "unit".
It turns out that we can accomplish that without too much code by extending the MbUnit test runner, which is open source. (I really wish TestDriven .NET were open source, but that's beside the point for the time being...) So I created a new Visual Studio solution with a single project: "OxyMbUnit.AddIn." I updated the properties page to specify the assembly name and default namespace both to be "Oxygen.MbUnit.AddIn" and specified a keyfile to give the assembly a strong name.
You'll also need references to the MbUnit.Addin, MbUnit.Framework and TestDriven.Framework assemblies.
Then I added a single class: "OxyMbUnitTestRunner.cs"
C#:
-
using MbUnit.AddIn;
-
using MbUnit.Core.Filters;
-
using TestDriven.Framework;
-
-
namespace Oxygen.MbUnit.AddIn
-
{
-
public sealed class OxyMbUnitTestRunner : MbUnitTestRunner, ITestRunner
-
{
-
TestRunState ITestRunner.RunAssembly(ITestListener testListener, System.Reflection.Assembly assembly)
-
{
-
testListener.WriteLine("OxyMbUnitTestRunner executing tests with the FixtureCategory 'unit'", Category.Info);
-
CategoryFixtureFilter filter =
new CategoryFixtureFilter
("unit");
-
return base.Run(testListener, assembly, filter);
-
}
-
-
TestRunState ITestRunner.RunNamespace(ITestListener testListener, System.Reflection.Assembly assembly, string ns)
-
{
-
testListener.WriteLine("OxyMbUnitTestRunner executing tests with the FixtureCategory 'unit'", Category.Info);
-
AndFixtureFilter filter =
new AndFixtureFilter
(
-
new NamespaceFixtureFilter
(ns
),
-
new CategoryFixtureFilter
("unit")
-
);
-
return Run(testListener, assembly, filter);
-
}
-
}
-
}
Note how we're passing a CategoryFixtureFilter to the base class' Run(...) method. That's all it takes!
Note that the third method (besides RunAssembly and RunNamespace) that is defined as part of the ITestRunner interface is RunMember, which refers to a specific fixture or test. We'll leave our base class' implementation to handle that. If a test or fixture is explcitly selected, we want TestDriven .NET to run it, even if it is an integration test.
Once we have that assembly building, all we need is the registry entry to force TestDriven .NET to prefer this TestRunner for MbUnit-based test assemblies over the one that ships with MbUnit. It should look something like this:
CODE:
-
[HKEY_CURRENT_USER\Software\MutantDesign\TestDriven.NET\TestRunners\OxyMbUnit]
-
@="5"
-
"AssemblyPath"="C:\\Program Files\\OxyMbUnitTestRunner\\Oxygen.MbUnit.AddIn.dll"
-
"TypeName"="Oxygen.MbUnit.AddIn.OxyMbUnitTestRunner"
-
"TargetFrameworkAssemblyName"="MbUnit.Framework"
-
"Application"="C:\\Program Files\\MbUnit\\MbUnit.GUI.exe"
The @="5" ensures that this TestRunner will have priority over the standard MbUnitTestRunner, which has a value of 10. After setting up this key manually and moving the built assembly to C:\Program Files\OxyMbUnitTestRunner, I fired up Visual Studio and verified that the TestRunner was working as expected. Cool!
Realizing that manual registry and Program Files directory installs are a drag, my colleague Sebastian whipped up a WiX file to build an installer for our TestRunner. Here's the source to the WiX file:
XML:
-
<?xml version='1.0'?>
-
<Wix xmlns='http://schemas.microsoft.com/wix/2003/01/wi'>
-
<Product Id='????????-????-????-????-????????????' Name='TestDriven .NET plugin' Language='1033' Version='0.0.0.0' Manufacturer='MOxygen Media'>
-
<Package Id='????????-????-????-????-????????????' Description='Installs a TestDriven .NET TestRunner DLL and the required registry key.' InstallerVersion='200' Compressed='yes' />
-
-
<Media Id='1' Cabinet='product.cab' EmbedCab='yes' />
-
-
<Directory Id='TARGETDIR' Name='SourceDir'>
-
<Directory Id='ProgramFilesFolder' Name='PFiles'>
-
<Directory Id='OxygenMediaFolder' Name='OxygenM' LongName='Oxygen Media'>
-
<Directory Id='OxygenProductFolder' Name='OxyMbUn' LongName='OxyMbUnitTestRunner'>
-
<Component Id='WixFileComponent' Guid='A36634EC-5D03-47ab-96E7-568896D4B2DD' DiskId='1'>
-
<File Id='WixExampleFile1' Name='MbUnit1.dll' LongName='MbUnit.AddIn.dll' src='MbUnit.AddIn.dll' />
-
<File Id='WixExampleFile3' Name='MbUnit3.dll' LongName='MbUnit.Framework.dll' src='MbUnit.Framework.dll' />
-
<File Id='WixExampleFile5' Name='MbUnit5.dll' LongName='Oxygen.MbUnit.AddIn.dll' src='Oxygen.MbUnit.AddIn.dll' />
-
<File Id='WixExampleFile7' Name='MbUnit7.dll' LongName='TestDriven.Framework.dll' src='TestDriven.Framework.dll' />
-
</Component>
-
</Directory>
-
</Directory>
-
</Directory>
-
<Component Id='OxygenMBUnitRegistryKeyComponent' Guid='A36634EC-5D03-47ab-96E7-568896D4B2DC'>
-
<Registry Id='WixExampleProductRegistryKey' Root='HKCU' Key='SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\OxyMbUnit' Type='string' Value='5' Action='write' />
-
<Registry Id='WixExampleProductRegistryKey1' Root='HKCU' Key='SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\OxyMbUnit' Name='AssemblyPath' Type='string' Value='C:\\Program Files\\OxyMbUnitTestRunner\\Oxygen.MbUnit.AddIn.dll' Action='write' />
-
<Registry Id='WixExampleProductRegistryKey2' Root='HKCU' Key='SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\OxyMbUnit' Name='TypeName' Type='string' Value='Oxygen.MbUnit.AddIn.OxyMbUnitTestRunner' Action='write' />
-
<Registry Id='WixExampleProductRegistryKey3' Root='HKCU' Key='SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\OxyMbUnit' Name='TargetFrameworkAssemblyName' Type='string' Value='MbUnit.Framework' Action='write' />
-
<Registry Id='WixExampleProductRegistryKey4' Root='HKCU' Key='SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\OxyMbUnit' Name='Application' Type='string' Value='C:\\Program Files\\MbUnit\\MbUnit.GUI.exe' Action='write' />
-
</Component>
-
</Directory>
-
-
<Feature Id='WixExampleProductKeyFeature' Title='Oxygen MBUnit Addin' Level='1'>
-
<ComponentRef Id='OxygenMBUnitRegistryKeyComponent' />
-
<ComponentRef Id='WixFileComponent' />
-
</Feature>
-
</Product>
-
</Wix>
And there you have it. Happy TDD days are here again!