Peter Seale's weblog

Tuesday, August 09, 2011 5:40:45 PM UTC 

I’m unmotivated today at work, partly because I’m switching us from MSTest to NUnit. I’ll be happy again once it’s done, but not until then.

With that in mind, I’m ready to give the second half of my “using TFS as a CI server” advice, borne out of my experience on a real team project running TFS as our CI server.

This one’s going to be less positive than my Using TFS as your CI Server part one, and if you’re not in the mood to read, I’ll just summarize:

• Don’t use MSTest as your unit testing framework, and
• If forced to use TFS 2010 as your CI server, minimize your exposure to the XAML build script, instead delegating your entire build script to PowerShell or MSBuild or whatever else tickles your fancy. Don’t use TFS 2010 Build XAML, it isn’t worth the effort to set up a real build entirely written in Workflow Activities. It’s probably possible, but not worth the effort.

Switch to NUnit: The MSTest test runner is non-deterministic and will do great harm to your CI experience

We’ve had serious problems getting consistent results out of our MSTest test runs for our two projects. Turning off various features (such as code coverage) has helped some, but not enough. It’s worth your effort to  switch to NUnit if you’re serious about doing unit testing. Sorry MSTest, I tried, but the test runner fails way too often.

For nitpickers, you don’t have to switch to NUnit. You could switch to anything.

Switch to NUnit: MSTest leaks memory and cannot support our test runs

This isn’t as important as the failing test run. It is important if it ever happens to you and you have to rework your test runs such that you don’t run out of memory any more. I’ve searched and we’re not the only people running into this problem.

I hate Windows Workflow Foundation, and Windows Workflow Foundation hates me

Ayende ruined me with his JFHCI series of blog posts (I blogged about the topic here). After being enlightened to the fact that code (or if you prefer, script) is better in every way* over XML configuration, I’m ruined on ever using Workflow Foundation for anything. Ever.
*exaggeration

With that in mind, I’m not a fan of the reworked TFS 2010 XAML build system. However, this post is only the positive takeaways, so I shouldn’t get carried away talking about the build system, and instead talk about what you should do when told to set up your build in something called “xaml”.

TFS 2010 Build XAML is a V1 Microsoft Product

Some of you are not going to like this, but: avoid the XAML.

Avoid the XAML build system. It takes a long time to test build scripts, it is painful and the designer is buggy, it has almost no built-in Workflow Activities (e.g. there is no a “copy file” activity), it is harder to follow, harder to modify, painful to use with multiple branches sharing the same build XAML. PowerShell’s REPL shortens the feedback loop to something like 10 seconds, and MSBuild and NAnt can be configured such that you get feedback within a few seconds as well. TFS Build’s feedback loop is something like 10+ minutes, depending on how long your entire build takes.

To be clear, the TFS Build feedback loop is as follows:

• Save, wait 10+ seconds for the save operation to complete.
• Navigate to the Source Control WIndow, check in the XAML file in the BuildProcessTemplates folder.
• Navigate to the Team Explorer and kick off a build manually.
• Wait until the build completes.
• Open the build summary for the build you completed.

Takeaway: minimize your XAML exposure

My preferred method of avoiding the XAML build system is to call out to PowerShell immediately for your entire build script. I’m serious—don’t even try to build your entire build script in the XAML designer.

This blog post explains how to call PowerShell from TFS. I’m not giving you the full solution, because working with TFS build is demotivating and I don’t want to spend any more time than is necessary here, but I’ll link to a partial solution.

Here’s a rough idea of what to do:

• Find a XAML build script for your starting point, delete almost all of it, and add one InvokeProcess activity that calls out to PowerShell.
• Make sure to pass in necessary arguments like SourcesDirectory, BinariesDirectory, etc.
• Put all your compiling, test running, ClickOnce manifest building, packaging, deploying to Dev environment-ing, XML configuration modifications…put all these things in the PowerShell script.
• Investigate psake if you’re serious about doing your build in PowerShell.
• If you’re not a PowerShell fan, by all means call out to MSBuild or NAnt using the InvokeProcess activity. Whatever you do, just don’t try and wrangle with the TFS 2010 build XAML.

It’s worth the extra effort to get the call-out mechanism working, even if it seems like “this is taking longer than it should.”

Use Arguments for your TFS 2010 Workflow Builds

The one thing I like about the TFS 2010 build system is the concept of workflow arguments, wherein you can change settings “at runtime,” or specifically when queueing up a build. This is particularly good for us if we want to temporarily turn off tests or run a “deploy” build from TFS with certain parameters provided only at runtime. In TeamCity there were a few freetext text boxes that allowed you to type whatever arguments you wanted, but there was no guidance per-se. Nothing to tell you “Our build script is looking for precisely three things: a) the NUnit tools directory (though I’ve provided a default); b) whether or not you want to deploy to the Dev environment; c) whether to run tests.” The TFS 2010 Workflow does exactly this in an extensible way. Nice.

You can set up your “call out to PowerShell/MSBuild/NAnt/whatever” activity to pass any of these runtime-provided arguments as you need them.

My framework/platform strategy

I have a few basic strategies for using frameworks or platforms or basically anything to do with computers:

• If it works, learn to use it well. For example, Windows 7’s new features/hotkeys/start menu search, Google Reader hotkeys, C# syntax, ReSharper, the commercial ORM we’re using. I’ll generally spend the time it takes to a) learn the product, and b) use it as intended.
• If it doesn’t work, avoid it. For example, Windows Vista’s start menu search—I turned it off completely. The MSTest test runner falls in this category. I am also not a fan of most of the more advanced WPF language features, and don’t use them.

I also react very differently to frameworks I trust and those I don’t trust (i.e. those that “work” and those that “don’t work”):

• If I’m experiencing a problem with a framework I trust, I’ll read up and try to find the correct solution because I’ll assume I’m at fault. Today this means, if I see NUnit’s test runner throw an OutOfMemoryException, I’ll blame us first.
• If I’m experiencing a problem with a framework I don’t trust, I’ll write the dirtiest, quickest workaround available because assume the framework at fault. I learned this lesson the hard way while working on a “quick” SP Workflow project a few years ago. Today this means, if I see MSTest’s test runner throw an OutOfMemoryException, I’ll blame MSTest and switch us to NUnit.

Something I don’t think I’m saying outright is, these labels of “it works” or “it doesn’t work” affect how I deal with everything I do with software. With TFS as a source control solution, I’m dealing with it as

1. A product that works great for SVN-style source control. Edit, merge, commit. Works great. Merge even works as of TFS 2010. Try to figure out why you’re having problems.
2. A product that does not work offline or remotely. Don’t try offline mode, period, and avoid doing heavy TFS work (e.g. moving directories of files around) remotely. Avoid or work around the problem, in other words.
3. A product that branches, painfully. If you experience problems with branching, work around the problem, potentially by losing source control history. I’m okay with losing file history. A lot of people are not okay with that. Branch less, because it’s less painful than dealing with the problems of having too few branches (and boy howdy do we ever need more branches).

With MSTest, I deal with it as

1. A unit test syntax and local test runner that works great (if slow). Learn how to use it properly.
2. An inconsistent CI test runner. Avoid it if possible.

With TFS Build, I deal with it as

1. A bad language/environment for writing build scripts. Avoid it/escape the XAML as soon as possible.
2. A reasonably consistent CI server that is painful to navigate. Learn to use it, and make the conscious choice to lose 5-10 minutes every day to navigating TFS menus, and to allow for confusion given the TFS tray app doesn’t work well and most of the build status UI is confusing and inconsistent. Once you’re consciously okay with losing some time navigating through the menus and closing+reloading build status windows, you stop caring about those 5-10 minutes. It works. If you can’t stop caring about everything, you’ll eventually go crazy. Right?

Did you see the pattern there? I have an internal list in my head of which features I can trust and which ones I can’t trust. This list keeps me sane.

Do others maintain their own internal “I can trust this software” list, or am I just crazy?