Peter Seale's weblog

Tuesday, May 27, 2008 8:00:52 AM UTC 

Update 2009-03-05 - I've changed a bit

I've been meaning to update this post for a while, and here we are. Others have surfaced with interest

• Francis Cheung [MSFT] has written a similar piece outlining the challenges of unit testing SharePoint. He did a better job organizing his arguments, so go read his article instead of this one :)
• The SharePoint Guidance sample project was released—it's the closest thing we have to the "ideal* way to build SP apps*," and it exists, and you can download it today. And they have unit tests, and you can look at them. Awesome!
• Andrew Woodward has started a series about TDD in SharePoint, I'm interested to see this develop further with more focus on how to make it work in typical* SharePoint scenarios.
• Tom Clarkson posted an interesting article about testing his workflows. I stay away from Workflow development as a kind of metaphorical water hazard, but it's good to see testing Workflows is possible.
• Eric Shupps has posted a long contrarian piece on unit testing SharePoint. Be sure to read the comments for the counter-arguments, which boiled down to "your sample is too small to unit test anyway," AKA it's all about context. Perhaps the most interesting thing is that BIl Simser replied in the comments (comment #3) about how he structures his SharePoint projects, something I'd be interested in hearing more about.

*footnote: defining "ideal" and "app" and "typical scenario" is an exercise left for the reader, or the commenters on Eric Shupps' post.

Meanwhile, I'm spending my time learning coding/OO fundamentals, which is an important step to doing real TDD. Practicing TDD at home has already changed the way I structure my SharePoint projects, even the small ones, even when I'm not doing TDD at work.

I still don't unit test my SharePoint projects, but on my next project I'll probably try to do full-blown TDD. I still won't attempt to test code that interacts with SharePoint, but at least I can structure my code such that I can write down, and test, my intent. In other words, TDD can't tell me whether the listItem.Update() function is the correct way to save changes to a listItem, but it can tell me whether or not I called the function that fully intends to do so.

If you're coming here from the JOPX post, note that I do want to do TDD, and I think TDD has great value outside of SharePoint projects, but. I hold firm that most of the challenges we encounter with SharePoint are either the fault of the framework, or the fault of our misunderstanding of the framework—and neither is addressed by TDD.

I also want to clarify that as a rule, I don't think SharePoint should be used to host massive applications. If you've ever seen Wayne's World, there's a classic bit where his girlfriend gives him a present:

"A gun rack… a gun rack. Shyeah, Right! I don't even own "a" gun, let alone many guns that would necessitate an entire rack. What am I gonna do… with a gun rack?"

In this case, TDD on my tiny SharePoint projects is the metaphorical gun rack. Also big props to myself for making that connection, I deserve an award for that.

Opinion piece

This is my opinion, and reflects my (lack of) experiences, personal bent, and traumatic experiences, collected together into something coherent. I am writing this 2008-05-19, and reserve the right to change my mind later. I want to change. Like Fox Mulder, I WANT TO BELIEVE.

Authority

I have minimal authority on this subject and do not believe my opinion is "better". I am not an expert on unit testing, and I am not a renowned SharePoint authority. I'm going to attempt to steer clear of making any statements about TDD, and will instead focus on the more concrete act of unit testing.

I love comments

Actually I wouldn't know if I love comments, because I've only got like 3 of them. I was pretty pumped when I got those 3 comments though; that was a good year.

I love explicit disagreement

If you read this description of the forthcoming SharePoint Best Practices book, you'll see what I'm attempting to do: I'm attempting to start the conversation (hopefully not finish the conversation) that will answer the question: how are we supposed to adequately test our SharePoint projects? So far I have only found two strong opinions. I'd like to collect more.

Also, this is not a plug for the best practices book; I haven't read it.

Disambiguation: unit testing versus automated testing

One common misconception I'd like to clear up is that unit test probably doesn't mean what you think it does. Whenever someone tells you they are doing unit tests, allow me to present the following handy fact sheet:

• Just because something is written in NUnit or MbUnit or xUnit.Net, does not mean it is a unit test. I think this is the #1 source of confusion—the name of the product has the word "unit" in it, and it's used for testing, so…unit plus testing, unit testing! That's what they're called! Hilariously, by this metric MSTest is the best-named automated testing suite. You heard it here first: MSTest is the best!
• TFS Web tests are not unit tests. Tests that involve WATIN or some other web automation tool are not unit tests.
• Anytime you're actually newing up an SPSite or SPWeb object inside your test method, it's not a unit test.

Now for a few facts:

• Unit testing is often confused for the broader world of automated tests, which includes both unit tests and integration tests.
• Unit testing is not easy in SharePoint.

At this point I'm going to be lazy and tell you to read up on the Wikipedia articles on Unit tests and Integration tests I just found by googling for 5 seconds. I'm lazy, ask someone else for a better explanation.

Opinions

Popular culture: no testing

When dealing with such a broad topic as SharePoint testing strategies, it's probably best to start with the default. Which is to say, nothing at all. By default, the out-of-the-box Visual Studio 2005 projects do not create a separate testing project for you to use for testing. Most developers will live with the defaults, which is to say again: nothing.

So if you're here, wondering what to do outside of the defaults, congratulations! You're already counter-culture.

Counter-culture: agreement on some testing, disagreement on specifics

(various): Integration testing is sufficient

Spence Harbar, MVP: Unit tests with TypeMock - this is the only page I can find from him on the subject.

Casey Charlton: Unit tests against Doubler-generated wrappers - Casey has written repeatedly about the pains of his unit testing approach, but the fact of the matter is, he now has a workable, no-TypeMock-required solution for unit testing in SharePoint. Check out the Doubler addin to Reflector, which is described as "useful when working with legacy code."

My Opinion

With the problems I encounter on a day-to-day basis, I don't think it's worth it to go out of my way to build unit tests. And the reason is as such: when I encounter a bug in my SharePoint application, it has nothing to do with my code, and everything to do with my interactions with the SharePoint framework. Let's do this by example:

• Spence Harbar on his TDD page mentioned specifically the awful quality of code in the navigation controls he's seen in the wild. His assertion is that the code quality is the biggest pain point. Unfortunately, when I'm attempting to build something, I'm worried less about the cyclomatic complexity or the C.R.A.P. metric (it's real, believe it), and worried more about which of the many portal-querying strategies to use. There's no way to know if you're using the right approach, unless you're read all the books, all the whitepapers, and all the blogs. And, yeah, don't look at actual control vendors' code via Reflector, because apparently they're doing it wrong too :) The major issue when building navigation controls is not traditional code quality, as much as a broad understanding of all the options presented to you by SharePoint's object model.
• In my most recent project, I encountered a race condition where the ItemAdded() event fired too early and my list item…did not exist yet. This was nothing I could control. Solution? What I affectionately refer to as the "try try again" pattern. It involves a while loop with a retry counter, a Thread.Sleep(), and a whole lot of swallowed pride. Feel free to complain about the solution, but don't complain to me, complain to Steven Van de Craen; I'm just thankful his solution works. This concurrency bug was out of my control and would not have been discovered (or fixed) by unit tests.
• Outstanding issue: I am intermittently faililng to update my item after the ItemAdded() event. There are several possible solutions; the two I will be looking into will be a) disabling event firing (shouldn't have been a problem); b) following this awesomely-detailed advice and replacing my ".Update();" call with a ".SystemUpdate(false);" call. Either way, proper testing would not have saved me from this pain; detailed (esoteric) knowledge of the framework is the only thing that could have saved me.

What I'm saying is that practicing TDD in my SharePoint projects would not help me improve my code enough to be worthwhile. In the meantime, I will use automated tests (mostly integration tests).

While I'm not especially pleased with it, I am resigned to avoid unit testing the difficult pieces of my SharePoint projects. I'll consider changing this stance in any of the following situations:

• I become a master ninja and TDD becomes more effortless to me than doing without;
• I'm working on a critical, high-impact, risky project and need any and all available methods of testing and program verification;
• I have a lot of code in my project (which begs the question: why?);
• I learn enough about SharePoint development that situations like the three above are no longer issues, i.e. I learn how to walk around the coding landmines and am confident with every interaction with the framework. (For now: BOOM)

Don't forget about your team

Another factor to consider is maintenance.  Remembering that by default no one builds automated tests, with some effort counter-culturists write integration tests, and only two people in the world practice TDD in SharePoint—it's a hard sell to say "hey, I want to try building this next SharePoint project into a structure that is so awful and alien, so overwhelming, that you will be paralyzed. I'm going to change the project into a gorgon, which, with a single gaze, will turn you to stone where you sit. You will be completely unable to function. Do you mind? This new method will allow me to capture up to 60% of the bugs. But probably a lot less than 60%, to be honest. TIA!"