Wednesday, October 28, 2009 7:01:27 AM UTC 

I think now's a good time to close out the SharePoint tag on this blog, marking the end of SharePoint 2007-focused content. I'm creating this post as a sort of table of contents for my SharePoint content. I'll attempt to group posts into themes and then editorialize. Some of my earlier posts I'll even recant! Here goes.

Things I think anyone (SharePoint community or otherwise) would find interesting or useful

• Thinking Creatively - what I consider possibly my best SharePoint-related post, because it contains transferable concepts. The idea is that we as developers must go beyond our traditional code monkey role and do some critical thinking while specifying/designing solutions to problems. This is illustrated with an excellent story told during an Agile conference session. Also I recommend the linked Agile Toolkit podcast episode that inspired my post.
• The SharePoint-Python connection - if anyone ever tells you "SharePoint was written in Python", I apologize. Anyway, misquoting aside, this is a fun little bit of history.
• Golden Rule of Troubleshooting - I misdiagnosed an error, posted the erroneous diagnosis to my blog, and to save face hurriedly changed the contents of this post to be about troubleshooting. The golden rule of troubleshooting, for those of you too lazy to click through: beware the invisible proxy! It takes many forms! It strikes silently! Everyone will think you're crazy when you tell them network gremlins are eating your incoming packets, but leaving your outgoing packets alone!

Hilarity

• Welcome to SharePoint - a real-life nightmare scenario I encountered while troubleshooting a SharePoint 2003 "desktop issue." It turns out, the 15 pages of IE settings + Active Directory group policy + various Office ActiveX controls + virus scanners + IE version mix + network security appliances + Kerberos + firewalls + IE Zone settings + DNS/DHCP issues + AD replication issues + expired password issues + routing errors + spammy IE toolbars…any and all of these things, if out of whack, show the same "username/password" dialog. The post was a joke, but after troubleshooting every flavor of this problem, it gets to you a little. Anyway, welcome!
• What's wrong with this picture? - mildly amusing scenario involving disaster recovery documentation. Trust me, this is as hilarious as disaster recovery documentation is going to get.
• STSADM: Spot the typo - a lament for the endash. This is as much hilarity as you'll find on the topic of Word AutoCorrect.

Frustration

• Angry at CAML - I remember writing this post after a three days of wrestling with GetListItems, most of which was wasted learning idiosyncrasies. I then deleted most of the unhelpful angry comments, so what remains is the milder parts. This was my first "surprised by how difficult it is compared to how easy everyone makes it sound" experience. Visit for a link to the greatest Oracle DBA ever. Or visit for my graphical representation of the MSDN Rage Meter.
• I'm like, angry at numbers - in burnout mode, and ranting. If there's anything to take away from this, it's a) keep a sense of perspective (i.e. this stuff isn't important), and b) don't invest time in new Microsoft frameworks as a rule.
• Disposing SharePoint Objects: Survival Mode - this was a tipping point wherein I realized that no one does SharePoint development properly, even most of the MVPs. Keith eventually discovered and wrote down all of the disposal rules, and from there someone at Microsoft released SPDisposeCheck (which I believe covers most scenarios). Anyway, the subject of disposing is now a moot point—the more interesting bit is that, as of two full years after RTM, we had incorrect guidance as to how to dispose ~2MB objects on web servers typically running a maximum ~1000MB in the worker process. Sort of an eye-opener.
• Beyond technical challenges - rant, wherein I close the SharePoint 2007 portion of my blog (oops—the ban lasted a full month anyway, until I couldn't hold out). There are some takeaways here, notably that everyone's struggling with SharePoint, including the MVPs and "experts." I make the statement that every person working with SharePoint should look beyond their immediate technical challenge and ask, is SharePoint the right solution? Also, I challenge the assumption that SharePoint is a good developer platform.

SharePoint as an app dev platform

• [also referenced in the "useful" section above] Thinking Creatively - what I consider possibly my best SharePoint-related post, because it contains transferable concepts. The idea is that we as developers must go beyond our traditional code monkey role and do some critical thinking while specifying/designing solutions to problems. This is illustrated with an excellent story told during an Agile conference session. Also I recommend the linked Agile Toolkit podcast episode that inspired my post.
• Argue with your customer - I think I posted this after failing to convince my customer to go with less SharePoint customization and more out-of-the-box features. I still get a lot of pushback when I try to prevent SharePoint customizations. If there's something to take away from this, especially as a non-SharePoint developer, it's that not all features cost the same, and not all customizations cost the same. Making the relative costs clear to your customer should (should!) encourage them to avoid the more costly customizations. I'm always shocked at how someone will tell you "no, we need it to work exactly as I've told you!" and then turn around settle for a vendor product that does about half of what they want, but costs more.
• 80%, then stop - wherein I tell a story about my experience with the Pareto principle as it applies to SharePoint solutions. Also: you don't have to write your apps in SharePoint! If it doesn't save time, and you don't know of any benefit you'll gain from using SharePoint, then why are you attempting to use it?
• Estimating SharePoint tasks: cry for help - scary realization that I'm still unable to estimate how long something's going to take, primarily because I'm constantly blazing new (new to me?) trails in the SharePoint API, making bad assumptions about its behavior, triggering bugs, and running into unexpected limitations. Any of these things can cause multi-day delays. It does get better if you're writing a second or third app that deals with the same part of the API.

Framework limitations or errors

• How many is too many [SharePoint list] items? - the SharePoint whitepaper announcing the 2000 (now 3000) item limit per container was something of a blow. To say it clearly: this limitation prevents you from using OOB lists for anything with real traffic. There are over 3000 days in 10 years, so at 1 item added a day, you're running into the recommended limit. Since then I've seen some crazy errors with large lists, mostly revolving around OutOfMemory errors, crawl errors, using PRIME-derived features on the lists, exporting to Excel, breaking the Grid-view, and so on. So the list size limitation is real, if not a "hard limit."
• SharePoint Workflow Nuttiness, Volume 1 - My initial foray into SharePoint Workflow development ended in pain, where I had to scrap an entire approach because Workflow doesn't support state machines with replicator activites. Then I read Ayende's JFHCI series and it poisoned me forever against WF. I wonder now what problem Workflow attempts to solve, and why don't we just use a pure code solution instead? Note that Ayende wrote a book on DSLs (Workflow is a form of a DSL), so don't just pretend he's some crank with a blog*. Final note: this is one of my top 3 most visited posts, so apparently lots of people have run into the specific issue.
  *I'm aware that by definition, I'm a crank with a blog
• Illegitimate ErrorWebParts - a crazy solution to a crazy problem—here I use the "crank the chainsaw a few times" metaphor to describe loading a SPLimitedWebPartManager. Really, this is bad.
• Dingoes stole my babies! - wherein I discuss a problem with moving wiki content via the PRIME API.
• SharePoint awesomeness: User Profiles - wherein I discuss a potential benefit of functioning User Profiles. Unfortunately this post was premature, because the scenario I envisioned/laid out in the post wasn't possible out-of-the-box. Oops! Another framework limitation.
• SharePointPdfIcon project - wherein I announce my (failed) CodePlex project. It works great for single-server farms, incidentally. I just can't be bothered to spend the time to write all the timer job junk to make it work on multi-server farms when even this souped-up solution won't work when someone adds a new server to the farm after activating the Feature…this is one of those cases where using the SharePoint deployment framework causes more pain than deploying changes the "vanilla" way. Ugh.

PowerShell + SharePoint

• Find the DLLs - After determining that Lutz' (now RedGate) Reflector is a core tool for SharePoint development, the next step was acquiring the DLLs from wherever they lay. Enter gratuitous use of PowerShell to solve the problem.
• PowerShell is Magic: Part 1 - wherein I demonstrate PowerShell calling STSADM but also calculating on the fly. PowerShell is really, really useful.
• PowerShell is Magic: Part 2 - wherein I describe (poorly) how the PowerShell REPL is powerful.
• SP + PS: Working with wikis - wherein I give a pretty weak (but like the movies say, based on true events) example of how I use PowerShell to solve problems.
• Why PowerShell: Readability - wherein I take another shot at explaining how PowerShell launches processes (console apps)…and get the explanation wrong. I should probably post an update or something. Also, PowerShell can be made to be readable (though like Perl can be made to be abomination).
• PS + SP: Cornucopia - wherein I list all the real-world uses I've found for PowerShell working with SharePoint. PowerShell is uniquely useful for SharePoint, because SharePoint has a) an incomplete admin UI, b) huge object model that's loaded into the GAC, c) incomplete MSDN documentation, necessitating experimentation, and d) so much XML! Probably other reasons, but those are the big ones. Also, Visual Studio post-build tasks are the devil. I'm now ashamed of me of a year ago. Shame on you, 2008-me.
• Useful PowerShell functions I've written. I've looked at others' PowerShell functions and I think it's a lot simpler to do away with logging, comments, object disposal, and attempts to improve performance. All things are appropriate in context, but for me, these are mostly throwaway ad-hoc scripts, and are thus simple and focused.
• Write-ListDetails - particularly, discovering (and recording useful information about) large lists—and remember, this is PowerShell, so you can pull ANY data you want, no matter how complex the criteria or where the data originates.
• Run-Query - think of this as a REPL for your SharePoint Enterprise Search SQL queries. Returns pretty objects, not a DataTable.
• Get-CrawlHealth - I used this to prototype the functionality I wanted, then built it into a _layouts page. The script works though (with the exception of the $contentSource.CrawlCompleted property, which is inaccurate and worthless)
• Update-SearchScopes - on demand! You can't do this via the UI.
• Get-UserProfile and Get-UserProfileData - the first function retrieves the UserProfile object, the second function maps the (nigh-impenetrable) property collection to real properties. Useful for bulk data export and for examining your user profile data in a meaningful way.

Informational (knowledge, not concepts)

• Briefest introduction to GetListItems using CAML and lists.asmx - by now there are much better (and more accurate) guides to GetListItems. What may be amusing to you is the comments I leave on each line of code—wherein I document how uncertain I am of what each element does. The MSDN documentation has improved since 2007. As a small bonus, I'll note that this runs against WSSv2, not SharePoint 2007.
• Don't delete the default app pool - nitty-gritty details on IIS configuration. Note to anyone who has rolled out a SharePoint farm: congratulations, you're now qualified to roll out any ASP.NET app! Personally I'm pumped this knowledge transfers.
• Firefox supports auto-NTLM logins - most of you aren't aware that you can use Firefox and visit your SharePoint sites, and not be so aggravated by login boxes—Firefox supports automatic NTLM authentication in a manner similar to IE! Follow the directions to enable it.
• Cisco NLB setup in SharePoint - because I'm still the only resource for this in the entire world. Ridiculous.
• SharePoint search - faster than you think - wherein I complain about how slow IE is and how it is to blame for many of SharePoint's "performance issues." Honestly, it's true—try loading SharePoint pages in Firefox, they're way faster. Also it helps if the page doesn't load 1MB of JavaScript and another 1MB of inline style text.
• SharePoint Timer Jobs - here I attempt to shift from unhelpful ranting, to a post designed to help others avoid pain. I'm happy to say this is one of the top 3 posts, and hopefully it's helping people. Specifically, I mention that Timer Job updates require the manual reset of each Timer service on each server, and provide a script to quickly reschedule a timer job. Small footnote: I would rewrite the PowerShell script today such that it was a single function that takes arguments instead of requiring customization of the script. Functions are self-contained, and can easily be pasted into a PowerShell window (e.g. a PS window running remotely on a server!) without accidentally executing anything.
• Project retrospective on my People Search project - raw stream of consciousness, in bullet-point form. I didn't want to spend much time prettying it up, but reading the list of limitations, recommended customizations and preferred AD setup can save you weeks (and pain!) on your People Search project.

Cruft

• My SharePoint search page - to be clear, this is a static HTML page I made with search boxes to to search Google, search USENET, search the Technet forums, and search Google Reader. It's mostly broken now, and eventually I'll take it down. I used it A LOT while doing farm architecture-y type work, and used it heavily when troubleshooting in the early days. Now that I'm more development-focused, I've found I don't use it. Ever. Takeaway for everybody: Technet forums search covers more than Google does. If you're desperate enough, search both Google and the Technet forums (called MSDN Social now?).
• SharePoint search page - hottest of the hot! - wherein I add hotkey support to my (now-defunct) search page.

Op-ed (opinion pieces with almost no useful, actionable content—sorry)

• Dear MSFT, please talk to your Office division - op-ed, Sorry. Summary is, please don't obfuscate all your DLLs. Side note: InfoPath is pain.
• One Language A Year - wherein I dedicate a year to learn C#—that is, actually learn C#. I'll dig into Scala/Clojure/Haskell/Ruby/Python/Lisp/Scheme/Erlang/JavaScript/Io/Factor/OMeta/Smalltalk some other day. Also, I outright deny the claim that you should learn one language a year. It's cheap to give advice. It's not as cheap to follow advice. I have a new rule on following advice: does the person giving the advice actually do what they say? I got similarly disgruntled when "Uncle Bob" said something to the effect that you should dedicate 40 hours to work and 20 hours to learning. That's just crazy talk.
• SharePoint wikis are awesome, I swear - another of my top 3 visited SharePoint pages. I now apologize for defending SharePoint 2007 wikis. Afternote: I wish this wasn't such a popular page. Of all things, a wiki op-ed piece is one of my top pages, ugh.
• SharePoint: not unit testing - I've waffled a bit on this one. My current stance is that I'd really like to do continuous, automated functional testing (i.e. drive a browser window with code) to give me confidence my SharePoint solution actually works. True unit testing wouldn't cover enough space to give me confidence in my project, and most of my SharePoint projects are tiny, such that the "designing your API via design-by-example TDD" argument for TDD doesn't apply. Also, read this post for a short anecdotal survey on what kind of problems I run into when developing SharePoint solutions.
• Say no to makecab.exe - Here I rant against using makecab. I think I had just read yet another MSDN article that made casual use of MAKECAB.EXE and pretended like it was a good idea. Also I apparently just read the CodingHorror post on "Strong opinions, loosely held" which I now think is a terrible formula for my blog posts. At least I include a somewhat-useful PowerShell snippet that bypasses makecab, that's something.
• Surviving your first SharePoint project: Part 1 - wherein I sloppily argue that WSPBuilder is superior to STSDEV, VSeWSS, and makecab. It's true though, and somebody's got to counteract all these MSDN articles and books that pretend WSPBuilder doesn't exist…
• Does this describe you? - short, unhelpful post that quotes Niklaus Virth and laments SharePoint's accidental complexity.

SharePoint 2010/SharePoint 14 predictions

• SharePoint 14: Everything we know - it turns out from what I've heard from SPC09, this was dead-on accurate. They kept PowerPivot silent through the NDA period. Interestingly, SPC09 was silent on "Bulldog", the MDM product Microsoft purchased. Also I apparently missed out on the TownSquare bits, which they publicly discussed, and which evolved into the Facebook-like features.
• Preparing yourself for SharePoint 14 - I'm proud of my track record here, because I nailed pretty much everything. Written a full year ago.

"Other" category

• Yet another SharePoint VM: RIP - there was a period of time where I was Doing Something Wrong with my VMs. I now blame either/any of: a) saving state/restoring from saved state in Virtual Server and Virtual PC, b) running my external USB hard drive off of laptop battery power, c) lots of plugging and unplugging of said USB hard drive. I haven't had a problem in a long while now. Takeaway: back up your VM every so often "offsite", just in case.
• ASP.NET MVC is a MAGIC FLYING CARPET - wow, it's been two full years since the announcement! Anyway, here I mention how SharePoint development feels like alchemy sometimes, and separately, how the SharePoint developer community doesn't seem to value the things I like about ASP.NET MVC. Posting this had the side effect of sending lots of poor souls to my blog from google searching on "how to create an ASP.NET MVC app inside SharePoint."
• SUGDC Conference 2008: Recap - wherein I give a similarly-huge recap of each session I attended. Also: layoffs drive big SharePoint adoption! So, get with the layoffs!
• SharePoint + ASP.NET MVC - wherein I troll for people searching for these keywords.

Thursday, December 18, 2008 9:48:39 PM UTC 

It's all business again.

First: I'm going to be away from (hopefully all) computers for a while

I'll be on vacation, in a very real and non-metaphorical sense. Which is awesome.

What it means for you is, if on the offhand chance you leave a comment or email me, I won't respond. Sorry, I'll be away. Sucks for you, awesome for me.

Second: ASP.NET MVC and SharePoint, together as one

It still boggles the mind how many people are searching for this term. The only thing I want to ask you, the many people who are searching for "how do I get ASP.NET MVC running underneath SharePoint," is: why? Why do you want to do this?

I'm not here to provide any answers today; instead, I'm posting this as a kind of googlebait to lure you in. Maybe it's wrong, but whatever. Why do you want to use the still-beta ASP.NET MVC framework on top of SharePoint? I honestly don't know why anyone would do this. So please, if you search for "how do I combine SharePoint with ASP.NET MVC," and you hit this page, leave me a comment! I want to get in your brain and swim around a little.

I'm pretty sure I can get ASP.NET MVC running underneath SharePoint; the only magic will be removing pieces of SharePoint from the MVC project's web.config, and (maybe) integrating with SharePoint's security (or maybe not). Besides ugprading to .NET 3.5 SP1, which is by far the most arduous step on a production farm, it shouldn't be too tricky to get this working.

Anyway, there's a somewhat rambling teaser—it's probably possible, even if I can't imagine why it would be a good idea.

Tuesday, October 28, 2008 11:46:47 PM UTC 
I couldn't hold out.

It's okay though, because today we're strictly business. In the course of developing a bunch of SharePoint timer jobs recently, I've learned several things, most of which aren't obvious from the get-go:

1. Storing and retrieving configuration data is a problem. Because I don't have a farm-wide configuration list (yet—the temptation grows every day), I was forced to do some ugliness in order to store and retrieve configuration data. I don't necessarily recommend my approach; instead I'll just say I'm using a custom SPPersistedObject as my Timer Job's config store and I'll further say that it works, roughly, though I'd now prefer a better way. Consider setting up a farm-wide config list, it's a relatively big time investment but is probably worth it. Other traditional config storage options (such as the web.config, or a site-local config list, or your site-local property bag) aren't accessible to Timer Jobs without some sort of…configuration's configuration…hmm, yes…without something extra pointing the way. Anyway, it's a problem.
   
2. ANYTIME YOU UPDATE YOUR TIMER JOB CLASS (or custom assembly), YOU MUST BUMP ALL SHAREPOINT TIMER SERVICES ON THE FARM! I learned this lesson the hard way. If you fail to bump the timer service, it will blissfully run the old copy of your timer job class; I don't know how or why it caches your assembly, but it does, and bumping the SharePoint timer service is the only way to clear the assembly cache(?) and force it to use your minty fresh assembly.
1. Side-note: can we report this as a bug in the Solution framework? Because this is a big enough gotcha that the Solution framework needs to include an option to -bumptimersvc …or something. Maybe a custom stsadm command (stsadm -o resetadmsvc) or maybe tack something onto the Solution deployment API and associated stsadm commands…we need something.
   
3. There are differences between the context of a test harness (i.e. something like an NUnit integration test running under the NUnit test runner) and a timer job running in the Timer service. This may sound obvious, but when you're troubleshooting something that "only breaks on the test farm," this little bit of trivia is important. If you need to troubleshoot your timer job as it runs on the timer service, specifically, this can get tricky.
   

Also, in the course of troubleshooting just such an issue (as outlined in #3) I've created a little script to speed up the code-compile-test loop; instead of scheduling a timer job for "0100 hours" and waiting until tomorrow to see the results, why not reschedule the timer job by your own self? And that's exactly what I did.

My script below will reschedule your timer job to run 10 seconds in the future (read: instantaneously). All you have to do to get this script working for you is customize four variables to match your own timer job, then follow the quick "usage instructions" at the bottom of the script. Below is a rundown of the four variables:

• $siteUrl - the site collection root URL. We need this to get a reference to the SPWebApplication that holds your Timer Job.
• $customAssemblyName - the partial name of your custom assembly. This is necessary because we're going to new up an instance of your timer job, and thus we'll need to first load the containing assembly.
  
• $jobName - this need only be a rough equivalent of your job name. I'm usually lazy and say something like "*custom profile job*" or the minimum necessary to identify my job from all the rest. Messy is good; we're running a one-off script, right? Or you can go ahead and type the perfect exact case-sensitive job name in there, that's fine too.
• $timerJobClassName - again, we need this because we're going to new up a rescheduled timer job.
  

Assumptions:

• Your original schedule doesn't matter, and may be destroyed. Because that's exactly what this script does, by the way—it destroys the original schedule and sets a "10 seconds from now" schedule. Incidentally, whatever it did before, your job now runs on a daily schedule :)
  
• You're only concerned about the timer job running on one web application's context. Because in truth that's all that mattered to me when I wrote this script, I didn't consider the possibility of multiple jobs.
• You're running a single-server farm (i.e. a developer VM). My script only stops the service on the local server.
  
• No one else cares if you bump the SPTimerV3 service, including any other timer jobs that may be running presently. Note in the script below, PowerShell has some cmdlets to work with Windows Services. I was totally unaware of them until I had to bump this service; neat.
  

While these assumptions sound scary, trust me—you won't care. On a single developer VM, you won't care about all these things. Even on a multi-server test farm, you won't care—because this script is going to save you hours of troubleshooting.

The PowerShell script is as follows:

$siteUrl = "http://dev"
$customAssemblyName = "Corp.SharePoint.Assembly"
$jobName = "*your job name*wildcards*work*"
$timerJobClassName = "Corp.SharePoint.Namespace.TimerJob"

[void][reflection.assembly]::LoadWithPartialName("Microsoft.SharePoint")
[void][reflection.assembly]::LoadwithPartialName("Microsoft.Office.Server")
[void][reflection.assembly]::LoadwithPartialName($customAssemblyName)

function Run-Init
{
    $global:s = [Microsoft.SharePoint.SPSite]$siteUrl
    $global:webApplication = $s.WebApplication
    $global:job = $webApplication.JobDefinitions | ? { $_.Name -like $jobName }
}

function Create-NewJob
{
    Stop-Service "SPTimerV3"
    Start-Service "SPTimerV3"
    $global:job.Delete()
    $global:job = new-object $timerJobClassName -arg $webApplication
    $sched = new-object Microsoft.SharePoint.SPDailySchedule
    $now = [datetime]::now.AddSeconds(10)
    $sched.BeginHour = $now.Hour
    $sched.EndHour = $now.Hour
    $sched.BeginMinute = $now.Minute
    $sched.EndMinute = $now.Minute
    $sched.beginsecond = $now.Second
    $sched.endsecond = $now.Second
    $global:job.Schedule = $sched
    $global:job.Update()
}

#Usage: paste this script directly into a PowerShell console; the quickest
#way is to right-mouse-button click. Then when you're ready,
#run the following commands (minus the # of course):
#
#Run-Init
#Create-NewJob
#
#Anytime you update your custom assembly "Corp.SharePoint.Assembly", you will need to
#DESTROY your open PowerShell console/session and create a new one. This is the cleanest way
#to unload your old custom assembly.

That's pretty much it. Change the variables to whatever you need, open a PowerShell console, right-click, then type "Run-Init; Create-NewJob". You're done! Step 3: Profit!

Tiny footnote: if you don't care about "context", this script also allows you to execute the timer job immediately. First run the "Run-Init" function, then just type $job.Execute([guid]::Empty) in PowerShell. You can also attach to the PowerShell.exe process and do "remote debugging" of your timer job, if desired. Though if you're going to go that far, you should probably just write an NUnit test that performs the same task, and debug THAT. I'm very pro-unit testing frameworks, really, they're great. Anything that closes the code-compile-test loop, in any way, is a good thing.

Tuesday, September 23, 2008 3:10:01 AM UTC 

Summary first, because I just re-read this and it's ridiculous long

If you've subscribed to my blog exclusively for the SharePoint bits, sorry; now is the time to unsubscribe.

That goes for both of you/half my subscriber base :)

I've become increasingly frustrated with SharePoint (I'll explain why in, um, detailed fashion), and instead of souring like a lemon, I'm just going to refocus. This means I'm dropping out of the SharePoint blogosphere to focus on long-term fundamentals; knowledge that won't expire when SharePoint 14 hits beta. I may jump back on for SharePoint 14, but it all depends.

UPDATED 2008-10-18: fixed grammatical and factual errors. By factual errors, I mean that time when I attributed something to Chris O'Brien's blog, when oopsies! it wasn't him at all.

Introduction

I've become increasingly frustrated with SharePoint recently (again, I know), but what's been bugging me more lately is the fact that I'm solving all the wrong technical challenges. I think I have a (deeply suppressed) aesthetic sense, that every so often, rears its ugly head. Or flares out from my neck like a humongous goiter. Goiters are a serious problem—hey!—used iodized salt, it's that easy.

I get agitated at times staring at (undocumented) CAML, or anytime I work with InfoPath, or remembering when to surround hardcoded GUIDs with curly braces (and when not), or knowing when to dispose SharePoint-created objects, or remembering which Workflow activities only work in SharePoint Designer and not Visual Studio, or working around concurrency bugs, or troubleshooting failed Solution deployments. And so on, pick your topic; these are real issues by the way.

So what's the point of this post again?

I want to send the message out there to everyone working with SharePoint, to look beyond your immediate technical challenge, and think. Think beyond your immediate issue, beyond with what you're wrestling at this particular moment, beyond your immediate (sometimes overwhelming) technical challenge with me. Are you ready? Looking at the size of this post, maybe not!

Let's DO THIS.

Are you an expert at SharePoint development?

I recently watched Dave Thomas give a talk about developing expertise. It's interesting, go watch it if you have time; he's not making it all up on the spot, he relies on some kind of research. In the talk he does something I like: he categorizes everyone (using the Dreyfus model) somewhere along the expertise scale: Novice, Advanced Beginner, Competent, Proficient, Expert. And gives definitions for all five categories, and gives helpful tips for how to work with people of varying skills. It's not all fluff, go watch it.

Bringing this back to SharePoint: after trolling the entire SharePoint blogosphere for quite a while now, I've come to the conclusion that there are no expert SharePoint developers. I'd go as far as to say that there are very few proficient SharePoint developers, and you probably know all their names (hint: look for "MVP" in title and/or multiple Bentleys in the parking lot).

The rest of us are just struggling. The "everyone's struggling" theme really sunk in recently when I browsed the source code for SharePoint-tagged CodePlex projects. All of them (save one) disposed their objects improperly. Including mine, to be fair, I'm not hating on your free, labor of love project, I'm making a point. The point isn't to hate on your baby, the point is to say that no one is doing SharePoint development properly. Look on CodePlex; you'll see what I mean.

Let that sink in. No one is doing SharePoint development properly. And hey, your team may have tackled all the object disposal issues, but maybe instead, you're ignoring large portions of the framework and (unknowingly) rewriting large portions of it. Maybe you don't bother packing things in Features and Solutions; maybe you spend too much time packing things in Features and Solutions. Maybe you're great at writing Site Definition CAML but haven't gotten the memo that putting everything in your site definition is a bad practice (or the other memo telling you that custom Site Definitions don't upgrade to v14). Maybe you're unknowingly triggering framework bugs that ruin your customers' trust in your solutions. Maybe you write unmaintainable code.

Maybe you don't understand your customer's core problem!

Maybe, in your metaphorical dwarven greed, you've delved too deep into the framework, and unknowingly stirred the framework Balrog. And you don't want to wake the framework Balrog, let me tell you.

I don't know about you, but I'm there—I'm struggling.

Takeaway: just try to be competent

Dave Thomas says in his expertise talk that advanced beginners think they're much further along than they are, and those who are proficient or experts think they're beginners. Let me tell you: as far as SharePoint development skills go, I'm an advanced beginner, bordering on competent. And I mean it—knowing that he says "beginners tend to think they're experts, and experts think they're worse than they are"—with all that said and digested, I still think I'm a beginner (advanced beginner, I'm not a total scrub :) ).

So, and this applies to those of you who are still new to the SharePoint world and/or isolated—it's okay to admit that you're not an expert. If you need an ego boost, go look at CodePlex—you'll certainly learn a great many things from others' code, but you'll also realize that hey, these people don't have it together either!

Becoming a competent SharePoint developer

What I've also begun to realize, is that the more I learn about SharePoint, the less I want to apply it to different scenarios. Where before I used to say "SharePoint Workflow is supposed to be good at solving that problem," now I say "SharePoint Workflow is painful, and for long-term maintainability I'd recommend you manually code up whatever you need, elsewhere." Whereas I used to say "InfoPath is useful for simpler scenarios," now I further limit InfoPath to "end-user tool only; don't you dare use InfoPath's code-behind." I say these with specific scenarios in mind, I have biases, your mileage may vary, all things within reason, etc, but the point I'm making stands—I'm recommending SharePoint less.

Dave Thomas says, again, that as an advanced beginner you believe that nothing works. Unfortunately, every time I encounter a new piece of the SharePoint framework, I'm an advanced beginner. I don't have the opportunity of being the CQWP guy, every day. Today I'm that guy, tomorrow I'm the InfoPath guy, day after I'm the admin, day after I'm the help desk. I'm that guy.

And hey, I've been an advanced beginner before. For example, I distinctly remember the first time I tried to do Active Directory scripting with PowerShell. It was painful. But gritting my teeth and working through the pain, I became comfortable with it. And, now that I know my way around, I'm able to do some cool (simple) things with PowerShell and Active Directory.

This is a common pattern—I've been an advanced beginner elsewhere, worked through the pain, and by the end, I was comfortable and was feeling more confident.

What I'm trying to articulate in these long rambling paragraphs, is that as I learn more about SharePoint via experience, I become comfortable with pieces of the framework and trust the framework less. As I learn more, I trust SharePoint less. Today I'm totally comfortable in telling you that InfoPath is absolutely great for simple data entry, for simple review scenarios, but nothing else*. And this isn't just InfoPath. As I work my way around the framework, I am becoming comfortable with individual pieces, and as I do so, I am trusting each piece less.

And let me set the record straight. I'm less interested in whether a particular technology is Turing-complete—whether it can theoretically solve any problem—I'm more interested in whether using that technology is a good idea in the first place. And as I learn more about SharePoint, the world of good ideas seems to shrink.
*note for InfoPath nitpickers: yes, it's useful for other things, but yes, it's also easier for me to say "good for nothing else" than rattle off 15 edge cases.

Do you want to become proficient?

I think what is bugging me most is that I don't want to be a SharePoint apologizer. Not apologist, apologizer.

Oh, sorry, that doesn't work. Sorry, yeah, there's a bug. No, yeah, you heard it was great for ECM; well, yeah, it's cheaper, but we'll have to customize a lot. Oh, yeah, that's undocumented. Oh, no, we hit the list size limit early. Yeah, no, SharePoint Designer is unmaintainable, don't build solutions with it. Oh yeah, factor in cost of upgrades on every SharePoint project.

Last week we found out "the hard way" that SharePoint can't handle 12000 documents in a single folder, no metadata, no versioning, nothing fancy. Performance whitepaper aside, 12000 documents shouldn't be a problem, I don't want to hear about it—this is a bug. And meanwhile, from the other end I'm attempting to fend off a total migration of terabytes of network file shares to SharePoint, because "this should work, right?"

I don't want to progress with SharePoint, if my energy is going to be spent tackling the wrong challenges.

Add in the cost of learning

Something that I'm maybe not emphasizing well enough is the point that in theory, all things are possible in SharePoint. And if the cost of becoming a SharePoint expert was zero, SharePoint development would be no question, worlds better than coding by hand. No contest, no question, better.

But learning isn't free.

SharePoint gold rush

Dux Sy refers to a SharePoint gold rush. It's real. It's real in part because the time cost of learning SharePoint is so high. Chris O'Brien (I can't find the entry on his blog) Shane Young states that you should be wary of hiring developers if less than 5 of their last 10 projects are SharePoint projects. It's expensive to find people with that kind of SharePoint experience. Thus, high prices, thus, gold rush.

This equation works for all Enterprise solutions; when I talk about how SharePoint is aggravating my inner engineering brain, trust me, I've worked with other Enterprise systems, and they're worse. SharePoint's a great "Enterprise" product—believe it or not, Microsoft is quite open compared to other vendors, who require all independent consultants be licensed by the vendor, or who provide no public documentation at all—all these extra hurdles artificially inflating consulting prices. Microsoft thankfully doesn't do these things (at least that I notice). So, it could be worse.

No more SharePoint on my blog

I've posted exclusively SharePoint content for a while now, and as I do so, I notice my attitude continues to trend negative. It's my doom-and-gloom engineering brain, it can't just let it go. So, instead of ranting further, I'll just, hmm, let it go. No more SharePoint 2007-related topics on my blog, even if a potential topic is brilliant (WHICH IT INEVITABLY WOULD BE!). No more SharePoint content unless I just can't hold it in.

If you've subscribed to my blog exclusively for the SharePoint bits, sorry; now is the time to unsubscribe.

I'm also officially done learning SharePoint 2007 development, permanently—no more off-hours learning, no more digging through my humongous SharePoint blogroll. When SharePoint 14 is announced, I'll evaluate it to see if they've fixed anything—to see if the situation's improved any. If so, great! Maybe SharePoint 14 will realize the potential of what some are calling "the first real application development platform from Microsoft." There are some awesome business problems that SharePoint already attempts to solve—that with just a little extra help, SharePoint could be a powerful solution. It could be dominant, even.

But if SharePoint 14 is more of the same; if they announce SharePoint Designer Designer 2009, and we're all given the "look what you can do without coding!" demos, and the BDC is still 14 pages of XML without proper tooling, and pieces of the framework are "secretly known" to not work at launch (like the PRIME API), and SharePoint still produces noncompliant HTML out of the box, and I'm still handwriting Feature manifests, and I'm still handwriting CAML, and I'm staring at a doubly-XML-encoded internal field name passed as an argument to a custom XSL function emitting HTML and all this is wrapped in a XML-encoded WebPart tag, and a new SPDataGridView is released and it adds 50 more undocumented methods and properties, and we need to rewrite all our SharePoint 2007 customizations just to make them work with v14…well, we'll see. If MSDN rolls out a SharePoint 14 resource center and it has 3 articles and 50000 useless Sandcastle-generated stub pages—gold rush or no—it may be time to bail.

So what are you going to focus on, Peter?

I have three words for you.

FUN.
DUH.
MENTALS!

Project management (not getting a PMI cert; instead, the meat of project management). Proper object-oriented design. Programming languages (plural). HTML/CSS/JavaScript (they're not going anywhere).

Personal projects!

Programming for fun—something that may even inspire me!

And hey, maybe SharePoint 14 will be a pleasure to work with; maybe I'll jump on that bandwagon…I'm the guy who owns sharepoint14.com, after all.

Monday, September 22, 2008 8:00:22 AM UTC 

A while ago I came across a paper written by Niklaus Wirth. Well, okay, I don't read "papers", but I do read programming.reddit.com, and so came across some guy who does read papers, whose story made it to the reddit front page. I'll just link to him.

So here's how this goes: I'm only going to quote a small section of the original paper, then ask, "does this sound familiar to you in any way?"

Okay, here goes. Niklaus Wirth:

By contrast, modern languages are constantly growing. Their size and complexity is simply beyond anything that might serve as a logical foundation. In fact, they elude human grasp. Manuals have reached dimensions that effectively discourage any search for enlightenment. As a consequence, programming is not learnt from rules and logical derivations, but rather by trial and error. The glory of interactivity helps.

…so, ring any bells?

Monday, September 22, 2008 8:00:22 AM UTC 

Estimating is difficult to begin with, but add on

• a new technology stack with which you're unfamiliar,
• sometimes unreliable and often undocumented API (I can back this up with examples, just take my word for it),
• tight integration/reliance on Active Directory, SQL Server, IIS, Exchange, Office, and Internet Explorer, all of which may fail in difficult-to-debug ways;

Given all this, how much risk are you taking on if you provide a tight estimate? If you haven't tackled three or more Visual Studio Workflow projects, for example, how do you even go about getting a ballpark estimate on your first one?

On a recent project I put a 10x difference between my lowest and highest estimate for an SAP integration task, and everyone complained to me about how "there's too much variance." Well, what should I put there, I've never tried it before!

This isn't a passive-aggressive way of trying to win a work argument via my blog. If I start doing that, dude, let me know, I don't want to be that guy.

But it is a cry for help. How are we supposed to estimate SharePoint development tasks? If you're not an experienced SharePoint team, and by "experienced" I mean experienced in building the exact same solution, using the exact same method calls on the exact same SharePoint objects—if you're not truly experienced, then how do you even attempt estimating?

When you may be blocked anywhere between zero to ten times on a task, and each blockage may take up to a full week (or more) to resolve?

When your project approach hits a dead end? A brick wall?

In traditional build-it-from-the-ground-up systems, you can estimate the complexity of your requirements and envision a rough idea of what your solution will look like, and compare the complexity of your new task with similar tasks you've done in the past. This works for traditional development, so long as you double your estimate and increase it by an order of magnitude :). The idea is, when you're building it from the ground up, estimating is "as simple as*" estimating your effort based on the complexity of the problem.
* saying estimating is "as simple as" anything is an implied joke; you may laugh now

But if your task says "write a custom field that pulls data from SAP," or if someone says "hey, let's build this web part with Silverlight!" …how do you put down an estimate? When the task, performed by an experienced expert (see above for my definition of experienced) takes an hour, and in reality the bulk of your time is spent discovering how to do the one hour task?

What do you do when your honest estimate is, "this will take between 40 and 400 hours?"

I don't have the answer

I'm just throwing this out there because no one even talks about it. So here it is. When you're on a project and someone asks you for an estimate, and you answer "4 hours to 4000 man-months", and they laugh, hey—you're not alone. I can't figure it out either.

UPDATED 2008-10-21: I made mostly grammar fixes—so if you notice some changes, no, you're not going subtly crazy, this post changed a little.

Wednesday, September 17, 2008 8:00:30 AM UTC 

I've been fascinated with learning as much as I can about SharePoint 14, also known as "the next version of SharePoint." You can see the list of everything I can find publicly here: http://www.pseale.com/blog/SharePoint14EverythingWeKnow.aspx (nothing much has changed since I wrote that post).

One of the advantages of picking up these little tidbits (as well as reading every SharePoint blog known to mankind) is that you can make better guesses.

So, what follows are my educated guesses as to what technologies we'll be adding to the SharePoint arena, with the idea that you can brush up on some of them in preparation. Or just get depressed. I'll help speed the process, either way.

Technologies we'll see in v14

PowerShell

I've seen a hint dropped (by a MS employee) that PowerShell will be part of the next SharePoint framework. I'm guessing admin-style cmdlets, maybe better wrappers than using stsadm. I've become accustomed to using stsadm, but could definitely use some PowerShell shortcuts, especially if they could balloon the number of things PowerShell exposes. There's a lot of potential for using PowerShell as an admin interface, way more than even Exchange 2007 is realizing. And all this on top of everything PowerShell gives you today.

Anyway, learn PowerShell! It's going to be the easiest thing to learn, by far, and as I tell everyone, PowerShell is one of the few technologies that has saved me time, even including the time spent to learn it.

Windows Communication Foundation - WCF

It's inevitable that SharePoint will expose WCF services, and even if SharePoint doesn't, then we'll still eventually be writing our services in WCF (as opposed to ASMX web methods). So, learn the ABC's of WCF. Sahil Malik covers WCF for SharePoint 2007 on his blog, so maybe that's a good place for you to start.

We can hope for WCF services that replace the SharePoint built-in web services of today, though I honestly don't know how they'd improve. Maybe they'd be able to generate list-specific services? I.e. instead of a big "GetListItems" that takes the equivalent of a SQL SELECT query, they'll have something more context-specific? We can hope.

Silverlight 2.0

I'm not sure what form this will take; at minimum we'll have Silverlight Web Part templates. We can also maybe get some "all-Silverlight" site templates, where you visit the site and you don't see a single bit of HTML. As far as I'm concerned, Silverlight looks like a more difficult way to do the same thing we're already doing, but hey, that's me, I'm cranky. I'm going to skip this whole Silverlight thing until someone proves that it's easier to work with Silverlight than the equivalent solution using HTML/CSS/JavaScript; I'll re-evaluate periodically. I'm way too overwhelmed with learning to take Silverlight seriously, so don't expect me to jump on this bandwagon anytime soon.

It's also possible InfoPath will use Silverlight; think about the possibilities (but as always, don't get carried away :) ).

Also, if you're learning Silverlight, then you'll be learning WCF, gotta talk to the server from your Silverlight client somehow.

Claims-based authentication

This is a big shift and, while it may not happen for SharePoint 14, it is probably worth learning about anyway. Here's the NetworkWorld article from 2007 discussing the possibility.

.NET 3.5, IIS 7, Windows Server 2008

I'm including all these for completeness' sake; duh.

Stuff we haven't heard about yet

This section is also included for completeness. While I don't think v14 will be as big of a shift as 2003->2007 was, what do I know? Maybe the entire Dynamics platform will mash itself inside SharePoint; you can only hope (dread) the possibilities. I'm just saying here that no, I don't know what else we're going to get. I can't imagine what Access Web Access will look like. I don't know how Groove is going to work in the new version (are we going to get offline support out-of-the-box? I don't know). I don't know what the SQL-table-backed-lists will do and how they'll work. I don't know if they'll revamp the BDC. There's lots I don't know.

New slices in the SharePoint pie

Master Data Management (MDM)

This will be big. Read up on MDM. You're not going to be an MDM expert, but at least you'll know that it's not as easy as running UNION queries on disparate data sources. Hopefully that doesn't go into the marketing packet.

FAST search (Enterprise Search revamp)

This will change SharePoint's search engine, maybe not in time for SharePoint 14, but maybe so. Microsoft has proven they are able to do out-of-band improvements on their Search platform (MS Search Server), so even if this doesn't happen in time for RTM, you won't have to wait until v15.

Here's a discussion of the FAST Search acquisition on CMSWatch; you can read elsewhere (MS Enterprise Search team blog) about the acquisition, and of course you can check out FAST's product itself, as it exists in its non-integrated form.

Do you have time to learn all this?

If there's an argument to be made for specialization inside of SharePoint, combining this with the existing SharePoint 2007 architect skillset should do it. Don't even try to stay on top of all this; you can't.

Wednesday, September 10, 2008 2:45:24 AM UTC 

In my learning journey from earlier this week, I forgot to mention something:

Don't dispose SPWeb or SPSite objects you get from the "properties.Feature.Parent" property. This rule doesn't come from me—I got this information from a reliable source.

So, here's a little visual guide:

 

From my brief survey of CodePlex projects, it looks like most people were following this rule anyway.

Tuesday, September 09, 2008 11:02:20 AM UTC 

Read about Patrick Tisseghem, sad

Tuesday, September 09, 2008 10:29:40 AM UTC 

Update: 2009-03-31: Go visit someone else's blog

Go visit Keith's blog for the definitive SharePoint disposal article. Also be aware that Microsoft has released the SPDisposeCheck tool.

That's pretty much it.

Introduction

If any of you out there are looking to find reasons why SharePoint is a "challenging" developer environment, you need look no further than explore the guidance on when to dispose SPWeb and SPSite objects. I was reading Chris O'Brien's blog (which is a veritable font of knowledge on SP dev topics) and found this page, discussing his generic approach to disposing objects, and figured I'd better post this so I can just get it out. Keep in mind folks, that every single SharePoint developer in the world must deal with this issue.

To introduce the problem: Your friendly SharePoint objects SPList and SPWeb will on occasion interact with a COM component. Each pop, for each reference, they hold between 1MB and 2MB of memory. That adds up quick.

So far we're okay, because .NET has a facility for disposing these unmanaged resources (among other things)—through the IDisposable interface. This is why we tell everyone to wrap their code in using blocks—it's to make sure that the unmanaged resources (like the COM components I mention above) are properly released. If you're writing code in SharePoint and you're don't know what a using block is, go read up on it. If you're ignorant of this, it isn't the end of the world for you, but it's close.

Where were we again? Oh yes—SPLists and SPWebs hold unmanaged resources and must be disposed. The "duh" solution is of course, to wrap everything in a using block.

Unfortunately, disposing in SharePoint ain't easy

The first problem you'll run into is when you try and use SPContext.Current.Web - a static getter method. Do you dispose the SPWeb object here, or not? Note that if you dispose an object that's in use elsewhere, you'll cause new problems! Problems you didn't forsee!

This is starting to feel like whack-a-mole. Whack the unmanaged resource mole, up pops the "you whacked the wrong mole" mole.

Whack the "you whacked the wrong mole" mole

So now we know, okay, dispose SPWeb and SPSite objects, but don't dispose the SPContext.Current.Web object. Okay, two things to remember, I can handle that. Unfortunately, we're not done—we're not even close to done. We still have to deal with the "SPList.Web" reference mole, "SPSite.RootWeb" reference mole, the "enumerating through a collection" mole, the "I didn't use the unmanaged resources, this time, and thus don't really need to dispose" mole, and so on.

Disposal strategy Whack-a-mole

There are a bunch of resources you can read on this topic, including:

• Best Practices: Using Disposable Windows SharePoint Services Objects [on MSDN] - good introduction to the topic, but not comprehensive - thanks Jeremy for the link
• Roger Lamb's [MS employee's] nearly-comprehensive list of dispose patterns by example
• Disposing SharePoint objects - what they don't tell you by Chris O'Brien [MVP], as linked above
• Keith Dahlby has posted several times about this:
  • http://solutionizing.net/2008/07/23/spsite-spweb-leaks-revisited/
  • http://solutionizing.net/2008/08/10/is-splistparentweb-a-leak/
• …probably lots of others I haven't referenced

I'm still playing whack-a-mole, but I'm holding steady at the "apathetic about which mole to whack because there are too many moles to keep track of" mole. Oops, sorry, "apathetic about which mole to whack because there are too many moles of which to keep track—there, now I'm not ending my sentence with a preposition" mole.

I'm probably stretching the whack-a-mole reference too far

If I've lost you in the craziness above, let me just summarize by saying: a) object disposal in SharePoint isn't simple, b) I've given up looking for the "perfect solution".

Let's tackle my less-than-perfect solution for object disposal next.

Peter's (newer—thanks Keith) imperfect strategy

Let me first say that I wrote up "my current technique" in this post and secretly published it to my blog. Unfortunately (or fortunately) "secretly publish" doesn't work as well as it should and I got comments. My original approach, which was my firm stance as of Sunday, has changed quite a bit, it's been a full 24 hours. I've taken a little learning journey in the comments below, and in a post on the MSDN forums, where I got an authoritative answer within an hour. Nice!

So, anyway. My newer strategy is to basically copy either Chris O'Brien's or Keith Dahlby's approach, with the one extra little tidbit thrown in: if I'm unsure whether or not to dispose an SPWeb object, I will gather the minimal information necessary from this SPWeb object (e.g. RelativeUrl, Id property), and spawn a second SPWeb object, which I am safe to dispose.

Here's the full algorithm that I use to guarantee I'm safe:

• Follow published guidance to figure out whether the method call/property getter I'm using will produce a safely-disposable SPWeb or not. This applies really to any IDisposable SharePoint object, including SPSite, PublishingWeb(?), Area(?), and objects of which I've never heard. For this example, SPWeb.
  • If definitely yes,
    • Great! Enclose in a using block.
  • If we're unsure for any reason,
    • Use the SPWeb object we're given, and somehow, with the lightest footprint possible, create ourselves a second SPWeb object, that we're guaranteed we can dispose!
    • We can use this "spWeb2" object and proceed along our merry way, disposing it with glee!
  • If definitely no,
    • Great! Don't dispose it!

Tear it up

Feel free to tear this approach up (assuming I or someone else hasn't already done so). I can take it.

Tuesday, September 09, 2008 8:00:17 AM UTC 

I recently read Matt Blodgett's post called SharePoint is the Future of Microsoft-centric Application Development, which is interesting because there's a grain of truth in what he says (not more than a grain though :) ). I want to focus today on the following quote:

[…referring to SharePoint projects] Clients are amazed when they see 80% of their app done in a week. What they don't see until later is the weeks/months of excruciating work it then takes us to crowbar the remaining 20% into an ill suited platform.

Ok. I'm only quoting Matt above because his post was the closest at hand when I need to quote someone—I hear this everywhere (and have probably said it myself several times before). It's completely correct; you want to try and solve the customer's core problem, something SharePoint's out-of-the-box features are good at doing quickly.

But, I need to elaborate. Something I'm not sure everyone is getting, is the corollary:

Lesson: You can stop after the easy part!

You can stop at 80%! This sounds obvious to me, but maybe someone out there hasn't had their eureka moment. "Now wait a minute," I hear you telling me, "we need all of this to work, not just the easy parts!" Okay, let me tell you a story.

Story time!

Earlier this year, I was asked to estimate how long it would take to build an employee vacation scheduling solution in SharePoint. As I dug into the problem, the complexity jumped out at me. We were going to pull some data from the ERP system, run a gauntlet of vacation prioritization, and even include a period of "open enrollment" wherein employees' requests would be put in a prioritized queue. And write something back to the ERP system. And supply complex reports. And so on; it was clearly not something we'd be able to solve with out-of-the-box features and would take a while.

After asking the client about cutting bits from the app, they explained that they couldn't cut features; we'd have to do it all.

So, we had a relatively complex app to build, and limited leeway. What were we to do?

In the end, the client and we agreed on not implementing this as a SharePoint solution. We parted ways, so to speak.

Now, you may ask yourself, where was the "80%, then stop" in this story? I did say, after all, that they didn't pick SharePoint for the app, right? Right.

What I didn't tell you is that SharePoint was already running their vacation scheduling; it was doing the 80% of what they needed, and they came to us asking for the remaining 20%. Because SharePoint was not the most efficient way to solve this complex problem, and because there were no outside extra benefits to using SharePoint, there was no point in implementing the solution in SharePoint.

Lesson learned: wait, what was the lesson again?

Ok, here goes:

Lesson: You don't have to use SharePoint!

You can develop apps outside of SharePoint! I'm sorry I typed this up; it's too obvious. Let's take a more helpful approach, something you can use:

Lesson: Know 'why' you're choosing SharePoint as your platform.

Even if you need to run your app in SharePoint, so to speak, what specifically do you need to run in SharePoint itself? Do you need to store your data in custom lists? Do you need to run all on the same physical web server? …now are you sure about that?

I was inspired when I saw kroger.com, which is a public SharePoint site by the way, and how it handles integration—while the look-and-feel is all the same, it's clear that they're only using SharePoint for its publishing features, and "going outside" for any of their complex needs. They don't even try to get everything onto the www.kroger.com domain; if you look carefully you'll see you're being redirected to different servers. Which is fine. If simply "allowing us to run multiple domains" cuts costs by 80%…

Personal bias

I have not worked on any sort of ECM or huge portal project. If I had, I might have had a different opinion; in those cases, you may "need" to have everything run in SharePoint; maybe the data's stored in SharePoint document libraries, or maybe you need need everything to run natively on your portal. But as I mentioned above—if you don't know why you need it to run in SharePoint—then maybe you don't need it at all!

Sunday, September 07, 2008 4:44:06 AM UTC 

In a previous post, called Thinking Creatively, I asserted that in SharePoint, "we are forced to think creatively" in crafting solutions. Like when someone asks you to create a custom web part on the intranet home page to do employee search in SharePoint, you are supposed to say "hey, aren't you really asking for People Search?" And the idea is that you would then work with your customer to find the real solution to their problem, not just blindly follow directions. Maybe People Search works for them, maybe you end up writing a custom web part or configuring SharePoint's search. You solve the real problem.

Well, I was wrong.

If you remember to do so, you'll think "out of the box" in  "dynamic way" and "harness the synergies" et al, as mentioned in the scenario above. But if you forget, if you slip up, guess what?

Or what happens if your client is belligerent and wants it done their way? FYI for everyone who knows me, I'm not saying this is my client :), but have instead heard secondhand about these situations.

Bad times.

In Summary

Remember to argue with your customer! This is a fundamental aspect of any project, whether it's SharePoint or not—it just happens to sting more (a lot more) in SharePoint when you're forced to follow strict requirements.

I'm not saying you should add a "fistfight" bullet point to your next customer meeting agenda—or maybe that's not such a bad idea after all! Treat your meeting like a boxing match: instead of doing bullet points, say "ROUND 1: do we really need to brand SharePoint?" and "ROUND 2: Can we change the way this app "runs in SharePoint"?" Set the Outlook meeting location to be "THE OCTAGON". Bring a metal folding chair and lean it next to the door, in case someone needs some extra "chair therapy" across the back. Go wild, it's your meeting.

Tuesday, August 19, 2008 8:00:31 AM UTC 

One of my friends is starting his first SharePoint project, and I had a few words of advice for him. He doesn't know exactly what his project entails, I don't know what his project entails, but we will start with the following assumption:

The following advice assumes that we are working on a small-to-medium sized, line-of-business application, possibly something attached to a traditional portal (i.e. intranet). We are not working on a BI project, not rolling out an entire enterprise intranet, not creating a publishing site (WCM), not even doing formal records management.

First tip: Read this other guy's post on the subject

I found an excellent post by Mark Jones written mid-2007, which is ancient history in SharePoint terms. If you can't be bothered to read both his post and mine, choose his: Mark Jones on SharePoint projects (original post disappeared, this is a link to the post in my Google Reader)

Second tip: Use either WSPBuilder or STSDEV in Visual Studio

It is well known that no one really uses the Microsoft-created Visual Studio Extensions for Windows SharePoint Services (VSeWSS). Well, no, if everyone already knew, then I wouldn't be writing here, would I now?

So let's start with a fresh mind. You're new to SharePoint, and you need to do … something, let's say change the PDF icon that appears in document libraries on your SharePoint farm. Terrible example, I know!

I'll assume you have rudimentary knowledge of SharePoint's Solutions and SharePoint's Features. If not, read up on it, there are intro books and articles out the wazoo. I'm not going to into depth here; read up on Solutions and Features elsewhere.

Moving along.

So you fire up Visual Studio. You've already installed VSeWSS, and now you've got this "SharePoint" tab in Visual Studio, and you click on it.

I'm not here to only throw rocks; I'm here to say that, unless you want to endure a great deal of pain, don't stick with only VSeWSS.

You have alternatives:

• WSPBuilder via the WSPBuilder extensions
• STSDEV
• "Naked" class library project with WSPBuilder as a post-build task
• "Naked" class library project with MAKECAB (!!!) as a post-build task/MSBuild task

To the initiate, to the untrained eye, these are all equally good options. They're not. Let's work through the list again, this time with a critical eye:

VSeWSS - Visual Studio Extensions for Windows SharePoint Services 1.2

You're going to have to install this as it is the easiest way to get the SharePoint-specific Workflow activities. Otherwise, don't use it. Search for others' complaints about VSeWSS; general consensus is that it's okay, but painful.

STSDEV - a proof-of-concept utility

Link to STSDev project home 

Contrary to the aggravating unnecessary disclaimer prominently displayed on the CodePlex project home page, STSDEV can be useful in actual, real-life projects! Shh, don't tell anyone, they might figure it out!

Aside from my nitpicking, this is a good tool.

• Generates the Solution manifest.xml for you, for limited scenarios
  • In fact, generates an entire project skeleton. This includes the feature.xml manifest, something for which you'll be thankful if you're learning the ropes.
• Builds your Feature event receivers for you
• Screencasts on how to use it! Ted Pattison is an excellent communicator and presents three screencasts, available from the 1.2 (previous release) page on CodePlex.
• In other words, STSDev is a good way to start off a new project, assuming it has a template for the thing you're trying to build. I don't use this as I've pretty much settled with WSPBuilder, but acknowledge it's a good solution, and will definitely use it in the future if I find STSDev has something WSPBuilder lacks.

WSPBuilder extensions

Link to WSPBuilder project home

The WSPBuilder Extensions are relatively new—they add Visual Studio project templates, quick shortcuts, and WSPBuilder-specific item types. LIke the WSPBuilder command-line tool, the WSPBuilder Extensions also completely eliminate the pain of Solution packaging.

• WSPBuilder project template is a real, new Visual Studio project template. It saves you the trouble of manually configuring your Visual Studio project to work with SharePoint; stuff like making a .snk file, adding a "Build WSP/deploy/upgrade/uninstall" option to your project (further note: uses SharePoint object model directly for faster performance); and other little bonuses (see screenshot above).
• Downside: Unlike STSDev, you're locked into WSPBuilder Extensions once you start using the Project Template. I'm not worried, because I would feel comfortable switching the project over to a raw class library, but you may be allergic to dependencies like this.
• New Item templates (like "Web Service") - I haven't tried these, nor have I heard reports of how well they work; so I'll just say—try them out, if they work for you, great; if not, definitely don't use them.
• Assuming you arrange your artifacts in the correct directory structure, WSPBuilder will eliminate the pain of building Solutions. See my review of WSPBuidler.exe below for more details on what this does.

WSPBuilder.exe - command-line tool only

Link to WSPBuilder project home

In ye olde days, as far back as almost a full year ago, we were not blessed with either STSDev or WSPBuilder Extensions. We made do with what we had; WSPBuilder.exe.

WSPBuilder.exe is great because it is focused: it does one thing, and it does it very well. What is that one thing, you may ask?

It eliminates the pain of Solution packaging. That's all it does. To be honest, I don't even know how to build a Solution manifest XML file, and I attribute my brazen, unashamed ignorance to WSPBuilder. I don't need to know because WSPBuilder takes care of that for me.

The pain begins, however, with the rest of your project. WSPBuilder Extensions address some of this pain, and I recommend you start with WSPBuilder Extensions for new projects, but you'll find many projects that were started during the "WSPBuilder-only" era. Don't make fun of these old projects; we can only hope someday that we will be able to look back and say "what primitive tooling we had in 2008!" Just like you're wont to do when you see a project that uses WSPBuilder.exe or MAKECAB.exe to build its Solutions. We can only hope the tooling improves.

If you want to see a simple example of how to work WSPBuilder.exe into your Visual studio project, I demonstrate one way of doing this in my terrible CodePlex project. F5 is my build, and I'm (almost) unashamed!

Raw MAKECAB.EXE

This seems to be the preferred method for the "old school" SharePoint experts who have been around. I assume that, for them, they have learned how to use MAKECAB.exe and build their own Solution manifests. Ugh. Ultimately if you know what you're doing, this is as good as any other way, but for the first-time SharePoint developer, this isn't a good option. I want to point out MAKECAB's existence because the MSDN-sponsored articles and project templates all rely on MAKECAB.EXE. You don't have to listen to those articles; save yourself some pain and pick something else. There are far better alternatives to hand-crafting your own Diamond Directive Files and hand-crafting your own Solution manifests. Look around; better ways of doing this exist!

Recap

Both WSPBuilder Extensionss and STSDev are good solutions for the budding SharePoint developer; they both provide some tools to help you create and build SharePoint projects in Visual Studio. They are in no way complete tooling; you will still experience pain, but they both help greatly. Try both out, choose one and go.

Just don't stick with only VSeWSS or only MAKECAB; save yourself some pain and choose one of the alternatives.

MORE TO COME!

This post has gotten ridiculously long, and I barely just compared the major competing developer tools! We're nowhere close to done; more posts to follow on this subject.

Monday, August 18, 2008 8:00:15 AM UTC 

When converting an existing "please leave us a comment" form, I had a eureka moment: SharePoint's user profiles are incredibly useful. Let's do this graphically:

Before - ask the user, again and again

After - SharePoint stores user data automatically in the "Created" field

Well, you get the idea anyway

The point of this exercise is to show the power of integration. Specifically, user data like phone, email, work location, department, etc. In an ideal world, SharePoint will pull this data for you from wherever it's stored. HR type data from the HR/payroll system, email from Exchange/AD, and so on. With proper integration you can remove all the pain of online forms; you can completely remove the five "who are you again? and what do you do again? and how do I get a hold of you again?" fields. They're gone; thanks to proper system integration, you already know and don't need to ask these questions again.

Footnote: it's not all gravy

I must acknowledge that there is no easy way to grab and show all this extra data from the user profile without writing code. I wish SharePoint would have made this easier, but alas, 'twas not to be. In the browser, you can certainly click through to see their user profile data, which is good enough for the example above, but that won't always suffice.

Second footnote: user profiles in all their glory require MOSS Standard or better; this is not a WSS-only feature.

Sunday, July 13, 2008 4:50:49 AM UTC 

Ok, so I'm burned out.

I realized this while digging through my queue of SharePoint-related blog posts. I wasn't reading all the technical bits (which is normal), but I realized I didn't even want to know what any of the blog posts were about. This antipathy is new.

But I'm not writing here, today, just to tell you I'm burned out. I'm going to hit a few topics and hopefully draw them all together at the end. Maybe by the end we'll figure out why I'm so burned out. Let's do this.

Do you have time to learn everything?

I came across a post (via DotNetKicks) asking if we have enough time to learn everything in the Microsoft stack. It's an interesting question, and always leads to good conversation. "How do you keep up?" is a fun question to ask.

Unfortunately the author, in what has to be a misguided set of priorities, has chosen to devote all his learning efforts to LINQ to SQL and the Entity Framework. Which is a ridiculously bad idea. Don't worry, I told him the same thing in the comments, I'm not just sniping from afar.

What bugs me about this idea is that it's almost the exact opposite of my learning priorities. LINQ to SQL and the Entity Framework are almost dead last on my list—how did he end up choosing these two volatile technologies over everything else?

Think about that for a second. How did he arrive at the misguided conclusion that learning these two things was a good idea?

If you're interested in my specific critiques, they're all in my comment on his page; I won't repost them here.

When doing research, always consider the source

I also came across a discussion started by Andrew Connell on how CMSWatch is biased against SharePoint.

If you're unaware, CMSWatch is a site that sells a $1450 SharePoint report that critiques SharePoint in a variety of categories, as well as provides advice. Andrew Connell is an MVP and is a known authority on SharePoint WCM and developer topics.

AC's point was that not only is CMSWatch biased, but they've also caused him extra trouble in the past—he explained that he had to 'clean up' after them—where he has had to expend extra effort convincing clients that SharePoint can properly do WCM. People who haven't been 'poisoned' by CMSWatch don't require this extra effort.

My comment to him (poorly worded and not in total disagreement) is that I'd rather have CMSWatch and their criticisms of SharePoint than not—I'd rather "clean up" by convincing clients that SharePoint can work, rather than "clean up" by convincing clients that SharePoint isn't the 'everything solution.'

I'm not here to weigh in on the actual discussion of whether CMSWatch is biased or not; I don't know, I haven't seen the report. I will say that this summary written by one of the principals is unfair (go see the 'Ugly' section), but for the most part I'll stay out of it. I don't know.

Instead, the question is: why do I encounter SharePoint overexuberance more often than SharePoint skepticism?

Oren Eine at the Future of .NET Panel on DotNetRocks

A while ago I listened to a podcast, probably my favorite DotNetRocks episode of all time: DotNetRocks Show #346: The Future of .NET Panel. Even if you're totally bored by me, go check out the site and download the podcast, it's a fascinating (and you'll be happy to hear, civil :) ) discussion of .NET development in general.

The thing that really got me going is at ~1:06:00 into the podcast, when Oren Eine describes a challenged deployment of MS CRM customizations:

Question here: why did Oren's story strike such a chord with me? He wasn't talking about SharePoint at all!

Do Less. Get More. Develop On SharePoint. - www.mssharepointdeveloper.com

Also launched sometime recently is the SharePoint Developer introduction for .NET developers. Instead of writing, I'll express myself by defacing a screenshot of the front page of the site:

The question for you, dear reader, is why am I so virulently opposed to including Silverlight in the list above? It's the future, right?

And what's the deal with the Entity Framework vote of no confidence?

This is way too big a topic to cover properly; I'll just assume you know what I'm talking about and skip straight to my leading question.

Why does anyone care if the Entity Framework is released as-is or not? What difference does it make to them; they can just use something else to do data access! Why can't they just live and let live?

"SharePoint is a guild", by Tony Byrne

I also recently attended a presentation by Tony Byrne (from CMSWatch, yes, see section above). I've covered the session in more detail here, if you're interested; the quick summary is that he was providing a quick critical review of SharePoint, a sort of summary of their report. One statement that caught my attention was that he likened SharePoint to a guild. I don't recommend you take this metaphor too far, I'm not a medieval scholar and I don't think he is either—but the idea is that there are a select chosen few who are in the "guild" and have collected secret, arcane knowledge. Those outside the "guild" are not privy to the same knowledge and benefits of those in the "guild".

We could have a very long and unproductive conversation about this, there's several ways to go with it. But why does this crazy idea of a SharePoint "guild" have any merit at all?

Ok, let's pull all this together

We've wandered far and wide today. We've talked about some random guy's technical learning queue; we've discussed the impartiality of CMSWatch; we've heard a story of a painful MS CRM deployment; I've defaced a screenshot; we've approached the topic of the Entity Framework before running away; and finally, we've talked about medieval guilds.

Where does all this relate again?

Ok, I'll try to boil this down. [re-reading the section below, it appears that I failed to 'boil it down'. Too bad; we're stuck with it. -ed]

Overexuberance, belief that improvement is inevitable

What I experience on a regular basis is an overexuberance, a faith in the current crop of Microsoft's technology, as well as the common belief that future improvements are inevitable. In reality, however, no technology is perfect, and many improvements perceived as inevitabilities will not materialize.

Some of this overexuberance is the result of aggressive marketing. But I'm starting to realize as well—we're duping ourselves. We don't even need marketing to tell us that, say, Silverlight is going to be 'the best development toolset provided for the web'—no need for marketing, we'll do that all by ourselves!

We believe that SharePoint's WCM featureset will improve, without any evidence or indication from Microsoft that this is the case! Will SharePoint 14's HTML be accessible out of the box? Are you sure about that? Who told you that? No one?

Learning investments with limited time

What seems to be completely ignored by Microsoft is the hidden cost of learning—for each technology or framework or product released, there's an associated cost. Learning is the number one cost of introducing any new technology!

So why is it then, if our attention is so valuable, that when we are presented with the foundations of SharePoint development, we see Silverlight in the list? Not only is Silverlight completely useless to me for SharePoint development today, 2008-07-12 (note I timestamped this assertion), but it's probably actively worse than the competing technologies (which are some combination of InfoPath, HTML, and/or something "not in SharePoint at all")? Microsoft, instead of attempting to ease my learning burden, sneaked in the additional burden of Silverlight…which is useless!

Now hey, I hear what you're saying: so what if they threw a little marketing in there? You can just ignore it, right?  I can. You can. But what about the guy who has heard about this "SharePoint" stuff, and has also seen something about "Silverlight", and hey, look at the website! It says you can make these two things work together! Sounds easy!

Now fast forward a bit. When this poor guy can't deliver, whose fault is it? Sure it's his fault, somewhat. But is it his fault he was distracted for 3 weeks trying to get his Silverlight applet working in a SharePoint farm?

So is this why everyone's so excitable about the Entity Framework? Because it will make their jobs harder in the years to come—not easier?

Now let's hop back to SharePoint's WCM again, but this time, in the context of learning. If you installed the product sometime in 2007 and discovered a) emitted HTML was not accessible, b) branding was difficult, c) content deployment jobs failed often (hopefully you catch this one before go-live!), then were you supposed to think? Sure, if you knew all the right things, you could work around the problem. But what if you didn't? Is it your fault you can't deliver? If Andrew Connell, MVP, can do it, why can't you? If hawaiianair.com (which as I understand does not use MOSS's WCM features) looks so cool, why can't you brand your portal as successfully?

…Oops, sorry, I kind of drifted off while you were talking there, I was reading an article about how to AJAX-enable my web parts. It says here that I can throw in an UpdatePanel in two minutes! What could go wrong?

And the capper. What kind of crazy world do we live in, where someone is encouraged to learn the most volatile and transient technologies over fundamentals! His post had 11 kicks on DotNetKicks; 11 people agreed with him? He's not alone in his opinions?

So I'm burned out

I'm going to go on an "information diet," for as long as I can hold out; I'll be fine, I've been through this before. No technical learning outside of work, outside of my immediate duties. Shut down the Google Reader, close the book, hide the laptop at home. Not a problem, I'll make it back refreshed.

But if I can summarize my ranting above, it is:

Overexuberance in the current crop of technologies, along with faith in unproven, possibly actively unhelpful or worse future frameworks, makes my job harder, today.

 

Microsoft's strategy of releasing frameworks at an overwhelming rate has left every Microsoft developer overburdened with technical learning. Microsoft (and we ourselves!) worsen this problem by pushing new, unproven, possibly useless-out-of-the-gate technologies on ourselves, before they are even ready—and this will make my job increasingly harder for the forseeable future.

Aside: Houston ALT.NET 'geek dinner' at Star Pizza 6PM on Monday, July 14th

If you made it this far, then maybe you'll be interested in attending the ALT.NET informal gathering this Monday, July 14th, at 6PM at Star Pizza.

And as always, the "sweet place to hang out" is on the houstonaltdotnet mailing list; all the cool kids are subscribed.

Monday, June 30, 2008 11:27:04 PM UTC 

Quick link: http://www.codeplex.com/SharePointPdfIcon

 

This is possibly the most important customization you can make to SharePoint. OF ALL TIME.

By default, SharePoint does not include a PDF icon. When you upload a PDF to SharePoint, you see:

 

This is unacceptable!

Manually adding the PDF icon

I'm not here to tell you how to manually add the PDF icon to your SharePoint farm. Steven Van de Craen already does an excellent job, so I'll just link to him.

But manually adding a PDF icon? So 2006-2007. We're in 2008 people, let's get with the times.

Programmatically adding the PDF icon

Recently Steve Goodyear from Microsoft posted step-by-step details of how to programmatically add a PDF icon to your farm. Which is great.

What Steve did not do, however, was go the extra step and provide a working .WSP file.

Ok. At this point you're asking yourself: "Well Peter, since you're not bothering to tell me how to manually add a PDF icon, and you're not going to tell me how to do it programmatically, then why are we here?"

Introducing the most awesome SharePoint Solution package ever

Introducing: the PDF Icon installer!

I've developed a Solution package to change the dreary, dull, blank icon, to a dynamic, Web 2.0, hot hot hot PDF icon!

Feedback

If something is terrible, let me know and I'll fix it. Issues/Discussion page on the CodePlex project page. Or if it's really terrible, email me direct :)

Where to get it

http://www.codeplex.com/SharePointPdfIcon

Saturday, June 28, 2008 11:25:00 PM UTC 

Yesterday and today I attended the SUGDC Summer Regional SharePoint Conference. And by "Regional", we mean Washington, DC and surrounding provinces and territories. Telling the story via graphics:

I'll point out I didn't drive, I flew, though the flying experience probably took longer on the way up than driving would have. Yes, I'm aware that driving can take quite a while; yes, my flight up was nightmarish.

People

As I've been told before, the most important thing about attending a conference is not its sessions; it's the people you meet. I'm not going to call out names, but it was great to be able to meet a bunch of people of differing skillsets and backgrounds. And by differing skillsets, I mean "expert," "expert PLUS," and "Super expert plus, TURBO"—we're all pros here. Of course, of course; me too.

Fun Tidbits in the Sessions

Since I did dutifully take notes, I'll share the fun bits here:

• Errin (whose company is based out of Houston) presented an excellent overview of SharePoint's various capabilities and what projects they're doing. A good throwaway example is the slide that declared: "Documentum: $1.2M annual license. SharePoint: $400K one time fee." …the point being, even if you have to pay to customize, SharePoint is still the cheaper choice.
  ANYWAY, the thing that made this interesting (and different) is that he has done the projects—they're real, they're not all marketing hype. Believe the SharePoint pie chart! Well, kinda anyway, we'll get to that below.
• Errin also mentioned the importance of getting 'quick wins' in selling your SharePoint deployment. Throughout the weekend several others hammered this home as well, and it makes good sense: Step 1: find something that is a good match for SharePoint and is generally easy to do; Step 2: do that thing; Step 3: profit! Funny enough, I hear the same thing spoken about SOA implementations, although the guidance I heard was more like "3-5 projects." Since parts of SharePoint are designed to solve common business problems, you'll find it's, gasp, actually useful for your organization too!
• Interesting: he says the magic # to extract your storage requirements from your raw data is about 3.5. So if you have 100GB of files sitting on a shared drive that you plan to get into SharePoint, then you will need roughly 350GB of storage to accommodate.
• My Site governance: if we get the slide deck, I'll recommend looking at it to find a list of all the things that can go wrong with your My Sites. We're all aware of the big offenses, wait, let me capitalize that: Big Offenses—but we're not sure what else to look out for. Check his slide deck, there's a good summary there.
• Onto the next session! The Red Cross folk presented on their Communities of Practice, AKA their "Neighborhoods", which I'm loosely defining as "small targeted online communities." E.g. for the Red Cross, something many people are interested in is Measles. I don't know why either, something to do with saving lives. Anyway, my take on this whole thing is that the technical problem is solved, and has been since Philip Greenspun's ArsDigita started doing this sort of thing for corporations in the 90s. The only thing that remains is the human factors—getting people to, you know, use the stuff that's out there.
• Morbid tidbit: one of our presenters (I will withhold company) mentioned that one of the great catalysts spurring people to adopt SharePoint for knowledge management was a big pending layoff (RIF). Knowledge transfer was a real, immediate problem for them, and SharePoint provided a good solution. Takeaway: big layoffs drive SharePoint adoption! Slap that on the side of a bus and drive it around at the next conference!
• Hilarious quote: "You want to use a wiki? How much do you use the discussion lists you have on your site? None? Ok."
• As an aside: for the record, you are no longer allowed to use the term "wikis and blogs," as if they belong together. Small wikis are great. Wikipedia is great. Content tied to RSS feeds, which also allows comments, are great. Blogs that you can find on Technorati and can add to your subscriptions in Google Reader are great.  But in none of the 5 situations above, are you combining both "wikis" and "blogs" into one solution. You're not. Furthermore, nowhere are you even combining your intranet-facing content with your public-facing content. "Wikis and blogs" is a meaningless term, and it's growing ever more popular as the Enterprise 2.0 cloud blows in. I will talk about Enterprise 2.0 some other time; the summary is "part duh, part new good things, part money/hype/vendor."
• Also driven home: the fact that SharePoint implementations should not be "hardware + software + configuration + customization" only. It should include a great deal of effort on PEOPLE, PROCESS, and TRAINING. I'm with you there 100%. Training is always a good choice.
• Awesome nugget: when asked how to prevent sensitive or covered-under-regulation documents from being uploaded to SharePoint, Melvin's response was "what do you do for email?" Their inevitable answer is "nothing; we can't control email." The implication then, is that "oh, then why am I suddenly required to control SharePoint?" Melvin claims we can tell people that we will only hold SharePoint to the same standards to which we hold our email systems. I don't know about you, but this is an awesome "organizational hack" and I'm going to try it out.
• Job roundtable: it is clear that a) there's a lot of demand, b) there are at least three distinct roles for SharePoint, c) the "talent shortage" problem is only going to get worse. My advice: train up someone with no SharePoint-specific experience, almost as an intern-type situation. As you may imagine, this was a rowdy session and had lots of crowd feedback.
• Greg Galipeau - attended this intro to SharePoint site definitions development. What's fun is sitting next to a total newbie to SharePoint who is asking, "now what do I do to start a SharePoint project? Which project do I pick?" What was not fun was to realize that, while I was sleeping, WSPBuilder (my tool of choice) went out and added a bunch of features of which I'm unaware. So I'm out of touch. Also out of touch with: STSDEV. Also out of touch with: VSeWSS v1.2 specifically.
• Tony Byrne of CMSWatch, presenting portions of the CMSWatch report as his presentation. I want to talk about this one for a long time. I just want to focus on three things in his presentation:
• One: take a look at what you need, before deciding on SharePoint. I know a lot of us back into SharePoint and then look around, after having purchased the product, and say, "okay, let's try implementing <feature A>! I read it works great!" If you get nothing else from his presentation/report, it's that SharePoint is not equally good at everything. His quote was "it can do anything, more or less."
• Two: The other thing that's unique is that he focuses on the actual product we have today. This is where I think a big disconnect exists: even outside of marketing influence, people believe SharePoint will somehow get better, and assume that improvement is 'inevitable'. If you take away this second thing, just think for a second: what do we have, now, today? Not what we believe the product will look like in the future—what do we have now. Biggest example for me: Don't look at Silverlight 2.0 seriously, yet.
• Third: He claims that "SharePoint is a guild". Which is offensive, but, when you think about it long enough, yeah, it kinda fits. One quick rebuttal: if SharePoint's a guild (which I'm admitting), then it's the easiest guild to enter of all the enterprise systems. If you want to take something positive out of this, let it be: we still have a long way to go before picking up SharePoint is 'easy'. I could talk a long time about this, but will cut short.
• I want to do a separate post on his presentation. Summary: even if you're offended at times, he offers perspective, based on real research/investigation—which is something unique.
• Day 2: Sahil Malik presented on .NET 3.5 topics. I think I have an aversion to developing advanced customizations in SharePoint. This includes basically everything he presented, including hosting WCF in SharePoint 2007, using .NET 3.5 in SharePoint (requires some server config to enable .NET 3.5), and AJAX (Atlas). If I can see it working, even in a demo, I start to believe it can work; until then I'm skeptical. So, anyway, I now (as of ~11AM this morning) believe WCF can work in SharePoint 2007.
• Also interesting, Sahil points out you should use a single "big honkin'" (that's technical jargon for large) physical SQL Server, in conjunction with dev VMs. The more obvious/default choice would be to host your SQL Server instance on each VM; his setup combines all the SQL instances. He says this is best for performance reasons, which, now that I think about it, works.
• SharePoint for small businesses - actually I'm just writing this bullet point to apologize: I came in halfway through and zoned out for the rest. I blame my stomach, I was starving.
• Random quote: "if you don't have policies, you're up a crik."
• Implementing a PMO (and using SharePoint)
• This was an interesting talk because it focused more on the human factors than SharePoint. Something that should be obvious is that if you don't have the process down, installing MOSS or MOPS will not fix your process for you.
• Quote: "…PMO is not a project site [in SharePoint], it's not just rolling up all your project sites."
• Tip: access disparate data, don't recreate it
• Tip: "if I can see everything I need in one place, then … (handwriting is slow…general idea of "I'll be effective)"
• "Transparency" - sometimes there are fights because people don't want to be transparent. Ooh, I feel that one.
• Speaker prefers Lean, but wisely didn't use the word "Lean" anywhere in his presentation. :) He just said "The Toyota model"
• Fun quote for my PMI coworkers: "PMBOK is starting to get stale." This is fun because I can imagine their response.

I just read that, that was a lot of text

Wow, I need to learn to edit, bad me. Well, no way I'm going back and attempting to edit that behemoth; if you made it this far, then there must have been something good in there to keep you reading. Or you're a skimmer.

A+ would buy again

I do generally recommend this conference, especially to those of you in the area, as:

a) It's on a Friday/Saturday, which means you only lose one workday and one weekend day; (strikes a compromise between missing work and hating life)

b) The management track was excellent and had tidbits you can find nowhere else…well, nowhere else besides another conference or a $1400 report;

c) There was no "Intro to Silverlight" session.

Wednesday, June 18, 2008 2:02:02 AM UTC 

…which is very little. I will state for the record, that all this is public knowledge, primarily because I don't have the hookups to get some sweet sweet NDA action. Anyway, onwards.

The following is everything I could find (and ask others to find on my behalf) publicly available about SharePoint. There's not much we know yet.

Quick summary is: beta's coming soonish, MDM is a new slice in the feature pie chart; Enterprise 2.0; FAST search; claims-based auth; bunch of stuff they probably haven't announced. A lot of this has not been specifically promised for SharePoint 14, but may appear in vNextNext, or vNextNever. Who knows, I don't.

Onwards!

Early access - beta soon?

Office 14 TAP: Nominations are open

SharePoint 14 - available internally since February

Surprising/big-impact changes

Master Data Management

SharePoint may use claims-based auth

Thoughts on SharePoint and FAST search

Random SharePoint 14 tidbits

Enterprise 2.0 conference SharePoint tidbits

SharePoint Lists improvements

Ship date in 2009?

Transcript of Bill Gates' SPConference 2008 speech

SPConference 2008 writeup - Microsoft's strategy for vNext

MVP summit 2008 recap

Access Web Access: Speculation

The Bill Gates Interview

OOXML and PDF support

WinSuperSite Office 14 FAQ

64-bit only

The Future of Groove and SharePoint

Rampant speculation

Forrester's educated guesses

SharePoint vNext Rumors

The Next Version of SharePoint

Wednesday, June 11, 2008 4:46:27 AM UTC 

Today's fun challenge on SPOT the TYPO will be a piece of a SharePoint deployment script!

Let's bring on the challenge!

SPOT the TYPO!

(line breaks added for readability; ignore them) 

Still can't find it?

Maybe there's a problem with the image…let's try plain text!

stsadm.exe -o deploysolution -name MySolution.wsp –immediate -allowgacdeployment

Wait, I think I saw something!

Here's the image, enlarged:

I definitely saw something!

Now let me highlight the important bits. Enhance!

 

…enhance!

MYSTERY: why is this dash character wider?

Something is definitely fishy! Let's check out a hex dump and see what it says!

 

Hey, now that doesn't look like the standard ASCII character for a dash! That's not like a dash at all!  In UTF-8 encoding (as this file is encoded), ASCII characters should look like ASCII characters! Right? Right!

Dude, where's my cardash?

It turns out, it's the endash, "&#8211;"—heretofore dubbed "the script ruiner." Read all about it (and I mean all about it) in this incredibly detailed article, about the endash. I'm serious, check it out.

OK, WE GET IT

As is always the case with these "why did my character magically become some other character" mysteries, we can safely blame Word AutoCorrect. When we're typing up technical documentation in Word: blame AutoCorrect. Even when we're typing in Outlook, where we think we're safe: we're not safe, Word AutoCorrect lurks in the shadows! It's hiding under that huge ribbon, waiting to pounce! Ha-ha, those three dots just became an ellipsis! You didn't even notice!

Let's say we're looking at a complex Microsoft Knowledgebase article, oh, say #934838. Let's just say. And let's say that on this article, there are lots and lots of commands just begging to be cut-and-pasted into your favorite text editor, Notepad. Notepad is your favorite text editor because you're not crazy enough to attempt to install another one on the production server. And when you paste this into Notepad, everything looks great on the standard Notepad monospace font. It's a monospace font, think about it.

And then you paste the text directly into the Command prompt, which spits out the generic stsadm error, which is hilarious, because they put the error at the TOP and immediately kick off a 5-page command listing, so you have to dig for the original error message. And yes, the error message says "Command line error." And you're studying, and studying, and you have no idea why this isn't working!

LET ME TELL YOU WHY IT ISN'T WORKING. Because when you zoom in, REALLY zoom in, you'll see that the dash character is exactly two pixels wider! And boy, let me tell you—those two pixels make all the difference! Get out your microscope!

Zoomed-in SharePoint art

If you look at the hyper-zoomed-in picture from above, it almost looks like some sort of awful abstract art. Noticing this, I thought—might as well go with it! Allow me to add a few touches here and there, maybe work with the interplay of light and form and structure, maybe add some tasteful emotion words…and give birth to the worst SharePoint-themed abstract art ever:

As with all true art, each person will take away something different from this piece.

Reader challenge

I dare you, dear reader, to take on the challenge: can you create something worse? You can! You have it in you! Believe in yourself!

Tiny PowerShell footnote

There is a systematic way to find Unicode characters in your strings—just check the integer value of each character. Should be easy, right? I'd say so, yes:

Note I used the "%" shortcut instead of "foreach", and "?" instead of "where". Read it as: convert the string to a char array; for each character, convert it to an int; now filter these integers down to only those with a large (non-ASCII) value. The remaining ints are returned to the prompt, which displays them as best it can. There happens to be only one character today, which is the endash, "the script ruiner."

Anyway, the point is you can guarantee your string is ASCII if you make use of PowerShell's (.NET's) built-in Unicode support and write a simple script.

Monday, June 09, 2008 8:00:05 PM UTC 

When trying to convince others why I think PowerShell is hot hot hot, I find myself lacking examples. So, dear reader, I'm writing this list as a sort of "memo to self: don't choke next time and remember one or two of these things." Here goes.

Before the mega-list

If anyone is interested in a particular topic in more detail, let me know. To me, most of what I list below is either a) too broad a topic to cover properly, b) too narrow for anyone else to find it of use, and/or c) just plain obvious.

I can't tell what may be interesting to others; it's all a "solved problem" at this point to me.

Of all these things, since the following is a jumbled, unprioritized, stream-of-consciousness type of list, I'd like to point out that the things I find are most useful are:

a) Object model spelunking - during development, figuring out which function to call, testing out little 5-line scripts to see what happens—little things to help me gain confidence that what I'm writing will work. This is also the most difficult thing to express properly.

b) Visual Studio post-build task - it's just so easy to add and remove bits of script to help me do … whatever it is I need doing. Note this is not necessarily a replacement for NAnt or MSBuild—instead it's something of an informal post-build scratchpad.

c) Deep (focused) administration tasks - like getting the crawl logs programmatically, or getting a list of sites or features into an Excel-friendly format for further analysis. SharePoint allows deep access to its inner workings with APIs that it uses itself—this is unique among enterprisey-type systems.

Memo to self: Powershell + SharePoint

Ways I've used PowerShell + SharePoint:

• Object model spelunking. This includes
• The most commonly used SharePoint objects, e.g. SPSite, SPWeb, SPList, SPListItem. PowerShell's unique (to the .NET space) REPL environment is the best for exploring an object model. You get the full power of an object browser with all the benefits of an immediate window. I don't know how to explain this; you'll just have to try it sometime.
• Neat trick to help you write CAML queries. 1) Make a view that does exactly what you want. 2) Use PowerShell to dig into the view's SchemaXml property. Others have written Windows Forms tools to do the types of things I do in PowerShell.
• Digging into the SPLimitedWebPartManager class.
• Object model manipulations. This usually means really simple things that I do in a dev/test environment, like
• Changing the logo on a site to a different gif
• Changing the default master page property
• Changing lists to Hidden
• Changing fields to Hidden - there was a case where (for whatever reason) the ModerationStatus field became visible; it shouldn't have been; quick PowerShell script confirmed the fix.
• Deleting all items in a list. $site.OpenWeb("/awesomeness").Lists["awesomeness"].Items | % { $_.Delete() }
• Who cares? you may say. What happens if you have 1000 items in the list, that you're programmatically creating every time you run your integration test? Yeah. Yeah, you may need something like this.
• Comparing two fields in a list via PowerShell, looking for corruption in one of them. Amusingly, the problem was that one field belonged to a Content Type, thus was not visible on the list edit page, thus causing us MUCH confusion. Oops! Well, PowerShell was able to help us, just not in the way we thought. By the way, I used PowerShell's metaprogramming facilities to do a 50-property comparison using a one-liner.
• Enterprise Search administration. Includes:
• Getting a copy of the crawl log. Up until a few weeks ago, until the Enterprise Search team released a stsadm tool to do the same, this was quite novel.
• Swapping the "file inclusion list" into a "file exclusion list" in order to aid web crawling.
• Working with wikis - digging into a wiki library and extracting the page titles - it was actually somewhat lame, but hey, it worked.
• Ad-hoc querying of our farm's various sites; who's the owner, when were they last modified, etc.
• Listing the farm's current various peoplepicker settings, to help ease my confusion with their application. This was pre-SP1 before we had the really useful settings available to us.
• Comparing document templates (actual documents) in every library in the farm, comparing pre-migration to post-migration farm. It turned out, during the migration we "lost" two document templates. I'd like to give a big shout-out to SharePoint's WebDAV support, that was definitely a big help :)
• Copying a list using SharePoint's lists.asmx web service. Presumably I was doing some processing on the data, there's not really an excuse for just copying the list from one site to another.
• A different solution: used PowerShell as a "touch-free" solution to generating "reports" on a SharePoint server. I'm not particularly proud of technology choice in this case, but the results were good. This script involved a) pulling data from TFS, b) updating a custom list, c) creating InfoPath form data and "uploading" the forms. It wasn't the best choice.
• Listing all activated Features in the farm (well, actually, close: all sites in a site collection, the site collection itself, the web application, and the farm-level Features). Anyway, listing all activated Features in an easy-to-paste-into-Excel format.
• PowerShell as a Visual Studio post-build action
• Use it to deploy my project
• Use it to do a "fast deploy" - before building, just comment out the sections of the post-build script that we don't need to run.
• gacutil /i for quick code updates to the GAC.
• gacutil /u to ensure that when I run the TestDriven.NET test runner, it is running off of the most-recently-built project build, not off of whatever is sitting in the GAC. With this, I can guarantee nothing is in the GAC.
• Attaching to event receivers.
• Do an app pool reset.
• Warm up the app pool by programmatically visiting a page in the site.
• WANTED BADLY, MANY TIMES: a way to programmatically update a workflow association InfoPath form, WITHOUT having to go through the whole deployment rigmarole. If it's possible at all, I can automate it.
• PowerShell as a "manual setup" for integration tests. We're talking the "copy 100 files to the document library and see what happens" type of integration test, which is … let's just say difficult to achieve without some kind of artifacts. So, in other words, because I was too lazy to set up my integration tests the proper way, I used PowerShell.
• Doing name lookups from the people.asmx web service; checked ~400 names against the people list (and got back a rich list including email address, full name, job title, department, etc) in 5 minutes.
• $solutions = ([xml](stsadm -o enumsolutions)).Solutions
• You know all that worthless unreadable XML returned by stsadm -o enumsolutions? Well, with this one-liner, we have turned the useless into something powerful.
• Installing a Fab 40 demo site collection programmatically - it takes a sweet long time to finish, but the script is painless and perfect. Others have posted batch files to install the templates, but no one's attempted to build all the demo sites automatically. This is a good example of what PowerShell can do.
• Enumerating over customized/unghosted pages—all of them in the entire farm—and getting a list of all pages that are listed as uncustomized. This is more useful than what the PRESCAN.EXE log gives you.
• On a related note, attempted to run the "RevertAll()" or similarly named function to revert all pages in the farm to default. Again this was on a copy of production data. And no, it didn't work as well as I'd hoped.
• Use CabLib.dll to create CAB files, bypassing the horrors of MAKECAB.EXE.
• Tiny development-time administrivia, like getting the four-part assembly name of your project's DLL, and copying it to the clipboard—you need it to declare in your feature.xml file for your Feature event receiver. Here's a bunch of things I did.

Conclusions

• You don't have to use PowerShell. I'm sure many of you have solved similar problems with PowerShell alternatives, including
• SQL queries
• WinForms utility apps (check out CodePlex, you'll see what I mean)
• Console apps
• custom stsadm extensions (I'm talking to you, Gary LaPointe)
• NAnt tasks/MSBuild tasks
• Telling people "that's not possible"
• It's hard to nail down for what I use PowerShell specifically, as you may infer from above. Is "lots of stuff" an okay answer? No?

Friday, June 06, 2008 8:00:01 AM UTC 

I recently read Todd Klindt's post on dynamically generating filenames based off of dates, for the greater purpose of making daily SharePoint backups.

USING BATCH FILES.

While his solution serves the #1 purpose of all scripts—it works, and thus is an excellent solution—I'd like to show how to accomplish the same thing using PowerShell.

Task #1: generate a SharePoint backup in YYMMDD format

DOS:

stsadm -o backup -url http://awesomeness.inc/ -filename "AwesomenessInc - %date:~-2%%date:~4,2%%date:~7,2%.backup.cab"

PowerShell:
(line breaks added for horizontal readability)

Task #2: delete backups older than seven days

DOS:

forfiles /d 7 /m *.backup.cab /c "cmd /c del @file"

PowerShell:
(line breaks added for horizontal readability)

Conclusions?

• Both get the job done.
• Date formatting in DOS is disgusting. forfiles is terse, but there's no excuse for the date formatting (substring selection?) syntax. This is almost as nasty as the fabled LDAP bitwise-and operator (look it up, it's not a joke).
• PowerShell is far more verbose (at least today), but benefits us with improved readability.

Had I the inclination, I could have used standard procedural techniques to further express my intent in the code itself and further improve readability. E.g. instead of calculating the YYMMDD date inline in the string, I could have called a function to do that for me, or stored the value in a variable, i.e. either Get-TodayInYyMmDdFormat or $dateInYyMmDdFormat. So while I did an okay job readability-wise, I could have done better.

As for DOS: this is as good as it gets, readability-wise.

Aside: calling DOS commands from PowerShell

Final quick reminder: instead of all that PowerShell-specific nuttiness, we could have instead satisfied the "run this in PowerShell" requirement by prefixing the existing DOS commands with "cmd /c". So the DOS command

stsadm -o backup -url http://awesomeness.inc/ -filename "AwesomenessInc - %date:~-2%%date:~4,2%%date:~7,2%.backup.cab"

becomes the PowerShell-compatible command

cmd /c "stsadm -o backup -url http://awesomeness.inc/ -filename ""AwesomenessInc - %date:~-2%%date:~4,2%%date:~7,2%.backup.cab"""

Note that we're treating the entire line after cmd /c as a string in PowerShell, so we have to be sure to escape any special characters, e.g. the double-quote. While it's somewhat obvious when I encapsulate the entire DOSO command in the string, it is much less obvious if you fail to do so. PowerShell will attempt to 1) evaluate what is there, then, if it fails, 2) send that value to DOS as text. If you don't specifically tell the interpreter "hey, send this as text", you may be surprised by the results. An example of this oddness is something I've used before.

DOS:

stsadm | find "feature"

PowerShell:

stsadm | find """feature"""

Thursday, June 05, 2008 4:49:58 AM UTC 

Those of you looking for something to amuse you will be sorely disappointed. Reading over this post even bored me, and I just finished writing it! Yeah, this one's for the search engines to pick up.

By posting this, I've become the world authority on Cisco NLBs with SharePoint

…which is sad.

I'm posting this partially as informational content, but partially because I'd be interested to find out if anyone else in the world searches for it.

So here goes.

Introduction: All SharePoint Tutorials Assume You're Using ISA Server

This is something that's annoying, especially in situations where they refer to ISA as "NLB". The SharePoint admin companion book is one place I can call out as guilty…

So, I'll point out before you go looking elsewhere that unless otherwise specified, the other guys' tutorial assumes ISA. You've been warned.

What I'm doing: please critique via comments

Since there is no posted content anywhere else in the universe, I'm just going to say what we're doing. This sort of thing is incredibly dry, so we'll do this lazy and concise, bullet-point style:

• New SharePoint "health check" web application, running on another (hopefully firewalled) port.
• Anonymous access is on. This is important! I don't think it's possible to have our Cisco NLB attempt NTLM, so we have to fall back on "nothing".
• This is not the same IIS Web Site as our main web application.
• Double-check to ensure IIS logging is off for this site. You don't need the logs, but whatever, you make the call, bigshot. Maybe those logs are more valuable than I will ever know.
• This is not even the same content database. I created a new (incredibly small) content database just to run the health check.
• Created a site collection, and
• Underneath the site collection, created a new blank site.
• Enabled anonymous access on the site - full access to the site, so that our health check actually works.
• Health check URL: /_layouts/mobile/mbllists.aspx - this is the where we're redirected if visit the mobile (./m) page of any site.Page looks roughly like:
   
• Tell the health check to
• visit this page
• expect a HTTP code 200 response (i.e. "it's good"). I don't think checking the page content is important, but if you want, knock yourself out with that.

Why it's important to go through all this trouble

I will say we've had trouble with our NLB health check because we (at the time) had not set it up to test every dependency. Our first attempt, a static HTML page, didn't even run on the same app pool as our main SharePoint web application…so when the app pool started throwing Out Of Memory exceptions, the NLB health check remained blissfully ignorant. Footnote: we fixed this problem by implementing overlapped recycling; definitely check it out if interested.

Even later, when running a static page on the same app pool, we were still not testing the database connection (and/or AD authentication!).

We also didn't want the health check page to be a "heavy" page (e.g. the portal home page). I briefly experimented with creating a minimal aspx page, but this "mobile view" serves the same purpose—a light page that pings the database. And hey, it's built in!

Feel free to give advice

If I'm missing something obvious, definitely let me know. If not, also definitely let me know :) I just don't know, and I'm not sure exactly where to look, so…blast away.

Wednesday, June 04, 2008 4:15:51 AM UTC 

 

 

The answer awaits you below the fold…

.

.

.

.

.

.

.

.

.

halfway…

.

.

.

.

.

.

Did you know that some feed readers give you a text-based preview of each post, and thus, if you're attempting to hide answers in your blog post by using whitespace…well…let's just say that this little meandering path, this seemingly useless little discussion we're having right now, serves a purpose.

Continuing…

.

.

.

.

.

Horrifying, correct answer: your SharePoint disaster recovery plan is stored on your SharePoint site! Uh-oh!

One act play

I've wrote a full-length, dramatic re-enactment of this situation, originally a 600-page manuscript. Which was way too big, so I ran Word Autosummarize on it, 40 times in a row. What you are left with is below:

 

Um…branding?

Also, if you answered "the problem with this SharePoint site is that it looks like SharePoint," then you're technically correct, unless your SharePoint site already looks like hawaiianair.com, or is in fact hawaiianair.com. If you're not running hawaiianair.com, by now you've already been told that lack of branding is a 'problem'. So, anyway, good job on guessing "branding," which is so commonly the answer that you might as well throw it out there every time you're lacking ideas. "Um…branding?" is always a good idea.

Lesson learned: off-site backups

I'm not going to be any more specific, because I don't have any magic automation to eliminate this problem—I'm just saying do something.

Friday, May 30, 2008 8:00:07 AM UTC 

Before I start, I'd like to acknowledge the fact that having strong opinions about an obscure Windows command-line utility is so not cool. I'm with you.

Things I don't like about using MAKECAB.EXE:

• If you forget to set an obscure property in your DDF manifest, MAKECAB will silently and happily exclude large files, despite the fact that you explicitly included them in the DDF manifest. Not only is this bad error handling, but it also gives you a false sense of security. This leads to the most awful troubleshooting sessions, where you have to dig into every layer of the stack, before coming back and double-checking every layer. This aggravating troubleshooting session is followed by a brief eureka moment, followed immediately by intense rage.
  Obscure property to set: .Set MaxDiskSize=<<some massive bytecount, I'd go 1 billion plus>>
  • Fun footnote: most online samples fail to set this obscure property, including MSDN and several blogs who obviously copied their DDFs from MSDN, which is why this is such a huge problem. Very few of the online examples get this right!
• The MAKECAB.EXE SDK document uses Times New Roman as its font. That dates it at pre-1997, when we were all still using either Office 6.0 or Office 95. Awesome. This isn't a complaint as much as it is an observation—OLD!
• Diamond Directive Files! DDF manifests are awful and, in my limited needs, completely unnecessary! Whenever I need to build a CAB file manually (it's happened a few times), I fired up PowerShell and used a single CabLib Extract/Compress function call to do the same thing that your no-longer-necessary-but-you'd-better-get-all-the-syntax-perfect MAKECAB+DDF file solution did.  And I didn't even have to bust out the ancient Times New Roman-sporting manual! (see below for actual script; it's tiny)

Takeaways

• Obscure property to set in your DDF manifest: .Set MaxDiskSize=<<some massive bytecount, I'd go 1 billion plus>> 
• If you're using MAKECAB.EXE as part of your Visual Studio build process, switch to WSPBuilder! You're not losing anything by swapping out one post-build command-line EXE for another, and will gain MUCH more in WSPBuilder's functionality.
• Use the following PowerShell snippet (or use it as a starting point for your own improved version) to compress cab files:

 

Copy-pasty version:

function Compress-Directory ($dir, $cabFileFullPathAndFilename)
{
    [void][reflection.assembly]::LoadFile(
        "C:\Program Files\WSPBuilder\CabLib.dll")
    $c = new-object CabLib.Compress
    $c.CompressFolder($dir, $cabFileFullPathAndFilename, $null, 0)
}

Compress-Directory "C:\temp" "C:\sandbox\a.cab"

Wednesday, May 28, 2008 8:00:52 AM UTC 

And by "dingo" I mean "SharePoint's PRIME API via stsadm -o import/export", and by "baby", I mean "links to as-yet-nonexistent pages in a wiki library."

So, running that back, after using SharePoint's stsadm -o import/export, you'll notice broken links in your wiki library.

Wow, that sounded much more exciting with dingoes and babies.

What are as-yet-nonexistent pages again?

Let's do this by example. Say, you're filling in content in a team knowledgebase, the one I like to call "the 2008 edition". You get a new one every year, but that's another topic, let's not get distracted.

So you're filling in the Widget page, and as you go, you realize you will also need to come back later and write up everything you know about Sprockets. So, in your page, as you are typing the word "Sprocket" you surround it with the standard wiki tags [[Sprocket]].

Up to this point everything is fine. You finish what you're doing on the Widget page, you log off, you go home. Tomorrow comes and you've forgotten about the poor lonely Sprocket page. And the day after arrives, and the day after, and so on, and poor [[Sprocket]] remains a link to an as-yet-nonexistent page.

So that's what they are.

What's the problem with this again?

I'll be honest, you'll probably never encounter this problem. But: if you're migrating content using SharePoint's stsadm -o export and then stsadm -o import, your "babies" will be "stolen" from you. Your [[Sprocket]] link will be stolen and replaced with some abominable hardcoded anchor tag that points to an error page.

So be on the lookout: wikis don't move nicely via stsadm -o import/export.

Tuesday, May 27, 2008 8:00:52 AM UTC 

Update 2009-03-05 - I've changed a bit

I've been meaning to update this post for a while, and here we are. Others have surfaced with interest

• Francis Cheung [MSFT] has written a similar piece outlining the challenges of unit testing SharePoint. He did a better job organizing his arguments, so go read his article instead of this one :)
• The SharePoint Guidance sample project was released—it's the closest thing we have to the "ideal* way to build SP apps*," and it exists, and you can download it today. And they have unit tests, and you can look at them. Awesome!
• Andrew Woodward has started a series about TDD in SharePoint, I'm interested to see this develop further with more focus on how to make it work in typical* SharePoint scenarios.
• Tom Clarkson posted an interesting article about testing his workflows. I stay away from Workflow development as a kind of metaphorical water hazard, but it's good to see testing Workflows is possible.
• Eric Shupps has posted a long contrarian piece on unit testing SharePoint. Be sure to read the comments for the counter-arguments, which boiled down to "your sample is too small to unit test anyway," AKA it's all about context. Perhaps the most interesting thing is that BIl Simser replied in the comments (comment #3) about how he structures his SharePoint projects, something I'd be interested in hearing more about.

*footnote: defining "ideal" and "app" and "typical scenario" is an exercise left for the reader, or the commenters on Eric Shupps' post.

Meanwhile, I'm spending my time learning coding/OO fundamentals, which is an important step to doing real TDD. Practicing TDD at home has already changed the way I structure my SharePoint projects, even the small ones, even when I'm not doing TDD at work.

I still don't unit test my SharePoint projects, but on my next project I'll probably try to do full-blown TDD. I still won't attempt to test code that interacts with SharePoint, but at least I can structure my code such that I can write down, and test, my intent. In other words, TDD can't tell me whether the listItem.Update() function is the correct way to save changes to a listItem, but it can tell me whether or not I called the function that fully intends to do so.

If you're coming here from the JOPX post, note that I do want to do TDD, and I think TDD has great value outside of SharePoint projects, but. I hold firm that most of the challenges we encounter with SharePoint are either the fault of the framework, or the fault of our misunderstanding of the framework—and neither is addressed by TDD.

I also want to clarify that as a rule, I don't think SharePoint should be used to host massive applications. If you've ever seen Wayne's World, there's a classic bit where his girlfriend gives him a present:

"A gun rack… a gun rack. Shyeah, Right! I don't even own "a" gun, let alone many guns that would necessitate an entire rack. What am I gonna do… with a gun rack?"

In this case, TDD on my tiny SharePoint projects is the metaphorical gun rack. Also big props to myself for making that connection, I deserve an award for that.

Opinion piece

This is my opinion, and reflects my (lack of) experiences, personal bent, and traumatic experiences, collected together into something coherent. I am writing this 2008-05-19, and reserve the right to change my mind later. I want to change. Like Fox Mulder, I WANT TO BELIEVE.

Authority

I have minimal authority on this subject and do not believe my opinion is "better". I am not an expert on unit testing, and I am not a renowned SharePoint authority. I'm going to attempt to steer clear of making any statements about TDD, and will instead focus on the more concrete act of unit testing.

I love comments

Actually I wouldn't know if I love comments, because I've only got like 3 of them. I was pretty pumped when I got those 3 comments though; that was a good year.

I love explicit disagreement

If you read this description of the forthcoming SharePoint Best Practices book, you'll see what I'm attempting to do: I'm attempting to start the conversation (hopefully not finish the conversation) that will answer the question: how are we supposed to adequately test our SharePoint projects? So far I have only found two strong opinions. I'd like to collect more.

Also, this is not a plug for the best practices book; I haven't read it.

Disambiguation: unit testing versus automated testing

One common misconception I'd like to clear up is that unit test probably doesn't mean what you think it does. Whenever someone tells you they are doing unit tests, allow me to present the following handy fact sheet:

• Just because something is written in NUnit or MbUnit or xUnit.Net, does not mean it is a unit test. I think this is the #1 source of confusion—the name of the product has the word "unit" in it, and it's used for testing, so…unit plus testing, unit testing! That's what they're called! Hilariously, by this metric MSTest is the best-named automated testing suite. You heard it here first: MSTest is the best!
• TFS Web tests are not unit tests. Tests that involve WATIN or some other web automation tool are not unit tests.
• Anytime you're actually newing up an SPSite or SPWeb object inside your test method, it's not a unit test.

Now for a few facts:

• Unit testing is often confused for the broader world of automated tests, which includes both unit tests and integration tests.
• Unit testing is not easy in SharePoint.

At this point I'm going to be lazy and tell you to read up on the Wikipedia articles on Unit tests and Integration tests I just found by googling for 5 seconds. I'm lazy, ask someone else for a better explanation.

Opinions

Popular culture: no testing

When dealing with such a broad topic as SharePoint testing strategies, it's probably best to start with the default. Which is to say, nothing at all. By default, the out-of-the-box Visual Studio 2005 projects do not create a separate testing project for you to use for testing. Most developers will live with the defaults, which is to say again: nothing.

So if you're here, wondering what to do outside of the defaults, congratulations! You're already counter-culture.

Counter-culture: agreement on some testing, disagreement on specifics

(various): Integration testing is sufficient

Spence Harbar, MVP: Unit tests with TypeMock - this is the only page I can find from him on the subject.

Casey Charlton: Unit tests against Doubler-generated wrappers - Casey has written repeatedly about the pains of his unit testing approach, but the fact of the matter is, he now has a workable, no-TypeMock-required solution for unit testing in SharePoint. Check out the Doubler addin to Reflector, which is described as "useful when working with legacy code."

My Opinion

With the problems I encounter on a day-to-day basis, I don't think it's worth it to go out of my way to build unit tests. And the reason is as such: when I encounter a bug in my SharePoint application, it has nothing to do with my code, and everything to do with my interactions with the SharePoint framework. Let's do this by example:

• Spence Harbar on his TDD page mentioned specifically the awful quality of code in the navigation controls he's seen in the wild. His assertion is that the code quality is the biggest pain point. Unfortunately, when I'm attempting to build something, I'm worried less about the cyclomatic complexity or the C.R.A.P. metric (it's real, believe it), and worried more about which of the many portal-querying strategies to use. There's no way to know if you're using the right approach, unless you're read all the books, all the whitepapers, and all the blogs. And, yeah, don't look at actual control vendors' code via Reflector, because apparently they're doing it wrong too :) The major issue when building navigation controls is not traditional code quality, as much as a broad understanding of all the options presented to you by SharePoint's object model.
• In my most recent project, I encountered a race condition where the ItemAdded() event fired too early and my list item…did not exist yet. This was nothing I could control. Solution? What I affectionately refer to as the "try try again" pattern. It involves a while loop with a retry counter, a Thread.Sleep(), and a whole lot of swallowed pride. Feel free to complain about the solution, but don't complain to me, complain to Steven Van de Craen; I'm just thankful his solution works. This concurrency bug was out of my control and would not have been discovered (or fixed) by unit tests.
• Outstanding issue: I am intermittently faililng to update my item after the ItemAdded() event. There are several possible solutions; the two I will be looking into will be a) disabling event firing (shouldn't have been a problem); b) following this awesomely-detailed advice and replacing my ".Update();" call with a ".SystemUpdate(false);" call. Either way, proper testing would not have saved me from this pain; detailed (esoteric) knowledge of the framework is the only thing that could have saved me.

What I'm saying is that practicing TDD in my SharePoint projects would not help me improve my code enough to be worthwhile. In the meantime, I will use automated tests (mostly integration tests).

While I'm not especially pleased with it, I am resigned to avoid unit testing the difficult pieces of my SharePoint projects. I'll consider changing this stance in any of the following situations:

• I become a master ninja and TDD becomes more effortless to me than doing without;
• I'm working on a critical, high-impact, risky project and need any and all available methods of testing and program verification;
• I have a lot of code in my project (which begs the question: why?);
• I learn enough about SharePoint development that situations like the three above are no longer issues, i.e. I learn how to walk around the coding landmines and am confident with every interaction with the framework. (For now: BOOM)

Don't forget about your team

Another factor to consider is maintenance.  Remembering that by default no one builds automated tests, with some effort counter-culturists write integration tests, and only two people in the world practice TDD in SharePoint—it's a hard sell to say "hey, I want to try building this next SharePoint project into a structure that is so awful and alien, so overwhelming, that you will be paralyzed. I'm going to change the project into a gorgon, which, with a single gaze, will turn you to stone where you sit. You will be completely unable to function. Do you mind? This new method will allow me to capture up to 60% of the bugs. But probably a lot less than 60%, to be honest. TIA!"

Wednesday, May 21, 2008 8:00:52 PM UTC 

Update: I used the word 'connection' in the title of this post for a reason

To everyone who links to this post with the phrase "SharePoint is written in Python", please allow me to make it super clear: no it isn't. The commerce module of a spiritual ancestor of sorts of SharePoint's was written in Python. That was Site Server.

To make it more clear: SharePoint doesn't have a commerce module. Commerce Server does.

Sorry I felt I had to do this, but, so many people were linking to this post with the semi-twisted-truth ("Did you know that SharePoint was written in Python?") that I will risk ruining this fun little history lesson by spoiling the ending. Sorry; on with the show.

Fun update: Greg Stein has written an article expounding on all this: Microsoft Ships Python code…in 1996

Original Introduction

Before you ask, no, this isn't a sneak peek into SharePoint vNext*. Instead, we are going to visit the ghost of Christmas Past:

FLOSS Weekly 28: Greg Stein for WebDAV

* However, if you can hook me up with any kind of sneak peek into SharePoint vNext, I'll pay you, cash**. Or, as good as cash anyway: old SharePoint books. Those things are timeless! Your knowledge never expires! (see depressing footnote at end of this post)
** Contrary to what has just been written, we don't endorse crime

Know your history

This discussion will be meaningless if you're not aware of SharePoint's origins. This SharePoint history page has an excellent summary, and even more excellent SharePoint history diagram. Go go Visio skills, good job sirma'am!

Anyway, if you're interested, click through the diagram below.

Let's talk to Greg Stein

From FLOSS Weekly 28, at the 47:30 mark:

SharePoint! Python! Scandal!

To spell it out for you: Site Server, the product which eventually became "Tahoe", which eventually became SharePoint as we know it, was written (at least in part) in Python! Python! In 1996! In 1996, a predecessor to SharePoint was implemented partly in Python!

I didn't believe it the first time I heard it, but there it is. SharePoint's predecessor was for a brief moment in 1996, a pretty cool web application written in Python. Sounds a lot like 2008, doesn't it?

 

Attribution: the diagram is not mine, though the offensively obnoxious green text is mine.

Depressing footnote: learning investments

Somewhere around the ~38:20 mark in the podcast, they ask him "what is your favorite programming language" and he replies that he has been programming in Python for 13 years, and using Emacs for 22 years.

I'll leave you for a while to ponder the incredible longevity of his ("the other guys") platform.

To me this news is incredibly depressing—how long have my programming language/platform choices been valid? How long will my SharePoint skills remain useful to any extent? I don't want to be on the Microsoft learning treadmill forever if I'm churning major parts of the framework every 2 years. Can you imagine evolving your ecosystem for 13 years? Can you imagine learning your IDE in-and-out, and continuing to use it for 22 years? It's almost unthinkable in our (Microsoft) ecosystem.

Argh, I'll stop. 13 years! 22 years! And counting!

Thursday, May 08, 2008 8:00:07 PM UTC 

UPDATE 2009-09-01: People Are Apparently Still Finding And (presumably) Reading This Outdated Article

First: I'm sorry. In the post below, I took a fun little journey comparing SharePoint's built-in wiki with MediaWiki. After some first-hand pain with wikis and reading the comments I received on the post, allow me to flip-flop my position and clearly state: SharePoint 2007's built-in wiki will impede any attempt to build a real wiki.

To put this statement in context, your main problems when deploying a real wiki will be to get critical mass. This means, you'll want to get hundreds or thousands (or tens) of individual contributors on your wiki. And you won't want them getting frustrated with the software. You don't want them trying to figure out the 4-step process to uploading an image; you don't want them attempting to build an HTML table (or anything really) using the source view; you don't want them to lose data when someone 'deletes' a page to the Recycle Bin and no one notices for 4 months. Preferably, you don't want them thinking about anything but 'how do I best contribute to this group knowledgebase?' And SharePoint will get in their way.

So, my new statement is: if you need MediaWiki, or something like MediaWiki, invest the time and effort to do so. Don't try to build on top of the minimalist SharePoint wiki. But…

But, with the most basic of needs, and given the assumption you already have SharePoint, and that Integrated Windows Authentication works for everybody, SharePoint is your best bet. Once you start doing 'fancy' things like adding images or attempting to go beyond the built-in WYSIWYG editor, you'll experience pain. Even with some pain, SharePoint's a good choice for small wikis. The original article (starting in the next section) lays out this argument in heavy detail.

Second: The new SharePoint will be out soon, invalidating whatever points I made in this article. Consider yourself on notice: this article is about the 2007 version, folks.

Third: I'm really trying to stop writing these "op-ed" pieces that really don't help anyone. I don't think I introduced any valuable concepts here, just pushed a lot of hot air into the atmosphere. How does this article (and those like it) help? They don't. So, I'll stop.

Final note: I've also flip-flopped on Wiki syntax. It's not that bad after all. WYSIWYG is much more important than markup. And while you're at it, everyone just copy PBWorks, because they have a sweet setup.

Original Introduction

A while back I came across this direct comparison of MediaWiki, the wiki engine powering Wikipedia, and SharePoint's built-in wikis.

As you might imagine, the comparison favored MediaWiki. And, as I left an embarrassingly passionate comment rebuttal (in which I link directly back to this address), I feel compelled to write the following post.

Summary first: because I meander quite a bit

The summary is: you probably don't need MediaWiki's extensive featureset for your corporate intranet. Assuming limited needs, SharePoint's wikis are a great, no-maintenance, small-learning curve solution to the "informal team wiki" problem. You can do better feature-wise, but why would you want to? Plus, you already have SharePoint—this is a no-brainer.

Well, let's get with the details.

Wiki syntax: unnecessary

So, let me start off by saying I hate wiki syntax. Actually I need to clarify: I hate any markup syntax built on top of HTML, and that includes MediaWiki's wiki syntax, the FlexWiki syntax I've used, whatever syntax ScrewTurn Wiki uses (it's different from FlexWiki), and the perennial favorite, [BBcode] markup. I hate all this for two reasons:

1. It's annoying to learn a new syntax for bolding up a word when <strong></strong> works just fine in HTML thank you, and
2. It's totally unnecessary.

We are now presented with the question: but Peter, what alternative do we have?

Just use HTML

Just use HTML! Allow me clarify in the next section:

You can parse the input and allow only a subset of HTML

I'm not saying to throw wide the gates and allow everything. It's perfectly reasonable to parse input and allow only specific markup, negating any and all security concerns! And when I say "parse", I don't mean some kind of abomination involving substring searches or even regex.

So if someone types: "Hey, I like to <strong>bold me up some text!</strong><script><!— some nasty js virus bomb —></script>", you know what? You can figure out which parts are legitimate and which parts are no-no! Don't just give up and say "oh forget it, let's just change the <b> to [b] and make our 1 billion users learn this new, inferior syntax." You know what? YOU CAN FIX IT, DON'T GIVE UP! 

It's possible; look into it—it's better than inventing Yet Another Markup Language. [[Insert your own awesome YAML joke here!! Ha ha, good one!]]*
* for the purposes of this bit, YAML stands for "Yet Another Markup Language"

Back to wikis

So with Peter's Fundamental Law #14 ("Don't invent yet another YAML") out of the way, let's get back to the comparison.

It's all about context

What are you looking for in a wiki—are you attempting to build "wikipedia for my global enterprise"? If so…I don't actually believe anyone's attempting to do this. Leave me a comment if you are, and if you have implemented MediaWiki in a corporate environment to accomplish this goal. There probably are people attempting to do this, which is fine. I haven't met them, but I'm sure they're out there. Maybe.

But I'm here today to tell you that most of you don't need MediaWiki. You may think you do, but if you actually go out and talk to some of your users, you'll find out: you don't. Your corporation is not filled with anonymous trolls—the trolls are still around, but [[insert your own boss joke here]]. Ow, my sides!

Anyway, MediaWiki is built to work with the semi-anonymous internet, and on a massive scale. You, on the other hand, have a grand total of 30 people in your enterprise who are clued into the fact that the mouse sitting on their desk has a right mouse button, and have a solid understanding of how a folder can contain other folders, and files. There's 30 of you.

The corporate intranet is not the same as the raw, unfiltered wilderness of crazy that is the internet.

What crazy world do you live in anyway

Let's just state this as a fact: your IT department does not have linux admins. Or if they do, they're hidden so well you may not recognize them. The point being: when you ask someone to install MediaWiki, you'll be asked: does this thing run on Server 2003, or does it require Server 2008? Um, gusty* gibbon or newer? That sort of answer will not register; please try again.

Oh, and while we're talking corporate IT politics, you may want to avoid using the words "Open Source" or "free" or "third-party" or "custom" or pretty much anything. Also, try and somehow pretend MediaWiki is made by Microsoft. If you say the wrong word, or fail to say the correct ones, you may alert your IT group to the fact that this software is unapproved!

* this is hilarious, think about it

Who becomes the admin?

Something else to keep in mind: if IT actually goes off and actually installs the thing, how are you ever going to get access to it? If you haven't learned by now, the easiest application to support is the one with the fewest users. And a good approach to keeping your user count down is to prevent them from crowding in too much in the first place. Zero is the best number!

SharePoint Time

Let's get down to business: actually comparing SharePoint's wikis to MediaWiki, head to head. I'll take the SharePoint perspective:

SharePoint PROS:

• SharePoint wikis take a grand total of two minutes to install and configure. Two minutes.
  • Infrastructure concerns, like high availability and security, are covered.
  • As Don King would say, it's "supportatudinous!" IT supports it already! It is backed up like the rest of your SharePoint data! You'll get upgrades for free!
  • You didn't even have to fill out the 8 page online help desk form, followed by a 10 page project request form, followed by a 3 page departmental project request form, followed by 2 security forms, followed by a third "surprise!" security form. You just made the wiki, all by yourself! Like a grownup! It was so simple I didn't even have to use fancy words like "provision"!
• SharePoint wikis use Integrated Windows Authentication*, emphasis on the word Integrated. The point is: seamless security. Footnote: security is configurable, maybe you're not using Integrated Windows Authentication specifically …but you should be.
• SharePoint wikis have minimal wiki syntax, instead relying on the rich text edit control. What's the syntax to bold up some text? Oh yeah, click the bold icon. Awesome. *tiny footnote: many wiki implementations have a WYSIWYG edit control, not MediaWiki specifically.

SharePoint CONS:

• SharePoint wikis are relatively feature-thin. I'm totally okay with this, but I'm sure the grizzled internet wiki veteran will be angered by, for example, SharePoint's extremely easy-to-delete pages, and lack of discussion pages, etc.
• SharePoint wikis are lists, and as such, should probably remain smaller than 2000 items. Yeah, there's a whitepaper on the subject; after too many items, performance of operations on a list slows to a crawl.
• SharePoint costs money, somewhere down the line—either for SharePoint itself, or for SQL Server, or just the server license, SOMEWHERE down the line, you're paying.

Intranet NOT Internet

If there's one thing you get through your skull—actually, if there's one thing, let's make it Peter's Fundamental Law #14 ("Don't invent yet another YAML")—okay, let's try for two things. If we have room for two things, the second one should be: the intranet is not the same as the internet. When comparing wiki engines, keep this in mind! The intranet is not the same as the internet!

Monday, May 05, 2008 8:00:29 PM UTC 

I just fixed an odd 403.14 server error on my Server 2008 VM, and what was the problem? My application was set to the "DefaultAppPool" app pool. The cause? "DefaultAppPool" doesn't exist, and yet my web site is somehow set to run on "DefaultAppPool". "DefaultAppPool" didn't event exist at the time I created the web application in IIS, much less now. By all means this should not have been a problem—IIS should set newly provisioned web sites to an existing, non-imaginary app pool—yet it was a problem.

So, advice: don't delete the default app pool, even in the new hotness that is IIS 7 and Windows Server 2008. You certainly can do so—I'm not saying it's impossible to get along without the default app pool; I'm just saying that you'll have surprises. And not the "congratulations, it's the prize patrol" type surprises, more like the "congratulations, your vehicle has been towed!" surprises.

You don't have to take my word for it—instead listen to Joel Oleson (talking about IIS6), and I quote:

Friday, May 02, 2008 8:00:26 PM UTC 

Pop quiz, hotshot: how do you double your SharePoint farm's searching speed?

(scroll down for the secret answer! If you have a large or otherwise high-resolution monitor, pretend you have tunnel vision!)

.

.

.

.

.

.

.

.

.

Do you need a hint?

.

.

.

.

.

.

.

.

TRICK ANSWER: Switch to an alternative browser! Instant speed boost!

I'm not trying to start a fight

I've been working with SharePoint's search recently, and have come away duly impressed by its capabilities. Today I'm highlighting yet another positive aspect of SharePoint's Enterprise Search—the fact that it is incredibly fast, and faster than you probably think—and at the same time, maybe stomping a little on IE. Just a little.

The point is, a lot of the time you spend waiting for search results on a SharePoint portal is spent by IE rendering the page, not by the SharePoint server. So, okay, what does this mean, and why do I care?

Takeaways - what to do about it

• Keep in mind, as an option, that the SharePoint search web service exists, and is a perfectly good way to serve up search requests on your other (non-SharePoint) portal. And it's fast.
• If you're demoing SharePoint, and especially as a Web Content Management tool, then by all means use the fastest browser available. If you're trying to show off SharePoint's capabilities, show them how fast it runs a search! But don't settle for IE's speeds. SharePoint is unfairly perceived as an "always slow" application, and we need to try and break that perception. Yes, it can be slow; no, it doesn't have to be slow. It can be quite snappy, believe it or not.
• Note also, that you don't need to load most of what's on a SharePoint search results page. I've been digging into a MS-PL (thank you!)-licensed Search Center branding project on CodePlex, and I'm amazed at how much cruft we can remove without losing anything. It should go without saying, but: the more you remove, the faster the page is served up, and the faster the browser renders. Win-win! Trim that search page! Take a chainsaw to it!

Thursday, May 01, 2008 4:34:40 AM UTC 

Update 2009-09-01

UPDATE: I no longer use the search page; it's in a sorry state of disrepair. The Technet forums search is broken, but wherever they moved it to, the real Technet forums search is still uniquely useful. Google Groups search has dried up as the traffic has moved to Technet. And, for whatever reason, my Google Reader search hasn't been as useful of late.

In any event, I'm going to eventually get rid of this page; if the page is somehow missing or broken, consider that your notice.

Original Introduction

I've updated my hotter-than-hot SharePoint search page, and let me tell you: it's hot!

Here are four reasons my search page is hotter than ever:

1. I've added Google Reader search to the mix! If, like me, you learn most about SharePoint from your blogroll, then it shouldn't surprise you to discover that searching Google Reader will return the absolute best results. Try it for a while and you'll agree! If you don't have a blogroll to search, borrow mine!
2. I've swapped the newsgroups search over to https://groups.google.com/ and instructed it only to search the microsoft.public.sharepoint.* groups. Google Groups search works more consistently (as in uptime) than Microsoft's Communities page search—bottom line, I can't trust the results from the MS Communities search page.
3. In-browser HOTKEYS! Believe it! They even work!
4. My page loads instantly! If you're doing the math, comparing this to any other page is a divide by zero error! NaN! My page is NaN faster!

Mission statement

The SharePoint search page is something I created for my personal use, and which I use daily, to search for SharePoint-related content, whether I'm looking for broad overviews, deep technical solutions, or looking for similar cries for help while troubleshooting.

While most people are satisfied to search Google or Live.com or whatever, I am not. Were it in my power, I would combine Google Search (or Live.com search; it's not bad)—anyway, I would combine whatever search with the complete Technet Forums content and complete Usenet archives; then rank the good weblogs higher; then somehow crawl MSDN's buried whitepapers. And then be done with it. That's my dream.

Until that magical day, however, the SharePoint search page will have to suffice.

Get at it!

http://www.pseale.com/SharePoint/search/

Monday, March 17, 2008 4:48:03 AM UTC 

I constantly encounter developers that just flat-out hate SharePoint. Which is totally okay, because it has warts, for sure. An easy comparison, as an example, is versioning your database. Let's do this:

• SQL Server: you can use Red Gate's SQL Compare utility, or Microsoft's DBPro SKU in VSTS, to get a diff that you can run at the time of upgrade. Let's call this a "mechanical" solution to the problem.
• SharePoint: any serious upgrade of your in-production content requires a full (manually performed or manually coded up) data migration.

To say the least, SharePoint provides some easy targets.

Developers always point out how much they hate SharePoint in hopes that I will be able to tell them why they're wrong, or at least be able to give them some extra perspective as to why I'm okay with SharePoint. We're not going to get into the full "why use SharePoint at all" question for now—I don't think I'm qualified to give the answer to an unknown audience in written form. What I will give instead is my throwaway response:

"You lose control with SharePoint."

Hopefully by the end of this post, you'll get an idea of what I mean by lose control.

From the mouths of Agilistas

I was most fascinated by this "Position of no power" presentation from the Agile Toolkit podcast series, apparently recorded at Agile 2006. Yes, 2006; that's not a typo.

Anyway, I was fascinated by a story told at 29:27 of the episode (transcribed below is 29:27 through 31:30):

 

What fascinated me most about this story is that a) following standard estimation practices, developers gave an estimate; b) customer declared we cannot cut business value; c) something magical happened, d) developers (legitimately!) reduced their estimates.

What happened in magical step c above? What does this have to do with anything? Let's roll onward and find out!

From the mouths of SharePoint experts

Let's go at this from another (more direct) angle. Check out DotNetRocks #268 with Vishwas Lele, and DotNetRocks #307 with Sahil Malik. They both start off by describing the fact that you get a great deal of value out of SharePoint if you are allowed to give an "80% solution". I.e., this may not be exactly what you wanted, but it gets 80% of the job done, and it took me three minutes to implement, while you were discussing who has the A for setting up teleconferencing for the next meeting. It's already done.

They both agree that this magical place is where you gain value out of SharePoint as an application development platform.

Take it from me

I'd like to draw together these two facts:

• By default, developers will attempt to implement exactly what you ask for. By default, customers dictate precise technical solutions.
• Developers and customers require something magical in order to get them to think creatively to craft solutions that solve the core business problem.

From the Agile 2006 presentation above, in order to reach this magical state we need what amounts to a pressure cooker: squeeze, and eventually you'll get toothpaste. But as she mentioned, they had to go through the stubborn phase first. This sort of thing did not occur naturally.

And now for SharePoint. It is my proposal that with SharePoint's wealth of built-in features, we are constantly attempting to fit the customer's "first pass at requirements" into SharePoint's existing featureset. If we don't find a fit, and if we don't find a solution, then we haggle—maybe they can change their human process, just a little? Maybe the formatting of that email alert doesn't matter so much? Maybe we don't need to format the web part precisely the way they asked? Maybe it's okay if the SharePoint site looks like a SharePoint site?

With SharePoint, we (both we the developers and we the customers) are forced to think creatively, and as such, are given a great competitive advantage over slimmer frameworks ("slim" like vanilla ASP.NET; yes, I know how ridiculous that sounds). This is my perspective. With SharePoint, you lose precise control over your technical solution, but you gain a different type of flexibility—you are thinking creatively, "out of the box."

Wednesday, February 27, 2008 10:00:17 PM UTC 

"Learn at least one new language every year."

    -Andy Hunt and Dave Thomas, The Pragmatic Programmers

For a while now many of us in the .NET space have complained of the never-ending torrent of new technologies and frameworks coming from Microsoft. While excited by the potential benefits (and yes, the shiny bits), we're so inundated with NEW that it's impossible to keep up. Something has to give.

Arrival stage left: the Pragmatic Programmers, circa 2000/2001, who inform us in their "Your Knowledge Portfolio" section that in order to be well-rounded developers, in addition to the learning we do related to our job, we need to set a goal to learn at least one new programming language every year.

What's unfortunate about this whole situation is that I actually think this advice is harmful.

One language a year: unattainable

No one learns a language a year, year after year. Except Ted Neward, and let's face it, he's nuts, and therefore doesn't count. So why should we set it as a goal? So we can fail, unless we're Ted Neward? GREAT.

Now I'm depressed

Why set unrealistic goals for ourselves?

Why a year?

Why not 18 months? Am I still broadening my horizons if I take 18 months to learn Lisp? or 10 years to learn C++?

Alternative to '1+ language/year'

Instead of hyperfocusing on '1+ language/year', why not take other advice offered by the Pragmatic Programmers on the very same page? I particularly like the way they present learning as an investment [the following is paraphrased]:

• Invest regularly
• Diversify portfolio
• Balance portfolio for risk
• Buy low/sell high (get in on technology early)
• Portfolio should be rebalanced periodically.

See! All this is useful, spot-on, helpful advice! And what's more, I don't even feel a sense of crushing despair!

Generalizing specialists

Let's work this from another angle: on this page Scott Ambler discusses what it means to be a generalizing specialist. The summary, by example, is that instead of sticking your head in the sand and learning SharePoint Designer 2007 to the exclusion of all other things (i.e. becoming a 'specialist'), you make a conscious effort to learn things that are NOT related to SPD 2007, however lucrative it may be at the time.

Nowhere on the description of generalizing specialists is there a mention of yearly language learning quotas.

So what language are you learning, Peter?

I'm going to make a conscious effort this year to learn C#. Wait a minute dude, I'm supposed to know C# already! Well, sure, yeah, but I would like to, you know, REALLY know C#. I would like learn C# such that I'm effective with it—such that when I read Bill Wagner's Effective C#, I'm not surprised by any of his 50 discussions of C# programming. Think about it.

I want to learn all the C# 3.0 features, aside from LINQ. I don't even honestly know if there are other 3.0 features, oops. Wait, extension methods. Awesome, I didn't totally forget.

I'd like to learn C# to the point that the very sight of Java code disgusts me. I want to see an Eclipse window and feel the physical urge to puke. That's how awesome at C# I want to be.

I'm not going to promise I will be C# dominant by EOY 2008. It may take longer. But I will work at it, on a regular basis. And I will get better.

And I think this is all the Pragmatic Programmers ever intended in the first place: pick a language; learn it and gain some perspective; pick another language; absolutely don't stop learning after your first language; don't take too long to learn the new one; constantly learn. There we go; that wasn't so bad.

Monday, February 18, 2008 2:00:28 AM UTC 

This announcement will be quick! Quickly:

I've posted a snapshot of my SharePoint blogroll (266 blogs; updated as of 2008-02-17). If you are interested in drowning in an ocean of CAML, governance, stsadm commands, browser configuration, branding, workflow, the other kind of workflow, the third kind of workflow, end-user training, InfoPath, SharePoint Designer, IE client settings, product announcements, third-party tool announcements, freeware announcements, CodePlex project announcements, whitepaper updates, service packs, hotfixes, beta programs, early peeks into SharePoint vNext, and oh yeah, some coding topics, well, THIS IS FOR YOU!

Download my SharePoint blogroll

Or you can just subscribe to Steve Pietrek's blog

Or, you can just subscribe, bookmark, or otherwise start visiting Steve Pietrek's blog, and let him do the sifting/aggregating on your behalf.

Monday, October 15, 2007 12:00:13 PM UTC 

I find it difficult to explain why I bother using PowerShell with SharePoint; it isn't easy to find a killer example. What I'm about to present is also not a killer example—I just figure that, after a while, I'll wear you down with enough mediocre examples that you'll be convinced. Or tired enough to give up resisting; either is fine with me.

Today's mediocre example

Introducing your team's new knowledgebase! To help everyone get started, I've already created the site and a wiki library (wiki libraries are not exactly the same as wiki sites; the differences are meaningless in the context of this discussion).

We're going to call this wiki library "Yet another soon-to-be abandoned knowledgebase", in an odd naming coincidence that I'm sure has no basis in reality. Everyone I know has plenty of time, incentive and managerial support to document all critical and common processes in an organized and thorough manner. I'm sure your experience mirrors mine.

In the screenshot below, you'll note we have created our knowledgebase.

 

 

Looking at the list view of this wiki library, you'll also notice that our team members have done their part, quickly and efficiently creating articles. Whoa there—watch your back, Wikipedia!

 

Looking at individual entries (like the one below), you'll notice that each entry begins with its own specially formatted title. We have all been so good as to consistently apply this convention across the entire wiki.

[Because I can't leave this alone, I'll note that no, those aren't real keys. Or if you find out that they are real keys, dude! DUDE! Let me know that I'm sitting on a gold mine here!]

 

Only one thing remains: now that everyone is finished with their articles (and let's face it, once the initial push is done, this knowledgebase is as full as it's going to get), you still need to create the wiki home page! Now how would we go about performing this repetitive task against hundreds, or thousands of regex-matchable wiki pages?

Enter SandmanPowerShell

Did someone say PowerShell? For the skeptics, this is where you make the "when all you have is a hammer, everything looks like a nail" joke. Go ahead; I can take it.

Comments before the code

The following code generates an unordered list of all the wiki articles, with the final goal being me pasting this unordered list directly into the home page. This admittedly is not the most exciting or shocking of PowerShell's uses.

The point is: I wasn't typing this data manually; and I wasn't attempting to use some other semi-automated, semi-manual method of get this data into the wiki home page; and I wasn't writing XSL transforms to display the data dynamically. Actually, XSL sounds like a pretty good idea, now that I think about it.

PowerShell code

Without further ado:

 

The Grand Finale: did it work?

You be the judge.

Emitted HTML (I pasted this directly from the clipboard):

<ul>
    <li>[[How To Use This Wiki Library]]</li>
     <li>[[Home]]</li>
     <li>[[Previous knowledgebases|Links to all of our previous knowledgebases]]</li>
     <li>[[Possibly incriminating information|Software activation keys]]</li>
     <li>[[Copy-Pasted Email 1|Re: Re: Fwd: Fwd: contact info]]</li>
     <li>[[Copy-Pasted Email 2|Re: Reboot the server?]]</li>
</ul>

 

Emitted HTML put to use:

Ok, well, it worked. Even for larger than average knowledgebases like my example above, five items aren't enough to justify all this. But, what about for 20 pages?

30?

50?

100?

1000?

etc?

ET CETERA?

At what point does PowerShell become too awesome an option to deny?

I'll wear you down, eventually.

Take-aways

NUMBER ONE: PowerShell is awesome.

NUMBER TWO: Even if you have no intention of using PowerShell, at least realize that through some informally-developed solution, you can work with SharePoint Wiki data programmatically. Whether it's a console application you run on the server, or a Feature you staple to the wiki's Actions menu, or some other Rube Goldberg-esque device, the point is: you can work with SharePoint wikis programmatically.

A short word about wikis (and hackiness)

The example above does not mirror reality; my real-life PowerShell script was much uglier, much hackier, and took almost no time to develop. In this example, I also skipped an intermediate Excel step (yes, Excel). The hackiness of the Excel step was such that it was unfit for publishing.

As for wikis: I'm aware that the wiki home page shouldn't necessarily point to every page in the wiki. I will say this definitively: there is no harm in linking to as much as possible off of the main page, in whatever form, whether it's an unordered list or otherwise.

Friday, October 12, 2007 6:04:32 AM UTC 

A while back I came across this entry which surprised me: Firefox supports automatic NTLM authentication.

Now, you may be thinking, when is this useful?

• Many SharePoint sites (note: not all)
• Other ASP.NET sites that use NTLM authentication
• IF YOU ARE TIRED OF SEEING THIS LITTLE GUY (image below):

 

A visual walkthrough 

 

 

 

 

 

 

 

 

 

Please note that the trusted-uris list does not include any protocol prefix, i.e. you don't type http:// anywhere. Please also note that each site is separated by commas with no space in between.

Why Firefox for SharePoint development

This brings many of us to the question: but Peter, why do you bother with Firefox in the first place? I have two major reasons:

1. Web Developer Toolbar
2. Firebug

These two Firefox addins help immensely with various web design tasks (e.g. CSS detection/changing on the fly/tracing to source; resizing the browser window to emulate a smaller monitor; a pixel ruler; etc!). Internet Explorer has no comparable equivalent to many of the features these two addins provide. The bottom line is I absolutely need these Firefox-only addins.

Internet Explorer's place

By default I use Internet Explorer for SharePoint, only switching to Firefox when I need to use one of the tools mentioned above.

This post shouldn't be interpreted as "Peter uses Firefox all the time for SharePoint development," because I don't.

Wednesday, October 10, 2007 5:10:25 AM UTC 

A while back, Ken posted this question asking "why are all web parts on a page returned as ErrorWebPart objects when I access them through the SharePoint Object Model?" And by "a while ago", I mean "sometime in the year 2006."

I am proud to say that I have the answer! Ken, today is your lucky day! Hope this answer comes in time!

The scenario

SharePoint migrations are a prime time to do a lot of cleanup. While this cleanup may take many forms, the specific form we are interested in today involves clearing out specific Web Parts from every page in the database.

Assuming our users permit it, we can close, delete, or even add web parts to sites, wholesale. This has the potential to be awesome, and by extension, we can be awesome as well.

Unfortunately, one of the issues you'll encounter with migrations is that (for whatever reason) you may not have all the original web part assemblies available to you on the new server. However, SharePoint expects to be able to use that assembly, and will stubbornly retain all web part references in individual pages on your site. What happens, you may ask, to these web parts already embedded in hundreds (or thousands) of pages on your site? They appear as ErrorWebParts.

It is appropriate to see ErrorWebParts where there is, in fact, an error—that is, when the ErrorWebPart describes a truly legitimate error.  However!

The problem

The problem is that we see everything as ErrorWebParts, even things that most definitely are not in any way broken!

Specifically, when writing code to access web parts on a page, we run into a curious problem: some pages return a collection filled only with ErrorWebParts. If you have 20 web parts on a page, you will see 20 ErrorWebParts.

This behavior does not occur 100% of the time, and it's not always the same sites that return ErrorWebParts. Sites that initially return a broken ErrorWebParts collection may work properly the next time around.

So what's the deal?

The answer—well, enough of an answer

I don't know why exactly, but I've narrowed the behavior down to this: errors only occur on pages that have not been visited recently.

Let me state this again for clarity, by example:

If you have visited a web page recently (say, http://moss.example.com/default.aspx), then attempt to query its Web Part collection, all will be well.

If, however, you have not visited a web page recently (say, http://moss.example.com/abandoned_old_site/default.aspx), and attempt to query its Web Part collection, you will see all ErrorWebParts. <- THIS IS THE BUG.

The appalling solution

The way to work around this problem is to guarantee every page has been visited before you access their corresponding Web Part collections. Think about it.

If you're not sick to your stomach yet, maybe it hasn't sunk in: you're going to have to programmatically visit each page first, before working with their corresponding Web Part collections. This is like cranking a chainsaw—you can't saw until you've cranked the engine.

Actually the chainsaw is a bad metaphor, because you have to crank the chainsaw, then maybe prime the engine with a few squirts of gasoline, then crank the chainsaw again, then adjust the choke, then finally, crank the engine successfully. Then saw.

The point is, you a) do some sort of chainsaw preparatory work, then b) get with the sawing, in strict order.

Alchemy

At times SharePoint feels like alchemy, and today's post most definitely illustrates one of those times. I don't think I should have to use a chainsaw analogy to describe an object collection of any sort. We can call this Peter's Twenty-First Law of Software Engineering Quality: chainsaw analogies may be an indicator of bad code smell.

Tuesday, October 09, 2007 4:43:48 AM UTC 

And to clarify, this is a good thing!

Maybe you haven't all heard the news: Microsoft is developing a new ASP.NET MVC framework. Go check out the original story for details. I'll cut-and-paste the featureset directly lifted from Jeffrey Palermo's entry:

ASP.MVC framework, summarized

• Natively support TDD model for controllers.
• Provide ASPX (without viewstate or postbacks) as a view engine
• Provide a hook for other view engines from MonoRail, etc.
• Support IoC containers for controller creation and DI on the controllers
• Provide complete control over URLs and navigation
• Be pluggable throughout
• Separation of concerns
• Integrate nicely within ASP.NET
• Support static as well as dynamic languages

If that was too much for you to read, allow me to attempt to summarize this visually:

 

Alternately:

 

Lastly:

 

Now I don't want to sound like I'm going overboard with pre-release enthusiasm—this announcement is significant not because we're getting "new stuff." Microsoft has, uh, announced new stuff before.

The key difference with this announcement is, as far as I know this is the first time Microsoft has released something new that bears absolutely no resemblance to FrontPage Server Extensions.

If that last sentence just made you angry, sorry: that wasn't my intent. Let me try to approach this from a different angle. I'm going to attempt to say the exact same thing in a different way; here goes: for once, we are not being treated to a new data grid, or a new designer, or a new UI stack (well, we are, kind of), or a new way to talk to databases, or another remoting/marshaling/service framework. We're not being treated to demos that tell us we can build applications "without a single line of code."

Maybe you haven't seen that demo before. You'll know when you see it.

A third way to describe this is that Microsoft is, for once, moving away from the smart UI approach to software development.

Take your pick of descriptions; whichever one offends you the least is the one I meant to say.

Authority

I'm not a mega-expert in a general sense, so I don't want you to quote me on any of this.

SharePoint

"Well Peter," you may ask, "where does SharePoint fit into all this?"

First, I will again point out that I'm not a mega-expert, so don't attach too much weight to my opinions. With that disclaimer in mind:

SharePoint is heading in a separate direction. While with this release the vanilla ASP.NET stack is focusing on quality, SharePoint is still busy adding features (and if there's time, writing some documentation). While vanilla ASP.NET is essentially deprecating portions of its framework, the SharePoint framework grows bigger and bigger.

What makes this such a complex subject is that SharePoint development cannot be summarized as easily as traditional imperative programming. While SharePoint allows for what is known as artifact development (loosely defined: stuff you change in Internet Explorer or FrontPage), SharePoint also exposes a deep object model, a stack of web services, and an underlying XML schema for everything (yes I mean CAML). This mix of web services, API and XML is referred to as assembly development (let's just call it "programming").

Dealing with SharePoint primarily as a programming platform, on par with "build it from the ground up" frameworks like ASP.NET, is certainly my goal. But even once I think I've mastered all there is to know about SharePoint; once I've built my own collection of tools and scripts to make all aspects of SharePoint development tolerable; once I have a sufficient library of samples from which to borrow on any project; once I can effectively answer the questions like "how do we patch an in-production SharePoint application"—at that point, SharePoint development at its pinnacle will still not resemble the new ASP.NET MVC framework. Let's do this by the numbers:

SharePoint's "software engineering quality" scorecard (as compared to ASP.NET MVC's new features):

• SharePoint: nope! Natively support TDD model for controllers.
• SharePoint: nope! Provide ASPX (without viewstate or postbacks) as a view engine
• SharePoint: nope! Provide a hook for other view engines from MonoRail, etc.
• SharePoint: nope! Support IoC containers for controller creation and DI on the controllers
• SharePoint: nope! Provide complete control over URLs and navigation
• SharePoint: N/A Be pluggable throughout
• SharePoint: nope! Separation of concerns
• SharePoint: too vague to judge Integrate nicely within ASP.NET
• SharePoint: who knows Support static as well as dynamic languages

I don't want this post to sound too negative regarding SharePoint, but it's probably too late, what with all the red marks above. Unfortunately this is the truth: if you're doing SharePoint development, you should realize that "high quality code by default" is nowhere in the SharePoint playbook. SharePoint provides a lot of benefits; it's just that "designed for software quality" isn't one of them.

I swear I'll get positive and gung-ho on SharePoint and post all about it here.

Alchemy

One topic I'd like to work on later is the feeling I get sometimes that I'm performing alchemy. Today I discovered that the SPLimitedWebPartManager will happily give you a collection full of ErrorWebParts, for most of the webs in a site collection. Sometimes. The web parts work (most of them); it's just that when you visit them using the SharePoint Object Model, they are transmutated into ErrorWebParts.

So what do I do to fix this? Look for more opportunities for using blocks? Poultices and cremes? Try accessing the deprecated SPWebPartCollection instead? Leeching? Attempt to unload, then reload the SharePoint assembly from my AppDomain to see if this fixes the problem? Attempt to rebalance the humours? Search Usenet? Search Google? Search the Technet forums? Examine the logs in Excel using lists and pivot tables? Examine the logs in Excel using qualitative mathematics (and no, Google, I didn't mean quantitative mathematics)?

It feels that way sometimes.

Monday, October 08, 2007 11:00:15 PM UTC 

Eulogy

SharePoint VM #3 lived a full life. A happy life.

But it was a short life. The SQL Server Express instance simply could not continue, and I made the difficult decision to put this to sleep. R.I.P., SharePoint VM #3.

What am I doing wrong?

I assume I'm doing something horribly wrong. I swear I'm not making any of the common mistakes, but let me enumerate the facts just in case:

• I store the VHD (hard drive), VMC (configuration file) and presumably the RAM file all on a single external hard drive.  This is dangerous but is a giant performance boost over running the VHD on the same spindle as the OS. No, I'm not some kind of OS guru, but this is common wisdom.
• I use Virtual Server to run my VM.
• I make sure to "Shut Down" the VM before leaving each day.
•  I use the Windows "remove external hard drive" tray icon every time before unplugging the hard drive. This is crucial as my external hard drive is (presumably) running on an NTFS partition, and unwritten delayed writes are a big potential source of corruption.
• I use VPC to run this VM from another computer. This should not be a problem.

What I refuse to do is enable Virtual Server's "undo disks" feature. Undo disks allow you to write all hard drive changes separately from the VHD file, thus allowing you to "undo" any changes you may have made. Basically, I could "revert" the undo disk any time I face corruption.

Undo disks take forever, literally forever, to merge your VHD with all the journaled changes. Forever! For daily, non-demo use, this 15-minute-plus delay is onerous. If I seriously needed undo disk features, I'd use VMWare instead—I've been told it's worlds faster and supports "branching" of VM changes.

VMWare is not an option at this time.

What I learned - VM backup strategy

This has renewed my dedication to have a solid backup plan for my VMs:

1. Store my SharePoint development work outside of the VM. In my case, I keep all assemblies in a Subversion repository on my host machine, and store a daily "stsadm -o export" site backup on the host PC. I will further note that I additionally backup the files from the host PC to another PC, thus eliminating my host PC as a point of failure.
2. Take a weekly backup of my VM's hard drive (VHD) file. This takes 15 minutes to complete, after I've turned the VM off, meaning no VM work may be done during this time. This has saved me heaps of time already.
3. Store an old VHD backup offsite. Define "offsite" yourself; whether "offsite" is your host computer's hard drive, or a file server at work, or burned to a DVD, the point is: it's stored somewhere away from the portable hard drive.

Friday, October 05, 2007 6:00:45 PM UTC 

I found out recently that Microsoft will be releasing source code for the .NET framework and libraries. There are some caveats, but we're all in agreeance: this is good news. Thanks, Developer Division, for opening up your code*!

Now, if you will allow me to turn my attentions toward the Office team.

Why Is Everything Related to SharePoint Obfuscated?

If it's not already clear: the only thing we gained from the announcement yesterday is an official blessing from Microsoft saying "yes, it's okay if you look at our source code." We are already (allegedly) doing this unofficially and in an unsanctioned manner, today, with a little additional help from Lutz' Reflector. But in the distant tomorrow, with the release of Visual Studio 2008, we'll be bona fide. Bona fide!

It seems like such a small chasm we've crossed; such an easy step to take! Well, yes, but I haven't told you about Office yet.

By example: the InfoPath Migration Tool

In the beginning, it's always simple. In the beginning, I just needed to know why did my InfoPath forms break after I moved everything to a new server? In theory this should be a simple question. Yeah. 

I've discussed elsewhere the problem of discoverability when working with SharePoint; this discoverability problem can be described by saying answers like this aren't easy enough to come by. It turns out, after this entire ordeal was over, I found this excellent page on the Office Center describing InfoPath migrations. Continuing:

After browsing Google, checking the Technet Forums and USENET, I (allegedly, Your Honor) decided to look into the source of this InfoPath Migration tool. If you don't know what the InfoPath Migration tool does, I'll quickly say that if you're moving InfoPath forms around, this tool makes the job a great deal easier. I'm thankful this tool exists, to be sure.

Anyway, because this tool was not working as I expected, I needed to be able to answer an entirely different simple question: does this InfoPath migration tool change the URN of each InfoPath form? I.e., if you have one form template and 1000 individual forms, does this tool update all 1000 individual forms, as well as the 1 template?

The answer I (allegedly, Your Honor) received from Lutz' Reflector is (allegedly) illustrated below:

Now we're getting to it

So the question I pose is: what harm is it to you, Someone In the Office Division, to skip the extra step of obfuscating this tool? Just, you know, release it like 99% of everyone else is doing? (ok, ok, 99% is an exaggeration; a significant portion of vendors release their source code un-obfuscated, but additionally release in Debug mode, granting us richer decompilation information. Yes, you'd be surprised to find what is released in Debug mode)

Why not open this one tool up? No one's sharing any secret InfoPath sauce in this tool; this tool works on InfoPath's XML documents directly. Allegedly; I wouldn't know.

Furthermore

The Developer Division has promised to open up much of their code; why not open up some (or most!) of the SharePoint code?

Or, and this should be more reasonable to consider, take a different approach from today: instead of obfuscating everything, please consider instead why obfuscate at all? The idea is: please don't obfuscate by default.

Just say no! No to drugs, and no to obfuscation! And no to both drugs and obfuscation (an especially deadly combination)!

Epilogue: the InfoPath answer

I was (allegedly) unable to decipher the raw, obfuscated IL inside of Reflector. So, defeated, instead of (allegedly) harvesting the answer directly from the source, I ran an experiment to find out what happens in this specific case.

I will point out that this is a terrible practice: combine two gargantuan, opaque frameworks, perform limited experiments on their interaction, and draw conclusions from any pattern that emerges. In science, they call this "testing a hypothesis", i.e. even if you get the desired result, they're not proven, and they can be disproved later. So having given a strong disclaimer, allow me to present the results.

After running the experiment, I concluded that InfoPath forms (specifically, the XML forms containing just the data) are 1) not modified by the InfoPath migration tool; instead, 2) InfoPath forms are modified during the "attach content database to MOSS web application" step of a WSSv2 -> MOSS migration. 3) Changes made to the web application's public URL afterwards did not affect the InfoPath forms (as it probably should have).

This is important to note, because if you're attaching your content database to a temporary (dummy) web application, then rename or otherwise move your web application's public URL, then guess what! Dead InfoPath forms**.

I'll make one last disclaimer: those three points above are not guaranteed to be true.

Footnote

* Everyone should be generally pleased with this announcement. It's not a full-blown "do-whatever-just-don't-sue-us" MIT-style license, but it's closer now. There are about a hundred arguments for or against this, either way, and they're probably all already represented among the 700+ comments responding to the announcement itself. I'm just saying: at the very least, you have to admit this announcement seems to lack malice. Contrast this with the Office Ribbon license.

** Dead InfoPath forms means, dead until you fix it. It's just XML, and if you don't like to think in XML, then it's just text, and you can certainly do something programmatically either way. But as you learn with SharePoint, it's best if you leave these sorts of uncertainties alone and figure out what everyone else has already done in this case. And they aren't writing programs to manually manipulate every InfoPath form's data as if it were a text file, I'll tell you that much.

Friday, September 14, 2007 6:08:22 AM UTC 

On the most recent DotNetRocks TV episode, Carl and Scott demonstrate PowerShell: both its concepts and features, and a lot of how generally sweet it is. Unfortunately Scott ran out of time before he got to the good parts (he was just getting to object piping). Clearly there's room for more PowerShell.

One positive thing about the show is that after watching it, I was inspired to write this post.

Outline

I'm going to cover as many random useful things I've found inside PowerShell as humanly possible. Many of these things will be stupid simple; the common pattern you should notice is that, one way or the other, these scripts save me time, sanity, or time and sanity.

Let's begin!

ONE: Calling into DOS

One of the first things I tried to do in PowerShell was a recursive directory listing. In the CMD shell (the shell formerly known as DOS), the appropriate way to do this is dir /s.

It was quite a shock to find out PowerShell's dir command doesn't support a "/s" switch. Had I known back then what I know now, I would have felt better—you can run any DOS command (including those defined in the command shell itself) from PowerShell simply by prefixing the command with "cmd /c".

I'm not saying that you should be typing "cmd /c dir /s" to get a directory listing—you should be learning PowerShell's superior dir command. But I think it's also important to note that you can get the job done today, immediately, inside the PowerShell prompt, without learning any PowerShell. Just prefix your DOS command with "cmd /c".

I'll enumerate some useful things I've found, even after learning PowerShell:

cmd /c start .

(this opens the current directory in an Explorer window. The "start" command also opens documents and URLs)

cmd /c pause

(this runs the DOS pause command. This is a better solution than a "echo 'Press any key to continue…'; read-host;")

TWO: I'm too lazy to cut and paste—PowerShell is my laziness enabler

I can't say enough good things about the Set-Clipboard cmdlet, and how useful I find it. Here are some of the different things I've run:

(dir)[-1].Fullname | set-clipboard

(dir)[-2].Fullname | set-clipboard

(The above lines are just one act in a five-act play: "How do I copy the full path and name of this file?" 1. Run dir on the entire directory. 2. Eyeball the file you want. If it's close to the bottom, you can use PowerShell's "negative index" functionality to easily count backwards. 3. Get the FullName property of the file, 4. Run the command to copy this to the clipboard, 5. Paste full path and filename elsewhere. I'll note that this hasn't necessarily saved me time.)

(history)[-1] | foreach { $_.CommandLine } | out-string | set-clipboard

(First let me explain what this line does. This pulls the last command from PowerShell's command history, and copies it to the clipboard. in theory, I could have used some other way to get this command into the clipboard—but this is so much easier.)

history | foreach { $_.CommandLine } | out-string | set-clipboard

(This takes the entire history and pastes it to the clipboard. I intend to paste the PowerShell history directly into my Notepad equivalent for pruning into a real script.)

([Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")).FullName | set-clipboard

(This gets the 4-part .NET assembly name, which you'd otherwise have to manually transcribe from your Explorer window's representation of the GAC.  I have used this with SharePoint several times already.)

$a = (sn -T MySharePointWorkflow.dll)
$a[-1].Split(" ")[-1] | set-clipboard

(Assuming the above lines are errorr-free, this gets the PublicKeyToken from "MySharePointWorkflow.dll". If I'm wrong, well, let's all pretend I'm right anyway. Something looking vaguely like this has worked for me once or twice. You should be able to guess by now, but in case you haven't: yes, it's for SharePoint development.)

THREE: Automating SharePoint deployment and administration tasks

stsadm | find """feature"""

First, I'll note that the "stsadm" command is SharePoint's command-line administration tool. I'm running the "find" command to parse the text output and display any lines containing the word "feature". In other words: I'm doing a poor man's grep.

What you may notice above is the odd arrangement of double-quote marks. This gotcha is a side effect of PowerShell's pseudo-hands off approach to interpreting DOS commands. The triple-double quotes above (""") look awkward, but will evaluate down to a single set of quotes once the command is interpreted by DOS. The specifics aren't important. What's important is that once you understand how to escape characters properly like I've done above, you can easily run DOS commands from inside PowerShell. DO IT NATIVE!

SharePoint deployment scripts - I can't tell you how many times I've seen 7-line C# console utilities that are built to supplement missing SharePoint administration functionality. While a small C# console application gets the job done, PowerShell is built for this sort of one-time use console utility. If I could get away with it (and I might yet), I would love love love to abstract away all the SharePoint deployment/development/general ugliness that makes me angry, and replace it all with a post-deployment task/small script/everything script.

Another example I've seen recently: we needed to wire up a custom 404 handler and a custom 404 redirect page. "Deployment" of this solution required both updating a property on a SPWebApplication object and copying files in a very specific manner. I'm proud to say I was able to abstract away this ugliness with PowerShell!

FOUR: REPL in .NET!

EXAMPLE: I need to figure out how the System.Uri class works. Aside: if you're working on web apps and you're unaware of the Uri class, you will want to take a look at it. Moving on.

So in doing quick research on the Uri class (or any class really), there are a few options: MSDN (ha), Google (ok), in-process debug-time experimentation with the Visual Studio Immediate Window (ok), Lutz' Reflector (often excellent), or PowerShell (yessssss). I don't know if you've guessed by now, but we're going to inspect and try things out on the Uri class with PowerShell.

$u = [Uri]"http://test.site.com/testing?query=1&query2=a&query3isempty=&query4=something"

$u
$u | gm
$newUri = new-object "System.Uri" -arg $u, ([Uri]"http://some.other.site/letstry/relative/").PathAndQuery

$newUri

What's awesome about a read-eval-print loop (REPL) is that it reduces experimentation time from minutes (or hours!) to seconds. Actually, anyone who has used any scripting language will be able to tell you why REPL is awesome; feel free to read THEM instead of ME. I hear LISP has been around for a while; maybe you could ask your grandparents.

Tuesday, September 11, 2007 3:48:50 AM UTC 

Workflow Tasks, the Replicator activity and State Machines

This post is in a response to a question I saw yet again on the Technet forums today. This question is in no way unusual—in the course of developing SharePoint workflows, many of us eventually ask "how can I fire off a slew of Tasks all at once?" The answer, after some research, is always to use the Replicator activity.

Replicator activity

The Replicator activity can be described as a "foreach loop for workflow." It's not that simple, but the gist of it is: if you have a list of things (let's hypothetically say, SharePoint Workflow Tasks), you can run parallel Workflow threads for each item in the list, without knowing in advance how many items you'll want to replicate. See, I even used the words "for each" in the description.

There are some workflow concepts that don't easily translate to traditional "imperative" languages, at least not in my vocabulary. Read up elsewhere on the Replicator task if you're interested in digging deeper.

ASIDE: If you're researching this topic, I highly recommend you look at the Approval workflow example in the MOSS SDK. I've searched, trust me, and the Approval workflow example is as good as it gets.

So far, so good, right? Unfortunately, there's a twist!

The twist: State Machine Workflows

I'll preface this with a disclaimer: I'm not a Workflow mega-expert (maybe I am by now, but as of 2007-09-19, I have not yet been awarded the "mega" rank). If you, dear reader, happen to know a better solution, by all means leave me a comment.

Back to the twist: you can't replicate SharePoint Workflow Tasks inside of a State Machine Workflow if you need to wire up multiple events. If you want to study that last sentence for a while, feel free; I'll be here waiting…and, we're back!

The specific problem you'll inevitably run into may be found in the collision of the following conflicting requirements:

• State Machine workflows allow only one event in each state.
• Replicator activities must be embedded inside a single state.
• In order to fire off a slew of Tasks, you'll need to embed multiple events inside a single Replicator activity.

Putting these statements together: if you're creating a useful workflow involving Workflow Tasks, you'll require two (or more) events. All these events must go inside a Replicator activity in order to, um, "replicate" them. This Replicator, in turn, must be embedded (and yet cannot be embedded) in a single state in the state machine.

I can't believe you made it this far

Let's sum up with bullet points:

• PROBLEM: You won't be able to use a Replicator inside a State Machine in many cases, due to limitations of the State Machine enviroment.
• SOLUTION: Rewrite your workflow as a Sequential Workflow.

Any advice and all alternate solutions are greatly appreciated. Besides idle curiosity, I have encountered this problem myself and would love to keep "my precious" state machine intact. There's still time!

This is neither a SharePoint-only problem, nor a Replicator-only problem

This is a problem you may encounter in many walks of life. Specifically, you may encounter this problem in the many walks of life that involve programming with all of a) multiple events, b) composite activities, and c) state machine workflows.

Ok. What I mean is that you can get in the same "composite activity containing multiple events" trouble with a Conditioned Activity Group, or a Parallel activity, or…actually I'm tapping out here; I've pretty much exhausted my Workflow knowledge with this post. The point is: it's not just Replicators, and it's not just the SharePoint Workflow Task-related activities.

This can happen to you!

Conceptual nonsense: embedded state machines

And while I have no idea what goes into building an extensible workflow engine, may I humblly submit the following suggestion: can we embed state machines inside state machines? While this sounds crazy (AND THIS IDEA IS, IN FACT, CRAZY), allow me a defense:

• CURRENT BEHAVIOR: Replicators are, in a conceptual sense, kicking off parallel Sequential Workflows. Hosted in a Sequential Workflow, this matches up nicely, as you can easily envision the Replicator activity as a sort of "accordion-style pile of Sequential Workflows—expands or contracts as necessary."  The point is, it fits the Sequential Workflow conceptual model.
• PROPOSED ALTERNATE BEHAVIOR: StateMachineReplicators could kick off multiple child State Machine Workflows, one state machine for each replication. This "mess of child state machines" would appear to the host as a single state (i.e. the StateMachineReplicator). Inside, you code up your events in separate child states.
  
  At this point, the word "state" is losing meaning, so I'll try to wrap up:
  • BONUS: You can now embed Replicators (and probably other similar activites, like Conditioned Activity Groups) inside State Machines. AND they'll now be functional enough to be useful!
  • BONUS: You maintain the conceptual integrity (see subtitle for this section) of the State Machine—you still maintain the 'no more than one event on a state' rule.

Sunday, September 09, 2007 11:05:23 PM UTC 

Introduction

Many months ago I listened to the DotNetRocks interview of Dan Appleman, wherein he discussed the concept of discoverability. I took two things away from that interview:

• The realization that we spend way too much time looking things up
• A useful tool in Dan Appleman's "Google Custom Search" engine: SearchDotNet.com

It's true, especially in the SharePoint world, that we spend way too much time looking things up. There's "big heap framework", but not "big heap documentation" to help understand it.

So as Dan Appleman has done for "vanilla" .NET content, I've done something to help us all find SharePoint content on the web.

Direct link to the page

In case you're already tired of reading, allow me to link to the page:

It's not nearly as awesome as you think

It's not as awesome as I would like, but it's a start, and it's already proved useful for me, today.

First Tool: Peter's SharePoint Search page

I've created an aggregated SharePoint search page containing search boxes for the three search engines I use regularly. This isn't incredibly awesome, but I figure a) the page loads in less than 1 second, which is faster than the corresponding pages on microsoft.com's site, and b) the search boxes are listed together on the same page, so you can quickly and conveniently search all three if necessary.

So why would I use anything besides Google web search?

While we all know and love Google's web search, I would love Google more if they did a better job indexing forums. At present, Google misses most of what goes on on the Technet SharePoint forums.

To get better access to Technet forum content than Google search provides, we can already use the search box provided on this page  on Microsoft's site. Up 'till now this is what I've been using, and it works moderately well.

To improve the search experience and also to spread awareness (everyone should be searching the SharePoint Technet forums), I've created a search box that queries Microsoft's SharePoint-specific Technet forums. I'm not making anything new; I'm just mimicking the search box you'll find on their page.

I've also included Microsoft's Newsgroups (Usenet) search for SharePoint. This searches all (and only!) the Microsoft SharePoint Usenet groups. While I haven't found the content on the newsgroups to be as bountiful a harvest as the Technet forums—hey, if you're desperate for information, you should be checking all available sources, right?

I haven't figured out how to search a limited subset of groups (i.e., all the SharePoint Usenet groups) with Google Groups, otherwise I'd be all over Google Groups search instead.

BOTTOM LINE: I've included two useful search boxes that index content not available to Google search. With these extra search engines, you'll find more stuff.

Second Tool: Custom Search Provider for the Technet Forums

I figure: I don't want to visit my SharePoint search page, and it's something I created by my own self! So I don't expect you, dear reader, to visit some total stranger's wacky search page every time you need to look for something.

To further improve the search experience, I made a custom search provider that works with Firefox and IE7, and resides in the top right corner of your browser window, waiting for your input. When you are struck by fancy, the specific kind of fancy that involves searching Microsoft Technet's SharePoint forums, the search box is there, waiting.

I haven't bothered to create a custom search provider for Newsgroups (Usenet) search—so if you're looking for it, it doesn't exist.

Why would I use the custom search provider?

The custom search provider makes searching Microsoft's Technet forums worlds easier, and is unobtrusive to boot. Let's walk through all the steps to run a search:

1. Press CTRL+K to set focus on the Custom Search box. [This shortcut works in both IE and Firefox] Or you can click with the mouse, you mental sluggard.
2. Type in your search query and press ENTER.
3. There is no step 3. I'm just including it because you're probably expecting more steps.

How do I install the custom search provider?

1. Visit the SharePoint search page—preferably in a new window, so you can read further instructions. :)
2. Click on the dropdown and select the "Add Microsoft Technet Forums" option (specific instructions will vary for IE).
3. There is no step 3. I'm just including it because you're probably expecting more steps.

Direct link to the page

Once again, allow me to link to the page:

Wednesday, August 29, 2007 4:13:26 AM UTC 

The magic representative line

The single line (#17) in the image above calls stsadm.exe (a console command) and passes several arguments to it: 1) a bunch of SharePoint stsadm junk/raw text, and 2) the GUID of the feature I was trying to remove.

What is certainly most uninteresting is the SharePoint details; this was development-VM administrivia. What is certainly interesting is that I have run the PowerShell interpreter where needed to grab the Feature's GUID, but sent the rest of the text directly to the console application without surrounding anything in quotes. Try and point out, on line #17, where DOS ends and PowerShell begins.

Thus, magic!

Aside from all the interpreter nuttiness, this was super useful and saved me heaps of time*, so there was some practicality involved as well.

*heaps of time => probably 20 seconds or less

Wednesday, August 22, 2007 4:53:59 AM UTC  UPDATE 2007-08-21: I've removed my earlier misinformation. In its place, I'll leave you with the following:

The Golden Rule of Advanced Troubleshooting

It's always the invisible layer.

In the most recent case, I was trying to eliminate a long web page loading delay. I'll go into details of the specific problem another time—the point being, the root problem was an invisible proxy on the network layer. Apparently HTTP proxies have problems working with gzipped packets. Who knew.

The Golden Rule: Enumerated

What do I mean by "invisible proxy"? Well, here are some examples (AKA culprits!):

• Firewall
• Antivirus
• HTTP (web) proxy
• SMTP (mail) proxy

If you're pulling your hair out troubleshooting like I was, you'll want to review the Golden Rule. Oh, and another thing:

Rare Problems

I think it's important to note that in this context  I'm talking about troubleshooting rare problems. If this was a common problem, you'd have fixed it already; you wouldn't be searching for invisible proxies or discovering new layers, because the visible/tangible layers are so much more discoverable. Why check your firewall settings when your network cable is unplugged?

Trust No One: You

Rare problems require a different mindset from common problems.

Trust nothing; start looking at your own assumptions and (of course) anyone else's assumptions; look to the Golden Rule of Troubleshooting. Double-check your earlier work. Double-check your double-checking to be sure you covered everything. Get some rest, then come back and double-check your double-double-checking.

Trust No One: Them

Double-check everyone else's work (yeah, I know).

When doing advanced troubleshooting, it's important to keep in mind that a) with computers, there is always a logical answer (even if that answer is "the hardware is intermittently breaking), b) you've already isolated every single layer possible, and c) it's now time to stop believing people and now time to start checking their assumptions. An important corollary to part c is you must also turn a distrustful eye towards yourself. Did you check everything? Are you sure? Before you make that accusatory phone call, are you 100.0% sure?

And of course, Google

I don't know what we did before search engines. If nothing else, even if Google can't help me find the solution, at least it can point me to others experiencing the same problem.  If nothing else, search engines prove that we're not alone—we're not crazy!

And Vendor Support

(this section intentionally left blank)

Thursday, July 26, 2007 12:37:57 AM UTC  Thanks to a new SharePoint performance whitepaper, we now know with certainty: even though the technical specs indicate you can store 10M items in a single list, the truth is: continue to keep your lists under 2000 items. Almost all CRUD operations perform well (surprisingly well sometimes), but delete operations take 50 seconds to complete! For one item!
YOW!

I need to find some uniquely obnoxious way to indicate how long it takes to delete an item. MARQUEES to the rescue!

'########:::'#####::::::::::'######::'########::'######:::'#######::'##::: ##:'########:::'######:::::::::::'##:::'##::'#######::'##:::::'##:
 ##..,..:::'##.. ##::::::::'##.., ##: ##..,..::'##.., ##:'##..,. ##: ###:: ##: ##..,. ##:'##.., ##::::::::::. ##:'##::'##..,. ##: ##:'##: ##:
 ##:::::::'##:::: ##::::::: ##:::..:: ##::::::: ##:::..:: ##:::: ##: ####: ##: ##:::: ##: ##:::..::::::::::::. ####::: ##:::: ##: ##: ##: ##:
 #######:: ##:::: ##:::::::. ######:: ######::: ##::::::: ##:::: ##: ## ## ##: ##:::: ##:. ######::'#######:::. ##:::: ##:::: ##: ##: ##: ##:
..,.., ##: ##:::: ##::::::::..,.. ##: ##..,:::: ##::::::: ##:::: ##: ##. ####: ##:::: ##::..,.. ##:..,..,..:::: ##:::: ##:::: ##: ##: ##: ##:
'##::: ##:. ##:: ##::::::::'##::: ##: ##::::::: ##::: ##: ##:::: ##: ##:. ###: ##:::: ##:'##::: ##::::::::::::: ##:::: ##:::: ##: ##: ##: ##:
. ######:::. #####:::::::::. ######:: ########:. ######::. #######:: ##::. ##: ########::. ######:::::::::::::: ##::::. #######::. ###. ###::
:..,..,:::::..,..:::::::::::..,..,:::..,..,..:::..,..,::::..,..,.:::..::::..::..,..,..::::..,..,:::::::::::::::..::::::..,..,.::::..,::..,::: Marquees are obnoxious!

Thursday, July 19, 2007 6:50:39 AM UTC 

I recently read Simon's post about using Reflector with SharePoint. Inspired, I set about to use Reflector as a sort of "MSDN 2" to help me work through holes in MSDN's (and to be fair, the rest of the world's) documentation. Unfortunately for me, I had to track down DLLs and ASMX files from different subfolders of the 12 hive and even from the GAC.

I'm writing this post to help me (and you) find the SharePoint 2007 DLLs and EXE's and everything else we might need to reverse eng…um, let me rephrase that, "embrace a lifestyle of self-service SharePoint documentation." That last bit sounds nicer, yes.

For various non-technical reasons, I can't just post the code directly here—instead, we must each go to our MOSS (or WSS) farms and copy out the DLLs. Thus far I've found three directory trees SharePoint is storing its code:

Name	Directory	SKUs	Fun Facts!
The 12 Hive	C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\	WSS, MOSS	What's sad is that I typed the full directory from memory—including the awkward capitalization.
The "Office Server" directory	C:\Program Files\Microsoft Office Servers\ (can be customized)	MOSS	Post your fun fact in the comments, otherwise I'll have to stick with "Fun fact: Excel Services .dllis hosted here!" See, you can do better.
The GAC	%WINDIR%\Assembly\	WSS, MOSS	Windows Explorer hides files in the GAC from you (it's a special folder), so you'll have to work a little to get at them.

The added value I'm here to present today is, well, I automated all of the copying—I wrote a PowerShell script to grab all the files. It's way overkill (especially the Vista-style nuttiness when I spend half of the script calculating the estimated time the other half will take to complete), but hey, why not?
Here we are:

Find-SharePointDllsVistaEdition.ps1 (2.06 KB)

#Freely distribute and use. Absolutely no warranty is implied.
$dirTwelveHive = "C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\"
$dirMossServers = "C:\Program Files\Microsoft Office Servers"

$subDirectory = read-host "Full pathname to write files?"
mkdir $subDirectory -ErrorAction SilentlyContinue | out-null
mkdir ([System.IO.Path]::Combine($subDirectory, "12-hive")) -ErrorAction SilentlyContinue  | out-null
mkdir ([System.IO.Path]::Combine($subDirectory, "MOSS")) -ErrorAction SilentlyContinue  | out-null
mkdir ([System.IO.Path]::Combine($subDirectory, "GAC")) -ErrorAction SilentlyContinue  | out-null

echo "Calculating time required to perform this operation…"
$excludedFileExtensions = dir -path "$([System.IO.Path]::Combine($dirTwelveHive, ""template\images""))
" | foreach { $_.Extension.ToUpper() } | group | foreach { $_.Name }
$excludedFileExtensions += ("MDF","LDF")
$exclusionCriteria = $excludedFileExtensions | foreach { "*$_"}

$allFiles = dir -path $dirTwelveHive -recurse -exclude $exclusionCriteria -ErrorAction SilentlyContinue
$allFiles += dir -path $dirMossServers -recurse -exclude $exclusionCriteria -ErrorAction SilentlyContinue
$allFiles += dir -path ([System.IO.Path]::Combine($subDirectory, "GAC")) -recurse -include "*SharePoint*.dll" -ErrorAction SilentlyContinue

$allFiles | Select Extension | group Extension | select Name, Count | sort Count -desc | select -first 10

$sum = 0
$allFiles | foreach {
    $sum += (dir $_).Length
}

echo "`n`n Total size in MB: $($sum/1MB)"

cp -recurse -force -path $dirTwelveHive -destination ([System.IO.Path]::Combine($subDirectory, "12-hive")) -exclude $exclusionCriteria
cp -recurse -force -path $dirMossServers -destination ([System.IO.Path]::Combine($subDirectory, "MOSS")) -exclude $exclusionCriteria

#does not preserve directory structure
$filesFromGac = dir -recurse -path "$([System.IO.Path]::Combine($env:WINDIR,""assembly""))" -include "*SharePoint*.dll"
$filesFromGac | foreach { cp -force -path $_ -destination ([System.IO.Path]::Combine($subDirectory, "GAC")) }

I'd better hit the "Post" button before my laptop battery runs out.

Thursday, July 19, 2007 4:42:28 AM UTC 

In my other, somewhat-less-happy post about CAML, I shared my frustration of working with the MSDN (and other) CAML documentation. What I failed to do in that other post was share the results, i.e. tell you how I eventually got the job done.

This will not bring you all the way from start to finish, but should help you hurdle some of the less obvious…let's call them 'hurdles'. I'm sticking with the hurdling metaphor—there's no point in switching metaphors midsentence, even if the word 'hurdle' is starting to lose all meaning.

Step #1: Go visit the U2U Community Tools page and download both of their excellent CAML query tools. If you stop reading now and click on the U2U links, it's probably for the best.

The two tools are:

• "CAML Builder" - this Windows app that will actually use SharePoint's web services to pull list metadata, meaning that you can actually build your CAML query almost as easily as every other query tool on the planet. The one gotcha is that (in this case) it mingles the three XML chunks together (see code below for an explanation of what I mean by "chunks").
• "DevFeaturesPackage" - yes, that's what it's called. This is a v3-only Feature that attaches itself all lists' Actions menu in your browser. And, it works like every other query tool on the planet.
  

My "works like every other query tool on the planet" quips above might sound sarcastic, but trust me: if you've spent any amount of time working with this stuff, finding things familiar and useful is a great relief.

Step #2: Find your lists.asmx web service URL and add it to your project as a web reference. In my case, and for probably every other case in the entire world, it will be http://servername/site/site/site/_vti_bin/lists.asmx - unless you've done something crazy, like install SharePoint on a subdirectory of the web application. Note that the URL does include the site heirarchy. This paragraph is a longwinded way of saying "server name, plus site heirarchy, plus_vti_bin, plus lists.asmx".

You can be absolutely certain by running your lists.asmx URL through the U2U CAML Builder and verifying you're connecting to the correct site.

Step #3: Build your CAML query. I'm not going to attempt to help with this; instead, I'll refer you to the U2U CAML query tools again.

Step #4: Write code to build up the six (6) parameters this web service takes. I've posted the PowerShell function I used to do this, and annotated where necessary (which is everywhere).

Step #5: You're pretty much through! Now all you have to do is bask in the glory of a job well done! Don't even test! You can't test—you're too busy partying and you got it right the first time! Woo!

Well, you might want to test.

Footnote: My PowerShell code, doctored some—if you plan on copy/pasting this, you'll probably want your code…cleaner. Without further ado:

#—
#Tested only against WSS v2. Presumably also works with v3 sites.
function Get-ListItems ($webService)
{
   
    $listname = "YOUR_LIST_NAME_GOES_HERE"

    #Empty string indicates we use the default view. Otherwise, we'll have to provide the view GUID. I think.
    #Presumably using a custom view is an excellent way to set a filter on your data set.
    $viewname = ""

    #I don't know what the rowlimit means, and though I can GUESS, I don't want to ASSUME.
    $rowlimit = "100"

    #I created an XmlDocument object ONLY so I could create valid XmlElements for the 3 web service arguments.
    #$x is not used directly.
    $x = new-object "System.Xml.XmlDocument"

    #<Query> - presumably you can set attributes or inner XML if necessary.
    $q = $x.CreateElement("Query")

    #<QueryOptions> - presumably you can set attributes or inner XML if necessary.
    $qo = $x.CreateElement("QueryOptions")

    #I'm posting my "ViewFields" element as-is. The goal of my query was to pull the ID of every
    #item visible in the default view. I will assume that your ViewFields element will have more content.
    $v = $x.CreateElement("ViewFields")
    #Don't be thrown by the odd PowerShell string syntax. "" inside a quote is like \" in C#.
    # the .set_InnerXml is just one of PowerShell's ways of representing .NET's property set operations.
    $v.set_InnerXml("<FieldRef Name=""ID"" />")

    #I can't remember why this was necessary anymore, but is probably related to my ignorance of the XmlDocument object.
    #If I'm guessing correctly, SharePoint requires XML submitted as <Query></Query> instead of <Query />. But that's guessing.
    #Either way, feel free to ignore these two lines and see if you can get your lists web service working without them.
    $q.set_InnerXml("")
    $qo.set_InnerXml("")
  

    $result = $webService.GetListItems($listname, $viewname, $q, $v, $rowlimit, $qo)

    #this is awkward PowerShell syntax that could probably be replaced with some proper XPath.
    $trimmedResult = $result.data.row | foreach { $_.ows_ID }
  
    #this line means "return trimmedResult"
    $trimmedResult
}
#—

Friday, June 29, 2007 3:33:13 AM UTC 

UPDATE 2007-07-18: I've posted some instructions revealing how to use the GetListItems method of the lists.asmx web service. Check it out: http://www.pseale.com/blog/BriefestIntroductionToGetListItemsUsingCAMLAndListsasmx.aspx

UPDATE 2007-10-09: I've re-updated this to be gentler and more respectful. Probably not enough, but at least I'm heading in the proper direction. If there's one thing you get out of this, it's that I really, really, really please please would like to have some thorough documentation on CAML (the XML itself). Not necessarily samples or "wisdom"-type articles; instead I'd like just a thorough basis (a solid foundation) upon which we can build.

In Other News

Before I get to the main event, let's divert to something less infuriating. I won't give this away, except to say the page I'm about to link to is the #1 search result on Google for "Oracle DBA redneck". Be enlightened in Redneck philosophy. I think this page speaks better for the man's Oracle skills than any resume or testimonial—because if he doesn't have the skills, then this page guarantees he will never find work again. Clearly the man has confidence.

And Now, CAML

I'll keep this short, so as to narrowly funnel the rage. Let's try to do this factually:

• CAML is an XML query language (Collaborative AML) used to do "a variety of things" in SharePoint. It's not the Caml programming language; this little "bit of trivia" becomes a little less…trivial…when you're searching for documentation.
   
• Which brings me to Point Number Two. Despite what you may have heard, there is no documentation on CAML. Sure, they've auto-generated some skeleton pages on MSDN, and added a bare minimum of exposition. So if you go visit the links, you'll see the form (if not the content): Headers, Subheaders, sparse text, and ooh, code samples! There's probably thousands of MSDN pages on CAML. Now look closely. What do you see? Do the code samples actually help? Are they really code samples, or are they the same function declaration in 7 language descriptions, including JScript and J#? Let's bring this back: Point Number Two:  Worthless documentation is, in the long run, worse than no documentation at all. You don't waste time reading if it's not there to read!
   
• Where are the CAML tools? U2U has a free CAML query-building GUI tool that works pretty much as you imagine—it loads your list metadata, you select your columns and your query type, you click the magic button, and…it actually spits out valid CAML! Except that (in my case) you have to split up the three emitted XML chunks, then reconstitute them with three other (thankfully non-XML (but one is a GUID, and you'll have to dig at another web service to find out what the GUID is…)) parameters. So easy! Incidentally there's another in-browser Query Builder deployed such that it appears on your List Actions menu.
   
• Lastly: Where's the outrage? I don't see enough people complaining about this, so I think a lot of people don't even see the pain to begin with.
   

I'd like to end on a positive note, but can't, so we'll have to settle for:

Action Items - Help Me Cope With the CAML Ecosystem in its Current State

• I will avoid CAML at all costs, only dipping in if absolutely necessary (underway).
• I will check around to see if I've missed something big, e.g. some actual documentation. It's entirely possible.
• I will search for more tools.

MSDN Rage Meter

This is what I like to call a "usefulness heat map". Or if you like to think in negative (or video game) terms, a visual Rage Meter. White indicates useful content; Red indicates filler.

This is how most of MSDN looks through the lens of my metaphorical Rage Meter. All, form does not equal content!

Wednesday, May 03, 2006 3:55:07 AM UTC 

Testing a SharePoint site, I managed to produce the following:

AAAAAH!

 

AAAAAH!

 

 

AAAAAH!

 

AAAAAAHHHHHHHHHHH!!!!!!!!!!!!!!!!!!

This is some sort of double-nightmare—the Office installer and Clippy present at the same time—enough to cause night terrors!