Challenges with TypeMock and Team Build

According to the TypeMock documentation it's a breeze to set up mocking during Team Build. Well I have just spent a week trying to get it to work and I think I finally cracked it. It all comes down to reading the f*** manual and believing it to be taken literally.

 

TypeMock location

The TypeMock manual say the following:

<!-- Use Typemock Isolator MsBuild Extensions -->
<Import Project="C:\Program Files\TypeMock\Typemock Isolator\TypeMock.MSBuild.tasks" />

<!-- Override this target to plugin custom tasks before running tests -->
<Target Name="BeforeTest">
  <!-- Start TypeMock -->
  <TypeMockStart/>
</Target>

<!-- Override this target to plugin custom tasks after running tests -->
<Target Name="AfterTest">
  <!-- Quit TypeMock -->
  <TypeMockStop/>
</Target>

Well first of all this isn't quite true. As of 5.1 (I think) TypeMock installs into another directory (C:\Program Files\Typemock\Isolator\5.2 in my case as we just upgraded to 5.2.1).  So you have to perform some trickery in your MSBuild files to detect whether your doing a build server or desktop build. Luckily there are ways to do this.

Anyway, this is not my main point. The main point is that you can get tricked by the way MSBuild is implemented and used during an EndToEndIteration build.

Where does the TypeMockRunner instance live?

What I saw was that my tests were running fine during DesktopBuild but failing miserably during E2EI (yes I ran a desktop build (msbuild TFSBuild.proj /t:test) on the build server and the tests ran fine). So that’s what I've been spending the last week trying to unravel. And here are the results from the Norwegian jury:

You must follow the instructions (above) from the TypeMock doc to the extent that you call TypeMockStart/TypeMockStop from TFSBuild.proj itself.


We have an existing build system (CruiseControl) so I was reusing my Test.proj calling the necessary targets from inside that. THIS WILL NOT WORK IN A TEAM BUILD CONTEXT! You have to call TypeMockStart from the same instance of MSBuild as the one you're running tests from (i.e. calling MSTest/TestToolsTask)!


Consequence: The multiple agents switch (/m) of MSTest is incompatible with TypeMock.


I had to disable building in parallel because of this. It has to be said that the guys at TypeMock are waiting for me to have some available time so we can look into this since this is obviously no more good for them than it is for me. Ay ve! So much to deal with and so little time.

A nice overview of the EndToEndIntegration model of Team Build

http://blog.accentient.com/TeamBuildTargetMap.aspx

Bug in how VS/VSTestHost loads symbols?

I have started running source server on our TFS build box. (blog about that later) and I just noticed a peculiarity in how the symbol files where loaded during a "Debug Checked Deploed Tests".

The two different findings arise when downloading the testresults from a team build.

It seems that when I started with an empty Visual Studio and from there opened the downloaded testrun the "Debug Checked Deployed Tests" handled the source server brialliantly and the debugger broke inside a copy of the versioned item, as expected.

However, when i HAD MY SOLUTION OPEN IN VISUAL STUDIO and the debugger breaks, it claims that there is no source available and the only choice it gave me was to open the dissambly. Now I',m curious so I open the Modules window, and there I see that the loaded pdb files are the correct ones (or so it seems). I open them in Notepad and they look correct.

Now the really strange thing happens when I hit F5. I get breaked at the next failing test and there everthing is ok, I get the downloaded versioned item opened and the debugger breaks at the correct location(?!?). Gotta investigate more on this. For now I will just list it as another mystery.

How to make VS 2008 swallow more memory

While wating for VS next (2010) which for sure will be 64Bit and massively concurrent on top of the new concurrency APIs in 4.0 (yeah right ;)) this one has been proved to work (thx to Steven Harman):

http://stevenharman.net/blog/archive/2008/04/29/hacking-visual-studio-to-use-more-than-2gigabytes-of-memory.aspx

This should make 4GB virtual memory (logical barrier for 32bit app) available (I think?) but it doesn't look that way. I seems to bump at 3GB virtual space now. It's still a massive improvement though and it makes my 100 project 1MLOC solution actually a bit snappy even with ReSharper running on all sylinders (even running solution wide analysis).

What about physical memory consumption? On my four core Windows 2008 Server 64Bit/8GB dev box I was hoping Stevens trick would release more physical memory than the 2GB it looks like its stopping at now. Anyone have one more trick up their sleeve to tweak the last logical GB out (and I would rather prefer electron flux (physical memory) over magnetic manipulation of iron dust (swap to disk)) while waiting for the above mentioned nirvana?

Actually I hardly see any disk activity at all. The dev box is equipped with a solid battery powered write cache (windows will take care of the read caching, thx Christian for correcting me on that little trivia ;)) in front of 10K+15K RPM disks :) Windows 2008 does a good job of caching stuff it needs in a situation like that. Cannot take the machine with me when I go home though (doesn't matter, I can remote it over VPN anyway).
----------------

Update: I think I have to try the same trick on ReSharper too (parts (most?) of it runs in it's own process) because I am still getting OutOfMemory execption from it.

Possible paralellism scheme in MSTest

While I am sitting here waiting for a test run to finish I might as well vent my frustration and hopefully spark up some ideas that might resonate with the Team Test crew in Redmond.

The possible paralellism in a run of MSBuild seems well under way to be taken care of and is now partly under the control of the user (pending disipline amongst the developers). Only yesterday I switched over to /m:8 on my 4 core build machine just to see what it would do to my build process and it was a sight for sore eyes seeing the 7+1 MSBuild processes firing at will on all four cores.

The only problem I have right now is that our code base is big and therefore divided into several VS solutions. Enabling cleaning of the output then tripped the whole wagon since it seems the solutions are batch-built as a whole. Since we have dependencies between the solutions the build order of the solutions must be observed. Too much goodies then so I switched back to single build process.

I have been working for a while on a single VS solution containing all our projects (8GB, 4Core developer blades with 15K + 10K RPM disks seems to be able to take the pain without too much(!) trouble), but thanks to some legacy(sic!) J# components (needs J#SE and VS2005) that seems hard to get rid of this is not a path that will be resolved for a while.

Meanwhile on the test side of things there is currently no free lunch up for grabs even from the product side let alone our own disipline. MSTest will allow you to specify whether tests should be run in separate containers or with no isolation, but it will not run the tests in paralell whatsoever (Google "mstest paralell" and be convinced).

IMHO this is definitely on the most wanted feature list for VS next (2010) and I hope to see it (haven't had enough time to check it out yet). In my dreams the feature would have these high level properties:

- MSTest switch to specify at which level the paralellism should take place
- The switch would take its value form the tuple {Solution|Assembly|Class|Method} (update: consider even TestList as a value)
- MSTest switch (like for MSBuild) to specify number of nodes (/m:)

It would be left to the developer to guarantee that the test units (not unit tests) at the given granularity observes independence of all other test units on that level. In other words, if I knew (because I have worked my behind of to get it like that) that all my [TestClass] adorned classes were completely isolated at runtime from all other [TestClass] classes, then I would specify /p:Isolation=Class in my test run. MSTest/VSTestHost would then run all tests in paralell on all available nodes and I wouldn't have to sit around looking at three out of four cores doing abolutely nothing while the tests take forever.

Now wouldn't that be great?