Challenges with TypeMock and Team Build

According to the TypeMock documentation it's a breeze to set up mocking during Team Build. Well I have just spent a week trying to get it to work and I think I finally cracked it. It all comes down to reading the f*** manual and believing it to be taken literally.

 

TypeMock location

The TypeMock manual say the following:

<!-- Use Typemock Isolator MsBuild Extensions -->
<Import Project="C:\Program Files\TypeMock\Typemock Isolator\TypeMock.MSBuild.tasks" />

<!-- Override this target to plugin custom tasks before running tests -->
<Target Name="BeforeTest">
  <!-- Start TypeMock -->
  <TypeMockStart/>
</Target>

<!-- Override this target to plugin custom tasks after running tests -->
<Target Name="AfterTest">
  <!-- Quit TypeMock -->
  <TypeMockStop/>
</Target>

Well first of all this isn't quite true. As of 5.1 (I think) TypeMock installs into another directory (C:\Program Files\Typemock\Isolator\5.2 in my case as we just upgraded to 5.2.1).  So you have to perform some trickery in your MSBuild files to detect whether your doing a build server or desktop build. Luckily there are ways to do this.

Anyway, this is not my main point. The main point is that you can get tricked by the way MSBuild is implemented and used during an EndToEndIteration build.

Where does the TypeMockRunner instance live?

What I saw was that my tests were running fine during DesktopBuild but failing miserably during E2EI (yes I ran a desktop build (msbuild TFSBuild.proj /t:test) on the build server and the tests ran fine). So that’s what I've been spending the last week trying to unravel. And here are the results from the Norwegian jury:

You must follow the instructions (above) from the TypeMock doc to the extent that you call TypeMockStart/TypeMockStop from TFSBuild.proj itself.


We have an existing build system (CruiseControl) so I was reusing my Test.proj calling the necessary targets from inside that. THIS WILL NOT WORK IN A TEAM BUILD CONTEXT! You have to call TypeMockStart from the same instance of MSBuild as the one you're running tests from (i.e. calling MSTest/TestToolsTask)!


Consequence: The multiple agents switch (/m) of MSTest is incompatible with TypeMock.


I had to disable building in parallel because of this. It has to be said that the guys at TypeMock are waiting for me to have some available time so we can look into this since this is obviously no more good for them than it is for me. Ay ve! So much to deal with and so little time.

A nice overview of the EndToEndIntegration model of Team Build

http://blog.accentient.com/TeamBuildTargetMap.aspx

Bug in how VS/VSTestHost loads symbols?

I have started running source server on our TFS build box. (blog about that later) and I just noticed a peculiarity in how the symbol files where loaded during a "Debug Checked Deploed Tests".

The two different findings arise when downloading the testresults from a team build.

It seems that when I started with an empty Visual Studio and from there opened the downloaded testrun the "Debug Checked Deployed Tests" handled the source server brialliantly and the debugger broke inside a copy of the versioned item, as expected.

However, when i HAD MY SOLUTION OPEN IN VISUAL STUDIO and the debugger breaks, it claims that there is no source available and the only choice it gave me was to open the dissambly. Now I',m curious so I open the Modules window, and there I see that the loaded pdb files are the correct ones (or so it seems). I open them in Notepad and they look correct.

Now the really strange thing happens when I hit F5. I get breaked at the next failing test and there everthing is ok, I get the downloaded versioned item opened and the debugger breaks at the correct location(?!?). Gotta investigate more on this. For now I will just list it as another mystery.

How to make VS 2008 swallow more memory

While wating for VS next (2010) which for sure will be 64Bit and massively concurrent on top of the new concurrency APIs in 4.0 (yeah right ;)) this one has been proved to work (thx to Steven Harman):

http://stevenharman.net/blog/archive/2008/04/29/hacking-visual-studio-to-use-more-than-2gigabytes-of-memory.aspx

This should make 4GB virtual memory (logical barrier for 32bit app) available (I think?) but it doesn't look that way. I seems to bump at 3GB virtual space now. It's still a massive improvement though and it makes my 100 project 1MLOC solution actually a bit snappy even with ReSharper running on all sylinders (even running solution wide analysis).

What about physical memory consumption? On my four core Windows 2008 Server 64Bit/8GB dev box I was hoping Stevens trick would release more physical memory than the 2GB it looks like its stopping at now. Anyone have one more trick up their sleeve to tweak the last logical GB out (and I would rather prefer electron flux (physical memory) over magnetic manipulation of iron dust (swap to disk)) while waiting for the above mentioned nirvana?

Actually I hardly see any disk activity at all. The dev box is equipped with a solid battery powered write cache (windows will take care of the read caching, thx Christian for correcting me on that little trivia ;)) in front of 10K+15K RPM disks :) Windows 2008 does a good job of caching stuff it needs in a situation like that. Cannot take the machine with me when I go home though (doesn't matter, I can remote it over VPN anyway).
----------------

Update: I think I have to try the same trick on ReSharper too (parts (most?) of it runs in it's own process) because I am still getting OutOfMemory execption from it.

Possible paralellism scheme in MSTest

While I am sitting here waiting for a test run to finish I might as well vent my frustration and hopefully spark up some ideas that might resonate with the Team Test crew in Redmond.

The possible paralellism in a run of MSBuild seems well under way to be taken care of and is now partly under the control of the user (pending disipline amongst the developers). Only yesterday I switched over to /m:8 on my 4 core build machine just to see what it would do to my build process and it was a sight for sore eyes seeing the 7+1 MSBuild processes firing at will on all four cores.

The only problem I have right now is that our code base is big and therefore divided into several VS solutions. Enabling cleaning of the output then tripped the whole wagon since it seems the solutions are batch-built as a whole. Since we have dependencies between the solutions the build order of the solutions must be observed. Too much goodies then so I switched back to single build process.

I have been working for a while on a single VS solution containing all our projects (8GB, 4Core developer blades with 15K + 10K RPM disks seems to be able to take the pain without too much(!) trouble), but thanks to some legacy(sic!) J# components (needs J#SE and VS2005) that seems hard to get rid of this is not a path that will be resolved for a while.

Meanwhile on the test side of things there is currently no free lunch up for grabs even from the product side let alone our own disipline. MSTest will allow you to specify whether tests should be run in separate containers or with no isolation, but it will not run the tests in paralell whatsoever (Google "mstest paralell" and be convinced).

IMHO this is definitely on the most wanted feature list for VS next (2010) and I hope to see it (haven't had enough time to check it out yet). In my dreams the feature would have these high level properties:

- MSTest switch to specify at which level the paralellism should take place
- The switch would take its value form the tuple {Solution|Assembly|Class|Method} (update: consider even TestList as a value)
- MSTest switch (like for MSBuild) to specify number of nodes (/m:)

It would be left to the developer to guarantee that the test units (not unit tests) at the given granularity observes independence of all other test units on that level. In other words, if I knew (because I have worked my behind of to get it like that) that all my [TestClass] adorned classes were completely isolated at runtime from all other [TestClass] classes, then I would specify /p:Isolation=Class in my test run. MSTest/VSTestHost would then run all tests in paralell on all available nodes and I wouldn't have to sit around looking at three out of four cores doing abolutely nothing while the tests take forever.

Now wouldn't that be great?

Why it is an error for MSBuild to build clean in the same order as it builds regularily

In a previous posting I wrote about the problem you sometimes face when using MSBuild to build VS solution files.

The investigation of this problem is now complete from my side and I have produced a short and sweet demo that illustrates the problem. The following necessary and sufficienct conditions apply:

- Two assemblies (Level0 and Level1)
- Level1 has a project reference to Level0 with Private=false (CopyLocal=false)
- Both projects have set OutputPath to a common directory (CopyLocal is an abomination and I will blog about why in the future)
- Level1 has "Register for COM interop" and [assembly: ComVisible(true)] in assemblyinfo.cs
- Level1 containing a type that inherits from a type in Level0. The inheriting type must have [ComVisible(false)] or the solution will not build.

The problem is of course that the inheritance across the assembly barrier means that the RegisterAssembly() and UnRegisterAssembly() methods of RegisterServices needs to load Level0 in order to resolve the types in Level1, even though the type that needs resolving is not to be registered for COM interop.

Inside the VS IDE this is no problem since VS in fact does build clean in the proper order, as the following output clearly shows:


------ Clean started: Project: Level1, Configuration: Debug Any CPU ------
------ Clean started: Project: Level0, Configuration: Debug Any CPU ------
========== Clean: 2 succeeded, 0 failed, 0 skipped ==========

MSBuild however does not do this properly, as the following ouput shows:

C:\Test\BuildDependencies>msbuild BuildDependencies.sln /t:clean /v:n
Microsoft (R) Build Engine Version 3.5.30729.1[Microsoft .NET Framework, Version 2.0.50727.3053]Copyright (C) Microsoft Corporation 2007. All rights reserved.
Build started 12/30/2008 2:56:51 PM.Project "C:\Test\BuildDependencies\BuildDependencies.sln" on node 0 (clean target(s)). Building solution configuration "DebugAny CPU".Project "C:\Test\BuildDependencies\BuildDependencies.sln" (1) is building "C:\Test\BuildDependencies\Level0\Level0.csproj" (2) on node 0 (Clean target(s)). Deleting file "C:\Test\BuildDependencies\Binaries\Level0.dll". Deleting file "C:\Test\BuildDependencies\Binaries\Level0.pdb". Deleting file "C:\Test\BuildDependencies\Level0\obj\Debug\Level0.dll". Deleting file "C:\Test\BuildDependencies\Level0\obj\Debug\Level0.pdb".EntityClean: Successfully cleaned the output for 0 EDMX files.Done Building Project "C:\Test\BuildDependencies\Level0\Level0.csproj" (Clean target(s)).
Project "C:\Test\BuildDependencies\BuildDependencies.sln" (1) is building "C:\Test\BuildDependencies\Level1\Level1.csproj" (3) on node 0 (Clean target(s)).C:\Windows\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets(928,9): error MSB3395: Cannot unregister assembly "C:\Test\BuildDependencies\Binaries\Level1.dll". Could not load file or assembly 'Level0, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.Done Building Project "C:\Test\BuildDependencies\Level1\Level1.csproj" (Clean target(s)) -- FAILED.
Done Building Project "C:\Test\BuildDependencies\BuildDependencies.sln" (clean target(s)) -- FAILED.
Build FAILED.
"C:\Test\BuildDependencies\BuildDependencies.sln" (clean target) (1) ->"C:\Test\BuildDependencies\Level1\Level1.csproj" (Clean target) (3) ->(UnmanagedUnregistration target) -> C:\Windows\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets(928,9): error MSB3395: Cannot unregister assembly "C:\Test\BuildDependencies\Binaries\Level1.dll". Could not load file or assembly 'Level0, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
0 Warning(s) 1 Error(s)
Time Elapsed 00:00:00.54
C:\Test\BuildDependencies>

Which prooves my point (look closely at the output and you will see that the projects are cleaned Level0 first then Level1 which is the opposite of what VS does). The MSBuild team, while copying the internal MSBuild model from VS, obviously forgot to copy the way VS handles the level info.

MSbuild of VS solution under target clean is just plain wrong

One of our solutions has been bothering us for a long time so I finally decided to make a simple demo of the problem.

In my demo solution I have three library type C# projects Level0, Level1 and Level2. Adding and removing a (project) reference from Level1 to Level0 and building with both default and clean targets using MSBuild illustrates the problem. Beneath is the ouput of the clean build without the reference in place:

C:\Test\BuildDependencies>msbuild C:\Test\BuildDependencies\BuildDependencies.sln /t:clean
Microsoft (R) Build Engine Version 3.5.30729.1[Microsoft .NET Framework, Version 2.0.50727.3053]Copyright (C) Microsoft Corporation 2007. All rights reserved.
Build started 12/29/2008 4:01:39 PM.Project "C:\Test\BuildDependencies\BuildDependencies.sln" on node 0 (clean target(s)). Building solution configuration "DebugAny CPU".Project "C:\Test\BuildDependencies\BuildDependencies.sln" (1) is building "C:\Test\BuildDependencies\Level0\Level0.csproj" (2) on node 0 (Clean target(s)). Deleting file "C:\Test\BuildDependencies\Level0\bin\Debug\Level0.dll". Deleting file "C:\Test\BuildDependencies\Level0\bin\Debug\Level0.pdb". Deleting file "C:\Test\BuildDependencies\Level0\obj\Debug\Level0.dll". Deleting file "C:\Test\BuildDependencies\Level0\obj\Debug\Level0.pdb".EntityClean: Successfully cleaned the output for 0 EDMX files.Done Building Project "C:\Test\BuildDependencies\Level0\Level0.csproj" (Clean target(s)).
Project "C:\Test\BuildDependencies\BuildDependencies.sln" (1) is building "C:\Test\BuildDependencies\Level1\Level1.csproj" (3) on node 0 (Clean target(s)). Deleting file "C:\Test\BuildDependencies\Level1\bin\Debug\Level1.dll". Deleting file "C:\Test\BuildDependencies\Level1\bin\Debug\Level1.pdb". Deleting file "C:\Test\BuildDependencies\Level1\bin\Debug\Level1.tlb". Deleting file "C:\Test\BuildDependencies\Level1\obj\Debug\Level1.dll". Deleting file "C:\Test\BuildDependencies\Level1\obj\Debug\Level1.pdb".EntityClean: Successfully cleaned the output for 0 EDMX files.Done Building Project "C:\Test\BuildDependencies\Level1\Level1.csproj" (Clean target(s)).
Project "C:\Test\BuildDependencies\BuildDependencies.sln" (1) is building "C:\Test\BuildDependencies\Level2\Level2.csproj" (4) on node 0 (Clean target(s)). Deleting file "C:\Test\BuildDependencies\Level2\bin\Debug\Level2.dll". Deleting file "C:\Test\BuildDependencies\Level2\bin\Debug\Level2.pdb". Deleting file "C:\Test\BuildDependencies\Level2\obj\Debug\Level2.dll". Deleting file "C:\Test\BuildDependencies\Level2\obj\Debug\Level2.pdb".EntityClean: Successfully cleaned the output for 0 EDMX files.Done Building Project "C:\Test\BuildDependencies\Level2\Level2.csproj" (Clean target(s)).
Done Building Project "C:\Test\BuildDependencies\BuildDependencies.sln" (clean target(s)).
Build succeeded. 0 Warning(s) 0 Error(s)

Here is the output with the reference (Level1 -> Level0) in place:

C:\Test\BuildDependencies>msbuild C:\Test\BuildDependencies\BuildDependencies.sln /t:clean
Microsoft (R) Build Engine Version 3.5.30729.1[Microsoft .NET Framework, Version 2.0.50727.3053]Copyright (C) Microsoft Corporation 2007. All rights reserved.
Build started 12/29/2008 4:02:14 PM.Project "C:\Test\BuildDependencies\BuildDependencies.sln" on node 0 (clean target(s)). Building solution configuration "DebugAny CPU".Project "C:\Test\BuildDependencies\BuildDependencies.sln" (1) is building "C:\Test\BuildDependencies\Level0\Level0.csproj" (2) on node 0 (Clean target(s)). Deleting file "C:\Test\BuildDependencies\Level0\bin\Debug\Level0.dll". Deleting file "C:\Test\BuildDependencies\Level0\bin\Debug\Level0.pdb". Deleting file "C:\Test\BuildDependencies\Level0\obj\Debug\Level0.dll". Deleting file "C:\Test\BuildDependencies\Level0\obj\Debug\Level0.pdb".EntityClean: Successfully cleaned the output for 0 EDMX files.Done Building Project "C:\Test\BuildDependencies\Level0\Level0.csproj" (Clean target(s)).
Project "C:\Test\BuildDependencies\BuildDependencies.sln" (1) is building "C:\Test\BuildDependencies\Level2\Level2.csproj" (3) on node 0 (Clean target(s)). Deleting file "C:\Test\BuildDependencies\Level2\bin\Debug\Level2.dll". Deleting file "C:\Test\BuildDependencies\Level2\bin\Debug\Level2.pdb". Deleting file "C:\Test\BuildDependencies\Level2\obj\Debug\Level2.dll". Deleting file "C:\Test\BuildDependencies\Level2\obj\Debug\Level2.pdb".EntityClean: Successfully cleaned the output for 0 EDMX files.Done Building Project "C:\Test\BuildDependencies\Level2\Level2.csproj" (Clean target(s)).
Project "C:\Test\BuildDependencies\BuildDependencies.sln" (1) is building "C:\Test\BuildDependencies\Level1\Level1.csproj" (4) on node 0 (Clean target(s)). Deleting file "C:\Test\BuildDependencies\Level1\bin\Debug\Level1.dll". Deleting file "C:\Test\BuildDependencies\Level1\bin\Debug\Level1.pdb". Deleting file "C:\Test\BuildDependencies\Level1\bin\Debug\Level1.tlb". Deleting file "C:\Test\BuildDependencies\Level1\bin\Debug\Level0.dll". Deleting file "C:\Test\BuildDependencies\Level1\bin\Debug\Level0.pdb". Deleting file "C:\Test\BuildDependencies\Level1\obj\Debug\ResolveAssemblyReference.cache". Deleting file "C:\Test\BuildDependencies\Level1\obj\Debug\Level1.dll". Deleting file "C:\Test\BuildDependencies\Level1\obj\Debug\Level1.pdb".EntityClean: Successfully cleaned the output for 0 EDMX files.Done Building Project "C:\Test\BuildDependencies\Level1\Level1.csproj" (Clean target(s)).
Done Building Project "C:\Test\BuildDependencies\BuildDependencies.sln" (clean target(s)).

Now, the point is that allthough adding the reference alters the build order MSBuild does not build the clean targets in the correct order.

If you don't beleive me have a look at the .sln.cache that gets generated (if you use MSBuild V3 and VS2008). Compare the Clean and the Build targets and you will see that the build order is the same (BuildLevel0 is done before BuildLevel1 etc.).

Cleaning a referer after the referee (is that the right term?) will fail in some cases. Specifically it will fail if you need the refered assembly to clean the current assembly.

When do you need the referred assembly to clean the current one? Well our build breaks during clean with the "MSB3395: Cannot unregister assembly" error. It seems our referring assembly needs some type in the referred assembly to properly unregister. I am currently investigating why this is the case but the result doesn't change the fact that MSBuild has an issue here.

It's obvious if you think about it. If you have access to dependency information you should always clean in the opposite order of the way you normally build. Am I the only one getting this?!? Shouldn't this be obvious to the guys in redmond too?!?