Possible paralellism scheme in MSTest

While I am sitting here waiting for a test run to finish I might as well vent my frustration and hopefully spark up some ideas that might resonate with the Team Test crew in Redmond.

The possible paralellism in a run of MSBuild seems well under way to be taken care of and is now partly under the control of the user (pending disipline amongst the developers). Only yesterday I switched over to /m:8 on my 4 core build machine just to see what it would do to my build process and it was a sight for sore eyes seeing the 7+1 MSBuild processes firing at will on all four cores.

The only problem I have right now is that our code base is big and therefore divided into several VS solutions. Enabling cleaning of the output then tripped the whole wagon since it seems the solutions are batch-built as a whole. Since we have dependencies between the solutions the build order of the solutions must be observed. Too much goodies then so I switched back to single build process.

I have been working for a while on a single VS solution containing all our projects (8GB, 4Core developer blades with 15K + 10K RPM disks seems to be able to take the pain without too much(!) trouble), but thanks to some legacy(sic!) J# components (needs J#SE and VS2005) that seems hard to get rid of this is not a path that will be resolved for a while.

Meanwhile on the test side of things there is currently no free lunch up for grabs even from the product side let alone our own disipline. MSTest will allow you to specify whether tests should be run in separate containers or with no isolation, but it will not run the tests in paralell whatsoever (Google "mstest paralell" and be convinced).

IMHO this is definitely on the most wanted feature list for VS next (2010) and I hope to see it (haven't had enough time to check it out yet). In my dreams the feature would have these high level properties:

- MSTest switch to specify at which level the paralellism should take place
- The switch would take its value form the tuple {Solution|Assembly|Class|Method} (update: consider even TestList as a value)
- MSTest switch (like for MSBuild) to specify number of nodes (/m:)

It would be left to the developer to guarantee that the test units (not unit tests) at the given granularity observes independence of all other test units on that level. In other words, if I knew (because I have worked my behind of to get it like that) that all my [TestClass] adorned classes were completely isolated at runtime from all other [TestClass] classes, then I would specify /p:Isolation=Class in my test run. MSTest/VSTestHost would then run all tests in paralell on all available nodes and I wouldn't have to sit around looking at three out of four cores doing abolutely nothing while the tests take forever.

Now wouldn't that be great?