{"id":22767,"date":"2016-10-10T12:11:35","date_gmt":"2016-10-10T19:11:35","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=22767"},"modified":"2019-02-14T15:56:13","modified_gmt":"2019-02-14T23:56:13","slug":"parallel-test-execution","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/parallel-test-execution\/","title":{"rendered":"Parallel Test Execution"},"content":{"rendered":"<p>An <a href=\"https:\/\/johnkoerner.com\/vs2015\/parallel-test-execution-in-visual-studio-2015-update-1-might-not-be-what-you-expect\/\">early post<\/a> on Parallel Test Execution drew attention to its subtle semantics. Three considerations directly contributed to that <strong>(1)<\/strong> Reach <strong>(2)<\/strong> Composability <strong>(3)<\/strong> Non-disruptive roll out.<\/p>\n<p>The Visual Studio Test Platform is open and extensible, with tests written using various test frameworks and run using a <a href=\"https:\/\/visualstudiogallery.msdn.microsoft.com\/site\/search?query=%22Test%20Adapter%22&amp;f%5B0%5D.Value=%22Test%20Adapter%22&amp;f%5B0%5D.Type=SearchText&amp;ac=4\">variety of adapters<\/a>. To reduce on-boarding friction, the feature ought to work on existing test code. It especially needs to work on existing MSTest framework based test code &#8211; there is a huge corpus of such tests already written, and it would be unrealistic to expect users to go in and update their test code just to take advantage of the feature. The feature must acknowledge that adapters for frameworks like NUnit, and xUnit.net already enable parallel test execution. And importantly it must not break existing test runs. Accordingly, the feature took shape as follows:<\/p>\n<ul>\n<li>Parallel Test Execution is available to all frameworks.<\/li>\n<li>It is available from within the IDE, the command line (CLI), and in VSTS.<\/li>\n<li>From within the IDE it is available from all launch points (Test Explorer, CodeLens, various &#8220;Run&#8221; commands, etc.).<\/li>\n<li>The feature composes with test adapters\/frameworks that already support parallel execution.<\/li>\n<li>The feature has a low-friction on-boarding experience \u2013 one that requires no changes to existing test code.<\/li>\n<li>The feature <em>meets the test code where it is<\/em>.<\/li>\n<li>The feature is OFF by default \u2013 users have to explicitly opt-in.<\/li>\n<\/ul>\n<h3>The Solution at Visual Studio Update 1<\/h3>\n<p>Parallel test execution leverages the available cores on the machine, and is realized by launching test execution on each available core as a distinct process, and handing it a container worth of tests (assembly, DLL, or relevant artifact containing the tests to execute) to execute. The unit of isolation is a process. The unit of scheduling is a test container. Within each container, the tests will be executed as per the semantics of the test framework. If there are many such containers, then as processes finish executing the tests within a container, they are handed the next available container.<\/p>\n<p>In effect a coarse-grained level of parallelism is supported. The test platform intentionally leaves fine-grained control over parallelism to the framework of choice \u2013 i.e. it composes over it. Both levels parallelism can co-exist.<\/p>\n<p>The feature is OFF by default.<\/p>\n<p>The <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/jj635153.aspx\">.runsettings file<\/a> is the artifact used to configure how the tests get run, and is common in the IDE, the CLI, and the VSTS workflows. The feature can be turned ON by authoring a .runsettings file with an entry for <strong>MaxCpuCount<\/strong>, and associating that with the test run.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/processparallel.jpg\"><img decoding=\"async\" width=\"696\" height=\"368\" class=\"alignnone size-full wp-image-23705\" alt=\"processparallel\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/10\/processparallel.jpg\" \/><\/a>\nThe value for MaxCpuCount has the following semantics:<\/p>\n<ul>\n<li>&#8216;n&#8217; (where 1 &lt;= n &lt;= number of cores): upto &#8216;n&#8217; processes will be launched.<\/li>\n<li>&#8216;n&#8217; of any other value: The number of processes launched will be as many as the available cores on the machine.<\/li>\n<\/ul>\n<p>Typically, a value of 0 indicates that up to all\u00a0of the\u00a0available free cores may be used.<\/p>\n<h3>Guidance for leveraging Parallel Test Execution<\/h3>\n<p>As mentioned earlier, not all test code already written might be done so in a manner that is parallel-safe. For pure unit tests, it should just work. For other kinds of tests, you will need to experiment a little to see if they are assuming exclusive use of global resources, and refactor\/rearrange them appropriately. In general, the following iterative approach may be used to leverage the feature:<\/p>\n<p>Partition tests in terms of a taxonomy as follows:\n<strong>(1)<\/strong> &#8220;Pure Unit Tests&#8221; (can run in parallel)\n<strong>(2)<\/strong> Functional Tests that can run in parallel with some modifications (e.g.: 2 functional tests trying to create\/delete the same folder can be \u2018fixed\u2019 to remove the assumption that they have exclusive use to the folder).\n<strong>(3)<\/strong> Functional Tests that cannot be modified to run in parallel (e.g.: 2 Coded UI tests doing mouse actions on the desktop, 2 functional tests writing to the Bluetooth\/IR port, etc.)<\/p>\n<p>Gradually evolve the partitioning as follows:\n<strong>a)<\/strong> Run the tests in parallel, and see what tests fail and classify them into (2) or (3) above.\n<strong>b)<\/strong> For tests in (2), fix them so that they can run in parallel.\n<strong>c)<\/strong> For tests in (3), move them out into a separate test run (where parallel is OFF by default).<\/p>\n<h3>Limitations<\/h3>\n<p>Parallel Test Execution is not supported in the following cases:\n<strong>(1)<\/strong> If the test run is configured using <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/jj635153.aspx\">a .testsettings file<\/a>.\n<strong>(2)<\/strong> It is not supported for test code targeting the Phone, Store, UWP application platforms.<\/p>\n<h3>How the feature evolved at Visual Studio Update 2<\/h3>\n<p>We received feedback that enabling parallel test execution using .runsettings was not very discoverable. Also, it is likely that some tests might fail when run in parallel, and our guidance on how to leverage the feature required the user to be able to separate out such tests in an iterative manner &#8211; an exercise that would require switching between parallel and non-parallel execution multiple times during one or more sessions. Therefore, we evolved the feature along the following lines:<\/p>\n<ul>\n<li>Make the feature more easily discoverable.<\/li>\n<li>Make it easy to turn the feature ON\/OFF.<\/li>\n<\/ul>\n<p>Starting with Visual Studio 2015 Update 2, we added the following alternative\u00a0ways to enable Parallel Test Execution in the IDE, CLI and VSTS:<\/p>\n<p><strong>IDE<\/strong>\nParallel Test Execution is surfaced as a button on the Test Explorer.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/parallelInTE.jpg\"><img decoding=\"async\" width=\"347\" height=\"107\" class=\"alignnone size-full wp-image-23706\" alt=\"parallelInTE\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/10\/parallelInTE.jpg\" \/><\/a>\nThis is an ON\/OFF setting.\nIf ON, the value of MaxCpuCount is set to 0.\nThe value is persisted with the solution.\nThe value is merged with any .runsettings file (if one is associated) just before a run begins.\nThe only way to tweak the value of MaxCpuCount is by explicitly adding a .runsettings file.<\/p>\n<p><strong>VSTS<\/strong>\nIn VSTS, &#8220;Execute Tests in Parallel&#8221; is surfaced as a checkbox in the VSTest task.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/parallelInCI2.jpg\"><img decoding=\"async\" width=\"806\" height=\"477\" class=\"alignnone size-full wp-image-23735\" alt=\"parallelInCI2\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/10\/parallelInCI2.jpg\" \/><\/a>\nIf ON, the value of MaxCpuCount is set to 0.\nThe value is merged with any .runsettings file (if one is associated) just before a run begins.\nThe only way to tweak the value of MaxCpuCount is by explicitly adding a .runsettings file.<\/p>\n<p><strong>CLI<\/strong>\nvstest.console.exe supports a <strong>\/Parallel<\/strong> command line switch.\nIf the switch is thrown, the value of MaxCpuCount is set to 0.\nThe value is merged with any .runsettings file (if one is associated) just before a run begins.\nThe only way to tweak the value of MaxCpuCount is by explicitly adding a .runsettings file.<\/p>\n<h3>Merging with .runsettings<\/h3>\n<p>But what if you already had a .runsettings file with MaxCpuCount? Well, the following tables summarize how the .runsettings will be merged:<\/p>\n<p><strong>IDE<\/strong>\n<a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/TEmerge.jpg\"><img decoding=\"async\" width=\"663\" height=\"183\" class=\"alignnone wp-image-23745 size-full\" alt=\"TEmerge\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/10\/TEmerge.jpg\" \/><\/a><\/p>\n<p><strong>CLI\n<\/strong><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/CLImerge.jpg\"><img decoding=\"async\" width=\"717\" height=\"183\" class=\"alignnone wp-image-23755 size-full\" alt=\"CLImerge\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/10\/CLImerge.jpg\" \/><\/a><\/p>\n<p><strong>VSTS<\/strong>\n<a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/VSTSmerge.jpg\"><img decoding=\"async\" width=\"826\" height=\"296\" class=\"alignnone wp-image-23765 size-full\" alt=\"VSTSmerge\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/10\/VSTSmerge.jpg\" \/><\/a><\/p>\n<h3>Summary<\/h3>\n<p>Parallel Test Execution is one of the key features we have shipped as part of the &#8220;<a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/2016\/07\/25\/evolving-the-visual-studio-test-platform-part-1\/\">Efficient Execution<\/a>&#8221; theme, and we hope that\u00a0this post has shed some light on the considerations\u00a0that shaped it.<\/p>\n<p>We are eager to know more about your experience with using this feature. Your inputs have informed the evolution of this and other features in the Test Plattform, so please keep the feedback coming. \u2013 post them as comments on this post, or using <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/mt632287.aspx\">Visual Studio\u2019s \u201cReport a Problem\u201d\/\u201dProvide a Suggestion\u201d<\/a> feature, or using <a href=\"https:\/\/connect.microsoft.com\/\">connect<\/a>, or <a href=\"https:\/\/twitter.com\/pvlakshm\">twitter<\/a>.<\/p>\n<p>Looking forward to hearing from you.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>An early post on Parallel Test Execution drew attention to its subtle semantics. Three considerations directly contributed to that (1) Reach (2) Composability (3) Non-disruptive roll out. The Visual Studio Test Platform is open and extensible, with tests written using various test frameworks and run using a variety of adapters. To reduce on-boarding friction, the [&hellip;]<\/p>\n","protected":false},"author":765,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[226,1,252],"tags":[],"class_list":["post-22767","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci","category-devops","category-testing"],"acf":[],"blog_post_summary":"<p>An early post on Parallel Test Execution drew attention to its subtle semantics. Three considerations directly contributed to that (1) Reach (2) Composability (3) Non-disruptive roll out. The Visual Studio Test Platform is open and extensible, with tests written using various test frameworks and run using a variety of adapters. To reduce on-boarding friction, the [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/22767","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/765"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=22767"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/22767\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=22767"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=22767"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=22767"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}