Running integration tests in a VSTS release pipeline

As part of our effort to put out code that we have higher confidence in, we decided to add integration testing to our Release process.  The idea, in addition to unit testing during build, was to release the code, then hit that environment and run a series of tests to ensure the API doesn’t error out or generally not work.  Obviously, you can make these tests as thorough or simple as you’d like, but the idea remains the same.

There isn’t a whole lot out there on doing this.  This post does an okay job if you’re using MSTest, however, if you use Xunit like we do, you must use a VS 2017 agent (we use hosted) to run the tests or you’ll end up with this sweet error:

Warning: No test is available in D:\a\r1\a\tests\Tests.Integration.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

I banged my head for an hour trying to figure out why, even though the Xunit DLLs were up there, the tests weren’t being found.  I’ll show you below how to make sure you’re using a 2017 agent.

Solution setup

We decided to include the integration tests in the solution with our API the tests are testing.  This was to keep it easy to update and to keep it in our minds that if we update the API parameters or returns, the test will probably need to be updated.  As such, our solution went from one API project to an API project and a Xunit project.

Don’t mind the blacking out; just removing employer identification namespaces!

The build

Two things to consider in your build pipeline:

  1. Make sure your tests, if any, are not running your integration tests by tightening your search regex
  2. Disable “Publish Web Projects” and have your “Path to project(s)” target csprojs:

The release

Minimal environment tasks

As you can see, the minimal tasks are pretty small and rather straight forward.  First, you keep your release task to kick your app to your Azure resource.  No modifications necessary (if it already existed).

Second, the integration test DLL isn’t unzipped for you.  You’ll need to do that yourself.  I unzipped it into a “tests” folder for easy targeting.

Once they’ve been extracted, you can then target that directory to with the Visual Studio Tests task.  There are a couple important configurations you need to make:

1. Test files: Tighten up your search regex to make sure you’re only firing your integration tests.

2. Path to custom test adapters: The Xunit test adapter is in the $(System.DefaultWorkingDirectory)\tests folder.  You need to specify that folder so vstest.console.exe can run the Xunit tests.

The release agent

Finally, the release agent needs to be running VS 2017.  In your environment tasks, click the “Run on agent” header, and change the “Agent pool” on the right to “Hosted VS2017.”

And that’s it!  Enjoy CD integration testing!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at

Up ↑

%d bloggers like this: