Load Tests – Recovering from SSO failures

Recently, we started to receive issues from our company’s SSO.  For the past six months or so we’ve been load testing, they have had an impeccable record. However, starting this release, we’re getting a handful of users that get stuck in a state where we have an access token, but getting a bearer token from the service fails.  This results in a weird state in our load test where they technically pass authentication, but fail identity.  Since a “user” in the load test persists session between iterations, that meant those users would continue to fail their tests over and over again, resulting in thousands of exceptions on the server, affecting response time for other, working users.

What we wanted to do was detect this issue in the web test, then clear out cookies so the user effectively came in clean and could retry SSO.  The issue that we faced, which we didn’t realize we’d face going in so it took us longer than we anticipated, was that context parameters renew every iteration.  So, we had to find a mechanism that persisted values between iterations but stayed within the context of the user with the issue.  Fortunately, this StackOverlow question provided us a solution for that.

Step 1 – Create a web test request plugin to capture the issue

There is no web request plugin that will update a parameter based on the outcome of the previous response.  We could have made a much more dynamic plugin, but this one served our purposes just fine.

public class SetContextParameterTrueByResponseCode : WebTestRequestPlugin
    {
        /// <summary>
        /// Gets or sets the duration in milliseconds.
        /// </summary>
        /// <value>
        /// The duration in milliseconds.
        /// </value>
        public string ContextParameter { get; set; }

        /// <summary>
        /// Gets or sets the condition.
        /// </summary>
        /// <value>
        /// The condition.
        /// </value>
        [DisplayName("Condition to set the context parameter")]
        [DefaultValue(StringComparisonOperator.Inequality)]
        public StringComparisonOperator ComparisonOperator { get; set; }

        /// <summary>
        /// Gets or sets the response code.
        /// </summary>
        /// <value>
        /// The response code.
        /// </value>
        [DefaultValue(WebTestResponseCode.Ok)]
        public WebTestResponseCode ResponseCode { get; set; }

        /// <summary>
        /// When overridden in a derived class, this method is run every time a request finishes before dependent requests are run. This allows the callback to obtain runtime information about the request.
        /// </summary>
        /// <param name="sender">The object that fired the event.</param>
        /// <param name="e">A <see cref="T:Microsoft.VisualStudio.TestTools.WebTesting.PostRequestEventArgs" /> object.</param>
        public override void PostRequest(object sender, PostRequestEventArgs e)
        {
            if (e.Response == null) return;

            var userContext = e.WebTest.Context["$LoadTestUserContext"] as LoadTestUserContext;
            var set = ComparisonOperator == StringComparisonOperator.Equality && e.Response.StatusCode == (HttpStatusCode)ResponseCode
                || e.Response.StatusCode != (HttpStatusCode)ResponseCode;

            if (set)
            {
                e.WebTest.AddCommentToResult($"Setting context parameter '{ContextParameter}' to true.");

                if (e.WebTest.Context.ContainsKey(ContextParameter))
                    e.WebTest.Context[ContextParameter] = true;
                else
                    e.WebTest.Context.Add(ContextParameter, true);

                if (userContext != null)
                {
                    if (userContext.ContainsKey(ContextParameter))
                        userContext[ContextParameter] = true;
                    else
                        userContext.Add(ContextParameter, true);
                }
            }
        }
    }

What you’ll notice in this class is that we try to update both web test context parameters and the user context parameters.  While we weren’t trying to overdo this for our purposes, we also wanted to account for both in the case we reuse this and we wanted to key off a context parameter mid-instance run.

Obviously, we could have done much more with this plugin, but like I said, it served what we needed.

Step 2 – Create a web test plugin to clear cookies

With the flag being set, we could now clear the cookies next time the instance started up.  Initially, we thought the post-web test overrides would fire even in the event of a run failing, but that is not the case.  It seems if a run fails, it just exits out and starts up a new one, which is somewhat disappointing and led us to search for a way to persist values between runs.  We already had a clear cookies plugin, so we just expanded what we had.

using System.Collections.Generic;
using System.Net;
using Microsoft.VisualStudio.TestTools.LoadTesting;
using Microsoft.VisualStudio.TestTools.WebTesting;

namespace LoadTest.Web.WebTestPlugins
{
    public class ClearCookies : WebTestPlugin
    {
        /// <summary>
        /// Gets or sets a value indicating whether [clear before test run].
        /// </summary>
        /// <value>
        ///   <c>true</c> if [clear before test run]; otherwise, <c>false</c>.
        /// </value>
        public bool ClearBeforeTestRun { get; set; }

        /// <summary>
        /// Gets or sets a value indicating whether [clear after test run].
        /// </summary>
        /// <value>
        ///   <c>true</c> if [clear after test run]; otherwise, <c>false</c>.
        /// </value>
        public bool ClearAfterTestRun { get; set; }

        /// <summary>
        /// Gets or sets the clear after test run context parameter.
        /// </summary>
        /// <value>
        /// The clear after test run context parameter.
        /// </value>
        public string ClearAfterTestRunContextParameter { get; set; }

        /// <summary>
        /// When overridden in a derived class, represents the method that will handle the event associated with the end of a Web performance test.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">A <see cref="T:Microsoft.VisualStudio.TestTools.WebTesting.PostWebTestEventArgs" /> that contains the event data.</param>
        public override void PostWebTest(object sender, PostWebTestEventArgs e)
        {
            _run(e.WebTest, ClearAfterTestRun);
        }

        /// <summary>
        /// When overridden in a derived class, represents the method that will handle the event associated with the start of a Web performance test.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">A <see cref="T:Microsoft.VisualStudio.TestTools.WebTesting.PostWebTestEventArgs" /> that contains the event data.</param>
        public override void PreWebTest(object sender, PreWebTestEventArgs e)
        {
            _run(e.WebTest, ClearBeforeTestRun);
        }

        private void _run(WebTest webTest, bool staticClearValue)
        {
            var userContext = webTest.Context["$LoadTestUserContext"] as LoadTestUserContext;

            var clear = staticClearValue 
                        || _getContextValue(webTest, userContext, "user") 
                        || _getContextValue(webTest, webTest.Context, "web test");

            if (clear)
            {
                webTest.AddCommentToResult("Resetting cookie container.");
                webTest.Context.CookieContainer = new CookieContainer();

                if (!string.IsNullOrWhiteSpace(ClearAfterTestRunContextParameter))
                {
                    if (webTest.Context.ContainsKey(ClearAfterTestRunContextParameter))
                    {
                        webTest.AddCommentToResult("Resetting web test variable to false.");
                        webTest.Context[ClearAfterTestRunContextParameter] = false;
                    }

                    if (userContext?.ContainsKey(ClearAfterTestRunContextParameter) == true)
                    {
                        webTest.AddCommentToResult("Resetting user context variable to false.");
                        userContext[ClearAfterTestRunContextParameter] = false;
                    }
                }
            }
        }

        private bool _getContextValue(WebTest webTest, IDictionary<string, object> context, string contextType)
        {
            var value = false;
            var keyFound = false;

            if (!string.IsNullOrWhiteSpace(ClearAfterTestRunContextParameter))
            {
                webTest.AddCommentToResult($"Checking for existence of '{ClearAfterTestRunContextParameter}' in {contextType} context.");

                if (context.ContainsKey(ClearAfterTestRunContextParameter))
                {
                    webTest.AddCommentToResult($"Context parameter '{ClearAfterTestRunContextParameter}' does not exist in {contextType} context.");
                    keyFound = true;
                }

                if (keyFound)
                {
                    var storedValue = context[ClearAfterTestRunContextParameter];
                    webTest.AddCommentToResult($"Value of '{ClearAfterTestRunContextParameter}' is '{storedValue ?? "[no value]"}' in {contextType} context");

                    if (storedValue is bool boolValue && boolValue)
                        value = true;
                }
            }

            return value;
        }
    }
}

All this guy does is check to see if

  1. Someone statically said “yes, clear cookies”
  2. A context parameter was set to true
  3. A user context parameter was set to true

If that is true, then we instantiate a new CookieCollection and set that to the context’s property.

Step 3 – Triggering these two plugins

With these two created (and make sure you build your project before you try to use the plugins), you simply add the SetContextParameterTrueByResponseCode plugin to any web request and add the ClearCookies plugin to the web test itself.  The two will take care of the rest.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: