Fifth in a series focusing on using JMeter for functional testing of an API.
When is it good for a test fail? When you expect it to!
In the last tutorial we created validation on our Signin Library module to make sure that when we expected the test to pass that we got back the data we expected. Our test would then fail when we expected correct data but got something else. But what about when we need to make sure something correctly fails? What if we want to intentionally provide bad data and make sure that the application returns the correct error?
It’s called negative Validation, and we will tackle it in this tutorial.
Before we begin make sure you have the test plan from the previous tutorial or download this one to use.
Functional Testing 05: Negative Validation or When Bad is Good
- First open your credentials.csv. We are going to add a few users who do not have accounts so that we can test failed logins. We are also going to add a third field to each user to indicate to our test if expect the signin to succeed or fail.
- To each of the existing user/password combinations add “,Success”.
- Add two new lines at the end like so:
Ultron,ShinyBadGuy.Fail
Thanos,PurpleBadGuy,Fail
Your csv should look like this:
- Open your test plan in JMeter and expand Signin Tests > While Controller and select the CSV Data Set Config
- In the variable names field and our field in the csv by appending “,outcome”.
This makes sure our test knows how to read our new Success or Fail field in our credentials file. Now we need to update our Sampler Library to use that outcome value correctly. - Expand Sampler Library > Signin
- Right SignIn and select Add > Logic Controller > If Controller and drag and drop it to right below the HTTP Header Manager.
- In the If Controller rename it to “If success” and set the Expression to be
“${outcome} == “Success”
Also uncheck “interpret Condition as Variable Expression?” - Drag the POST Signin Sampler with its child JSON Assertion into the If Controller. Your test plan should look like this:
- Whenever I am restructuring anything I am working on, I like to take the time to make sure that I haven;t broken anything that was previously working. Based on the changes so far, we should be able to run and see the exact same results we saw last tutorial. So lets go ahead and run the test just to make sure.
You should see Peterparker, TonyStark, SteveRogers, NataliaRomanov, and ClintBarton all pass, Loki Fail, and empty tests for Ultron and Thanos. Great everything is still working! Now lets work on hooking up the tests we expect the fail (Ultron and Thanos). - So our Signin Library module handles the case where outcome equals Success. Let’s teach it to handle the case where outcome equals Fail.
Copy the “If Success” Controller and paste it in the Signin module again.
Yes I know… please see my disclaimer below. - Click the new “If Success” and rename it to “If Fail”.
Set the Expression to
“${outcome} == “Fail”
- Right-click on the JSON Assertion underneath the IF Fail controller and select Disable. This will turn off the assertion for now.
- Let’s go ahead and run our tests again to see what happens.
We can see that now our new Fail case ran and did indeed fail. The server returned a Response code: 401 and Response Message: Unauthorized. This is good news! We are on the right track. But since we expect these tests fail, it would be good if they instead displayed as a Passed test case. Let’s get to work configuring our Library module to understand how to validate a correct Fail. - Right-click on the IF Fail > POST Signin Sampler and select Add > Assertions > Response Assertion. Drag it above the disabled JSON Assertion.
- In the Response Assertion
Check the “Ignore Status” check box in the Field to Test section. This will tell JMeter not to automatically fail the test when it encounters the 401 response code.
Click the “Response Code” radio button in the Field to Test section.
Click the Equals radio button in the Pattern Matching Rules section
Click the Add button at the bottom of Patterns to Test, and add “401” as a pattern
What we have now done is configured the Response assertion to check for a Response Code of 401. If any other Response Code is encounter the test will fail instead of pass. - Copy and Paste the Response Assertion and in the new Assertion
Click the Response Message radio button the Field to Test section
Click the Equals radio button in the Pattern Matching Rules section
Click the Add button at the bottom of Patterns to Test, and add “Unauthorized” as a pattern
Now we will also validate the Response message as well as the code. - Click the Equals radio button in the Pattern Matching Rules section.
Click the Add button at the bottom of Patterns to Test, and add “401” as a pattern - Right click the Disabled JSON Assertion and select Enable.
- Modify the JSON Assertion by
Setting Assert JSON Path to “$.code”
Checking Additionally assert value
Setting Expected Value to 401
- Now run your tests again.
We can see that the two cases that we expected to Fail now show up as Passes (because they failed correctly). If those cases were to fail in any way other than 401 Unauthorized (for instance if they return a 500 Server Error instead) they would now show up as failures.
Yep, I did it again!
You’re right. After pointing out how bad copy and pasting is, and devoting a a whole But Wait! tutorial showing you how to get rid of the previous laziness of copy and pasting, I went and did it again. In this case however, there is very little to be done about it. JMeter does not allow you to nest Logic Controllers underneath a Sampler. If it did we could simply put the IF Controller beneath a singular Signin Http Sampler and do our validations there. Unfortunately we are left with copy and paste of the Sampler itself in this case (without jumping through a ton of hoops and putting in an inordinate amount of effort to work around the limitation).
Sometimes you gotta break the rules because of the tools.
Documentation
Now that the Signin Module has become a bit more complicated I like to include a little bit of documentation so that someone new coming along and having to maintain the tests or use the Library (or when my forgetful self comes along a few months later) has some hints at how to use the Library Modules.
Unfortunately, JMeter doesn’t really provide a good way of including documentation, so we are going to have to re-purpose a different object to our needs.
- Right-click the Signin Library module and select Add > Listener > Beanshell Listener. Drag and drop the Beanshell Listener to the top of the module.
- Right-click the Beanshell Listener and select Disable. This is to make sure it doesn’t run as we are only going to use it for our documentation.
- Rename the Beanshell Listener to “Variables”.
- Input the following information into the Beanshell text field (be sure to use comments just to extra protect against Beanshell trying to run the info as Java code).
We will see the Beanshell Sampler in a later to tutorial and use it there for its intended purpose.
Downloads