Testing Web Services

Over the past year I have been working on several projects that have kept me quite busy. Therefore, it has been a while since I've posted anything to my blog. But during some recent discussions with co-workers, I realized that a post regarding how we used SoapUI as a functional testing framework would be of value to service developers and architects. And that's what this post is going to describe. It is not intended to be a discussion on the pros and cons of testing web services using SoapUI versus JUnit versus other testing frameworks. It is simply a discussion of how we used SoapUI's Test Suite capabilities to validate that our service was meeting its requirements and functioning correctly after deployments.

The Service and How Should We Test It?
The service that we were developing was one needed for basic authentication and user profile management. The data was stored in an Oracle database and there were requirements supplied from the business regarding data validation rules and business logic. Very standard stuff. Once the WSDL and database designs were completed, we discussed the best way to test our service once implementation began. We started developing test cases with JUnit and subsequently testing our deployments with SoapUI. After a very short period of time (and a better understanding of the capabilities of SoapUI), we realized that we were starting to duplicate our testing logic between the two frameworks. After some more consideration, it became apparent that through the creation of SoapUI Test Cases and Test Suites, we could perform the same level of testing as we could writing JUnit tests. The additional benefit that we got by using SoapUI was that we could validate deployments in various environments very quickly and easily. This allowed us to confidently notify service consumers that deployments were available and ready for consumption.

System Requirements
The requirements for using SoapUI for testing this kind of service are quite simple. First, you need SoapUI (obviously). For our needs, the open source version was more than capable. You need a service to test. Our service was deployed to Weblogic, and I have it running locally. If you intend for your Test Suite to clean the database up after your test run completes, you will need the appropriate JDBC driver. The driver needs to be placed in the /bin/ext directory. That's it. At this point, I am assuming you have a SoapUI project set up for your service. If not, it is pretty straightforward to create a project off of a WSDL.

Test Suite Overview
Our service was comprised of about 20 different service operations, some of which write to the database, and all of them read from the database. A test case was developed for each service operation, and the test steps within each test case were developed based off the requirements of the operation. Since the purpose of the service is User Profile management, we needed to define a number of attributes that could be reused across the test cases for testing and assertions. For example, our service has one operation for registering a user and another for getting a users profile data. So the data used for the registration test cases can be used subsequently to validate the return from the get user operation. Also, at the end of the test run, we want to clean up any data that was created and used for the test, so we needed to keep track of users created, etc.

So, the next few sections will describe how to go about creating the Test Suite, defining and using Properties, and provide examples of different Test Steps and how they were developed.

Creating the Test Suite
There are two ways to create a new Test Suite within SoapUI. The first is to right click on the service interface in the left hand pane of the application and select "Generate TestSuite" as shown below:


You will be prompted to select the operations to include in the Test Suite, along with some options for how you want the Test Suite constructed. Using "One TestCase for each Operation" provides a good modular approach to developing your Test Suite, and it also makes reuse and Load Tests easier. If you have been using SoapUI to exercise the web service and have existing requests defined in the Service Interface, you can choose to reuse those requests or you can have SoapUI generate empty requests.

An alternative way to create a Test Suite would be to right-click on the project and select "New TestSuite". You will be prompted for a name and an empty Test Suite will be created. Test Cases can be added by right-clicking on the TestSuite and selecting "New TestCase".

Then within each Test Case, you can define one or more Test Steps. These represent the actual test code that gets executed. Shown below is a screen shot of the RegisterUser Test Case.

As you can see, there are 15 test steps. These map to the requirements that we were implementing against. They are executed in the order shown, and their order can be changed by right-clicking on a Test Step and selecting either Move Up or Move Down.

SoapUI supports numerous types of test steps including: SOAP, Groovy, Property Transfer, Property Steps, Conditional Gotos, and Delays. In our project we used SOAP steps, Property Transfer steps and Groovy Script steps. More details on the various Test Steps we used and how we used them can be found later in this post.

Defining Test Properties
As mentioned earlier, we need to define a number of properties that can be reused across test cases and assertions. These properties are defined at the Test Suite level.

Once we have a Test Suite, we can define properties that can be shared and reused across Test Cases within the Test Suite. If you open the Test Suite editor by double-clicking on the Test Suite in the left hand pane, you will see the following:


By clicking on the "Properties" button on the bottom of the window, you can define Test Suite properties. Below is an example of some of the properties we defined:


At the top of the Properties window, there is a button that lets you add or remove property name/value pairs. Here you can also import property values from a file. Our approach was to define all the properties we needed ahead of time (or at least as much as we knew about), and access or set their values as needed throughout the test cases.

This approach worked pretty well, except that we realized that if developers were executing the test suite concurrently we could have data collisions. Especially with attributes like username and email address, which are unique across the system. So we needed a way to provide some level of uniqueness during the life of a test execution. We did this via the "Setup Script" option for the Test Suite. SoapUI will execute the Setup Script upon each Test Suite execution. It will also execute the TearDown Script upon completion of the test run.These Setup and Teardown scripts are Groovy based scripts . So, we wrote a little script to append the current time in millis onto some property (called "coreName") and placed that in the SetupScript section of the Test Suite, as shown below.

Within the Setup script, you have access to certain variables including log, context, and testSuite. These variables give you access to the SoapUI framework to enable logging and access to test steps, properties, etc. As you can see above, the script retrieves the "coreName" property from the Test Suite properties, appends the current time in millis, and then sets the value of the "baseName" property. This baseName property is used in the creation of usernames and email addresses in numerous Test Cases and assertions throughout the Test Suite. The green arrow at the top left of the script editor window allows you test the script.

Test Steps

SOAP Test Step
As mentioned earlier, we used 3 different types of test steps throughout our Test Suite. Given we are testing web services, the most used test step type is the SOAP test step. A screen shot of the SOAP Test Step screen is shown below.


There are several areas of interest on this screen. The top area has a number of buttons for creating assertions, default requests based off the WSDL, executing the test steps, etc. There is also a drop down that allows you to define what endpoint to which to send the request. This gets pre-populated based on the WSDL.

The left-middle pane of the window shows the request payload for the SOAP request. You can see that there are some property placeholders being used in this request. When using Test Suite scoped properties, the format is: "${#TestSuite#}". So you can see here that we are attempting to call the UserLogin operation, passing in the username property as the username and password request parameters.

The bottom portion of the window shows the assertions that we have defined for this test case. As with other testing frameworks, if the assertion fails, the test step fails. How this failure is handled by SoapUI is defined in the Test Step options. These can be accessed via right-clicking on the Test Step in the left hand pane of the SoapUI window. The option window is shown below.

As you can see, I have my test cases failing on error and aborting on error. Typically in my scenario, if one step failed, the following steps would usually fail, so I let the entire test case abort.

SoapUI supports a number of assertion types. The ones we typically used were SOAP response, ensuring no SOAP faults, and XPath expressions. There are many ways to define assertions and each developer has there own technique, but we have learned a lesson or two along the way. Initially we started out by using XPath to validate each of the return elements in the Response payload. So we would have an XPath expression that would extract a particular node of the response and we would test for a certain value. The XPath configuration window is shown below:



Here we can see that the top portion defines the XPath expression to execute against the response. In the top portion of the window, we declare any namespaces and define our XPath expression. In the bottom portion of the window, the expected result is entered. There are two actions that can be performed in this window against the XPath expression. The first is "Select from Current". This will execute the XPath expression against the response (this means that the request needs to be submitted), and place the result in the expected result window. This is handy to validate your XPath expression, not to mention having the expected result filled in for you. The next action that can be performed is the "Test" action. This will test the XPath expression against the expected result and assert whether they match. If they do not match, the difference will be indicated.

In the assertion above, we are looking at the alt-email node in the response. We know that there are 2 alt-emails for this user (since we registered the user in an earlier test case). The interesting thing here is that we do not know which alt-email will be returned first in the response payload. There is no real way to distinguish between the nodes. Therefore, we put in an "or" and specify that the first or second alt-email matched the pre-defined Test Suite property "${#TestSuite#username}${#TestSuite#alt1EmailSuffix}". The expected result of this XPath expression is "true". We also defined another assertion, comparing the alt-email nodes with the second alt-email Test Suite property. This way we are ensuring that both alt-email addresses come back in the response payload for this operation. Another example of an XPath expression would be looking for a specific value in the response payload, as shown below:



Here, we expect the code element to have a text value of "US-02000".

One more intersting option on the assertion window is the "Allow Wildcards" checkbox. If we do not care about specific values being returned in the response, we can replace those values with an asterisk, and the assertion will pass regardless of what is contained there. This can be seen below:


Here, we don't care about the actual alt-email addresses coming back, we just care that there are two of them and they both have a validated attribute of "true".

One lesson learned here was that it was easy to get lazy and have our XPath expression select the entire response payload and then replace the dynamic fields in the expected reponse with the Test Suite property placeholders. This made the test step development quicker and easier, but if the schema changed, we had to go through all the effected responses and reformat them. Now this would have been the case had we validated each node independently in the response, but the impact of the changes would be more focused and easier to manage.

Property Transfer Steps
Property Transfer steps allow you to extract a value from a source (SOAP request/response payload, another property, etc.) and place it into a target (SOAP request/response payload, another property, etc.). We used these to pull values from SOAP responses and place them into properties that could be used in subsequent Test Steps.

In our scenario, we had requirements that allowed a consuming application to request a login token that could be used in a "remember me" capacity. Thereby allowing an application to log the user in by using the token in place of username/password credentials. So we needed a test step to test the create token operation, then store that token for testing a subsequent login operation. This was accomplished using the Property Transfer step that retrieved the token from the response payload of the CreateToken test step and stored it in a property called "loginToken". The configuration for that step is shown below:

As you can see, we extract the response value using an XPath expression and define the target as the defined Test Suite property "loginToken". This window allows multiple property transfer definitions and also allows the testing of the transfer by clicking the green arrow button at the top of the window. The source drop down allows you to select any test step defined in this test case, and the target drop down allows the same. This is one area where the order of your test steps can come into play. The response where we are extracting the token-value from is executed before this test step. The step that requires this token is following the property transfer test step.

Groovy Script Test Steps
In the test case described above, we requested a token in one test step, used a property transfer step to store the response, and then performed a login using the token in another test step. An additional requirement stated that if the token has expired, the login should fail. Since the token expiration can only be set to be some future time, we needed a way to manipulate the underlying data. We used a Groovy Script Step to access the database and update the expiration time of the token to some time in the past. The code is shown below:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import com.eviware.soapui.SoapUI;
import com.eviware.soapui.model.testsuite.*;

def Connection con = null;
def Statement stmt = null;
def ResultSet rs = null;
try{
def tokenValue = testRunner.getTestCase().getTestSuite().getPropertyValue("loginToken");

def propertyStep = testRunner.getTestCase().getTestSuite().getTestCaseByName("ConnectionProperties").getTestStepByName("ConnectionProperties")
def dbDriver = propertyStep.getPropertyValue("dbDriver");
def dbConnString = propertyStep.getPropertyValue("dbConnString");
def dbHostName = propertyStep.getPropertyValue("dbHostName");
def dbHostPort = propertyStep.getPropertyValue("dbHostPort");
def dbServiceName = propertyStep.getPropertyValue("dbServiceName");
def dbConnection = dbConnString + dbHostName + ":" + dbHostPort + ":" + dbServiceName
def dbUsername = propertyStep.getPropertyValue("dbUsername");
def dbPassword = propertyStep.getPropertyValue("dbPassword");

log.info("Connecting to " + dbConnection);
Class.forName(dbDriver);
con = DriverManager.getConnection(dbConnection, dbUsername, dbPassword);
stmt = con.createStatement();
def sql = "update token set expire_date = to_date('2007-01-01 12:00:00 AM', 'YYYY-MM-DD HH:MI:SS AM') where token_value='" + tokenValue + "'"
count = stmt.executeUpdate(sql)
log.info("statement = " + sql)
log.info("updated " + count + " rows in token table")
}catch(SQLException e){
log.info(e.getMessage());
}catch(ClassNotFoundException e){
e.printStackTrace();
}finally{
try{rs.close();}catch(Exception e){}
try{stmt.close();}catch(Exception e){}
try{con.close();}catch(Exception e){}

}

As you can see in the code above, we are accessing both Test Suite scoped properties as well as properties defined in another test case. We decided to define our database connection properties in a separate test case/test step. This was done to make it easier to manage the connection properties for multiple environments. We could simply define a property test step for each environment, and by simply renaming the test step to the standard "ConnectionProperties" test step name, the appropriate connection properties would be used across the test suite. Also, the loginToken property that was defined and used in a previous test steps, is being reused again. This is where defining properties comes in handy.



This test step sets the expiration of the token to some time in the past, so the following test step can validate that attempting to login with an expired token will fail.





Another way in which we used Groovy Scripts was to clean up the data we created and manipulated during our test execution. This script uses property values defined in previous steps to delete the appropriate records from the database. This is the final test step executed in this Test Suite.





Putting it All Together


After going through and implementing all the SOAP Steps, Property Transfer Steps, Groovy Steps, etc. we finally have a Test Suite. Here is a portion of what the Test Suite looks like (due to size, I couldn't fit it all).

If we execute the Test Suite by clicking the green arrow at the top of the window, we can see the results. Here is a portion of the results from my last run:








You can see that some test cases passed and some failed. Since some failed, the entire suite failed. In order to determine what failed, you can double-click on the Test Case and it will bring up the test case editor.





We see here that the 5th test step failed, and we can drill into that by double clicking on it.





Here, we can see that the 3rd assertion failed, lets take a look at what the problem was. You can see a description just below the assertion indicating what failed. The message below is what you would read if you could see the entire string.





XPathContains assertion failed for path [declare namespace ns='http://www.foo.com/schemas/user/1.1';

declare namespace xsd='http://www.foo.com/services/user/1.1/xsd';


//xsd:login-success] : Exception:org.custommonkey.xmlunit.Diff


[different] Expected attribute value 'LASTNAME' but was 'FIRSTNAME' - comparing at /login-success[1]/login-user[1]/attribute[1]/@name to at /login-success[1]/login-user[1]/attribute[1]/@name





This says that the reponse expected a node with the attribute value of "LASTNAME", but actually received "FIRSTNAME". If we look at the details of our assertion, we see that the laziness I mentioned earlier came back to haunt us.





The basic problem is that there are multiple "attribute" nodes for a given user. These attribute nodes have a "name" attribute that defines the attribute. Since we are asserting the entire response payload, and these attributes can come back in any order, there is no guarantee that the response payload will match our assertion. The proper way to validate this test step would be to create an assertion for each node to ensure the response payload contains everything we expect.





Eventually after cleaning up all the test cases, we can get a test suite that executes cleanly. Also, SoapUI provides a command line interface and maven plugins, so that these tests can integrated into your build environments as well.





Summary


SoapUI provides a robust set of capabilities to test web services not only during development, but also to test the validity of deployments. As with good testing practices, it does require upfront planning and good test case definition.





Not only were we able to test the functionality of our web services during development, but we were able to deploy and validate new versions of our service in a manner of minutes. This was a great time saver and allowed us to confidently declare our web services as available to consumers.

Thanks to Steve Smith for the original  post.

---

evolving through...