soapUI: A Simple Data-Driven Testing Technique

Note: If you experience any issues with webservicex.net or the periodic table service used in this post, or would just like to see another example, I've provided an alternative post covering pretty much the same material with a different service here.

For this post, I'd like to walk through an example project based on the periodic table service used in previous posts to illustrate the use of Groovy script.  You can download the project and a data file used by the project here. After unzipping the files, open soapUI and use File - Import Project to pull the project (xml file) into your workspace. You can put the TestData.csv file anywhere you want, really, but its default location in the script is a directory called C:\PerTableData-- you'll have to modify one line in the project if you want to put it somewhere else (note: to see one way to modify this project to use an Excel data source instead of CSV, see my post here).  Here's what the project tree should look like:


Example project tree

The four operations provided by the service (GetAtomicNumber, GetAtomicWeight, GetElementSymbol, and GetAtoms) are grouped into two separate test cases.  The GetAtoms operation has no request parameters and is in its own test case.  The three remaining operations are grouped together in the ElementSpecificRequests test case for several reasons.  All of them use a single element name as their only request parameter, and there's some overlap between the data returned by the operations (the GetAtomicNumber request returns atomic weights and symbols, which are also returned by the GetAtomicWeight and GetElementSymbol requests, respectively).  Ideally, of course, we'd like to test these three "element specific" operations using more than one symbol, so they're all candidates for data looping (repeating the test requests with different input parameters), too.

Our strategy for data looping is as follows:

1) Create a data file that provides request input parameters (element names, in this case) AND the expected response data for each input so we can perform content assertions.
2) Read in a single line of the file and populate test case properties with the data.
3) Perform our test requests and assertions using the test case properties we populated from the file.
4) Repeat steps 2 and 3 until we reach the end of our file (i.e., no more lines to read in).
5) End the test case.

The TestData.csv file is our source file for step 1.  I personally prefer using csv files for this purpose because they generally have a clear delimiter and they're easy to create and maintain in spreadsheet programs like Excel.  Here are the contents of TestData.csv; each line consists of an element's name, atomic number, symbol, atomic weight, and boiling point (you may notice some errors-- these are there to simulate test step "fails" later on):

Hydrogen,1,H,1.00797,20.400000000000002
Carbon,6,C,12.0115,51000
Oxygen,8,O,15.9994,90.2
Gold,79,Gd,196.967,3239
Uranium,92,U,238.03,4091

For data looping there's also a test case option that we want to change.  If you launch the Options dialog (right-click on a test case in the navigator and select Options from the context menu), you'll see that there's an option called "Abort on Error".  We want to make sure this is unchecked-- otherwise, execution will stop and move on to the next test case the first time an error occurs; obviously, we want the test case to continue looping even if an error occurs during one loop.

The Groovy script to start reading our csv file isn't contained in a test step-- instead, it's in the ElementSpecificRequests test case's Setup script.   At the test suite and test case levels, Setup and TearDown scripts are available-- the Setup script is run prior to any of the test steps of the test case (or prior to any of the test cases when its at the suite level), and the TearDown script is run after the last test step (or test case for suite level TearDown scripts) has completed.


The Test Case Editor with the Setup Script shown (note the button to show the TearDown Script, too)

Here's our Setup Script:

//Create a new filereader object, using the context variable so it can be used between test components
context.fileReader = new BufferedReader(new FileReader("C:\\PerTableData\\TestData.csv"))
//Read in the first line of the data file
firstLine = context.fileReader.readLine()
//Split the first line into a string array and assign the array elements to various test case properties
String[] propData = firstLine.split(",")
testCase.setPropertyValue("ElName",propData[0])
testCase.setPropertyValue("AtNum",propData[1])
testCase.setPropertyValue("Symbol",propData[2])
testCase.setPropertyValue("AtWeight",propData[3])
testCase.setPropertyValue("BoilPoint",propData[4])
//Rename request test steps for readability in the log; append the element name to the test step names
testCase.getTestStepAt(0).setName("GetAtomicNumber-" + propData[0])
testCase.getTestStepAt(1).setName("GetAtomicWeight-" + propData[0])
testCase.getTestStepAt(2).setName("GetElementySymbol-" + propData[0])

The first (non-comment) line creates a FileReader object to read our data input file (if you put the csv file into a location other than C:\PerTableData, you need to modify this line accordingly).  Note that our fileReader object is created as a property of the context variable.  The context variable is readily available in Setup and Teardown Scripts-- no need to declare it or otherwise initialize it.  It provides a way to share objects, variables, etc., across the relevant run context.  In this case, the fileReader object will be available across the test steps of our test case-- handy because we'll need to keep track of where we are in our file as we read in each line.

In the remaining lines of the script we read in the first line of our file (line 4), split the line into a string array (line 6), and then assign the elements of the array to our test case properties (lines 7 - 11) using setPropertyValue(), a method of the testCase object.  In lines 13 - 15 we're actually renaming the test steps by appending element names to their default names-- this is not necessary, really, but helpful for reading the run log; otherwise, we'd see identical test step names for each iteration.  Note the use of the testCase variable in lines 7 - 15; like the context variable, this is automatically available for use in the Setup Script.

After our Setup Script runs, our test requests are sent and verified against content assertions.  We use property expansions with our test case properties for request parameters and assertion criteria:


GetAtomicNumber test request with an assertion-- note the property expansions used for the ElementName element in the request and the AtomicNumber element in the assertion

So after reading in the first line of our data file, the GetAtomicNumber test request gets sent using "Hydrogen" as the ElementName; the response is checked to see if it contains "1".

After our test requests, the next step in the test case is the ReadNextLine Groovy test step.  Here's the script contained in this test step:

/*Read in the next line of the file
  We can use the same fileReader created in the Setup script because it
  was assigned to the context variable.*/
nextLine = context.fileReader.readLine()
/*If the end of the file hasn't been reached (nextLine does NOT equal null)
  split the line and assign new property values, rename test request steps,
  and go back to the first test request step*/
if(nextLine != null){
 String[] propData = nextLine.split(",")
 curTC = testRunner.testCase
 curTC.setPropertyValue("ElName",propData[0])
 curTC.setPropertyValue("AtNum",propData[1])
 curTC.setPropertyValue("Symbol",propData[2])
 curTC.setPropertyValue("AtWeight",propData[3])
 curTC.setPropertyValue("BoilPoint",propData[4])
 curTC.getTestStepAt(0).setName("GetAtomicNumber-" + propData[0])
 curTC.getTestStepAt(1).setName("GetAtomicWeight-" + propData[0])
 curTC.getTestStepAt(2).setName("GetElementSymbol-" + propData[0])
 testRunner.gotoStep(0)
}

Most of this should look pretty familiar-- mostly, we're doing the same things we did in our Setup script.  In line 4 we're reading in the next line in our data file.  If the next line doesn't equal null (which is what would be returned if we had reached the end of the file), we go through the same steps we went through before: splitting the line up into a string array (line 9), assigning the elements of that array to different test case properties (lines 11 - 15), and re-naming the request test steps (lines 16 - 18).

Two new things: the last line of the script sends test case execution back to the first step in our test case, the GetAtomicNumber request.  Also note the use of the testRunner variable.  This is another one of those variables like the context variable that's ready to use from the start.  The testRunner variable is important in soapUI scripting as it's used as an entry point to get to other test objects and control execution.  Case in point: the testCase variable that was available as a "built-in" variable in our Setup script is not immediately available in our Groovy test step.  Instead, in line 10 we use our testRunner variable to get a reference to the test case.  Once we have the test case, we can get its test steps and their properties and methods, and so on.

So execution passes back to the first test step and our test requests are re-run, this time using property values read in from the second line of our data file for request parameters and assertion criteria.  This loop continues until we reach the end of the file-- when this happens, the next line read in is null, so the if statement in line 8 of the ReadNextLine Groovy step fails and execution is never passed back to the first test step.

Finally, before the test case ends its TearDown script is run:

//Cleanup: rename test steps to their generic names and close the file reader
testCase.getTestStepAt(0).setName("GetAtomicNumber")
testCase.getTestStepAt(1).setName("GetAtomicWeight")
testCase.getTestStepAt(2).setName("GetElementSymbol")
context.fileReader.close()

In this script we're just renaming the request steps back to their default names and in the final line closing our data file for reading.

Try running the test suite-- double-click on the test suite node in the navigator to launch the TestSuite Editor, then click the Run button in the editor window.  You should see the progress bar for the ElementSpecificRequests test case fill up multiple times, turning red after an error is encountered.  Once the test cases have completed running, open up the TestSuite Log.


The TestSuite Log for our periodic table test suite (with run results)

We can see right away that something failed for the GetAtomicNumber test request-- and because we were modifying the test step names as we went, we can see it occurred for the carbon element.  If you want to see more information about the failure, you can simply click on the error to launch the Message Viewer.  Here you can see the request and response XML (including the values resulting from property expansions) to get a better idea of where the problem may be.  In this case, when we compare the response with our error above we can see that our assertion was expecting a boiling point of 51000 but our response returned a value of 5100.


Message Viewer response

That illustrates one way to perform data looping with the soapUI free version.  It should be noted that the commercial version of soapUI has features to make data looping much easier with less scripting required.  If data looping is a feature you intend to use heavily, it may be worth it to weigh the cost of the commercial version against potential savings in time spent on script development and maintenance.

45 comments:

  1. Again - very well written! This is great work you did here in explaining it. Again I looked at various sites and they overwhelm with code and fail to break it down speaking for even beginners to understand.

    Thank you
    Rob

    ReplyDelete
  2. Not seeing how: context.fileReader.close()
    works.
    Problem I have using CSV is I use zipCode that starts with '0' and it cuts it out.

    Using .txt it works fine - but running the test, the loops works perfectly.

    After I run one testcase scenario and run another I get a "testCase failed [org.apache.xmlbeans.impl.values.XmlValueDisconnectedException:org.apache.xmlbeans.impl.values.XmlValueDisconnectedException]

    To fix I have to close SoapUI and reOpen.

    ReplyDelete
  3. Sorry you're having problems-- I've been unable to replicate this particular error so far, but a Google search on "soapUI" and this particular exception led me to posts like this one: http://apppsrv01.eviware.com/forum/viewtopic.php?f=2&t=12849&p=31333. It seems this error occurs when you try to access test components that have changed or been removed. Just to be clear, your script goes through one iteration of a test case (after reading a line from your data file), but you get this message when it attempts to move on to the second iteration/line of data?

    ReplyDelete
  4. Very well explained.
    Is it possible to use an excel instead of CSV.
    If so how would I retrieve the data from excel?

    ReplyDelete
    Replies
    1. I believe so-- that's actually a great idea for a follow-up post so I'm working on that now. To do it, I think you'd have to use some external Java solution for reading Excel files (one such tool is Apache POI). 1) Download any requisite jar files for your chosen Excel tool, 2) copy jar files to the bin\ext directory in your soapui installation folder, and 3) reference needed resources in your script using an import statement. You'd have to use your Excel reader's corresponding methods where I used methods in my script to read and parse the source CSV file.

      Again, I'm working on a post to illustrate this and get into the details. I tried using JExcel, but ran into a snag with reading numbers as strings; JExcel seems to round the numbers, preventing exact matches in some cases (in this script, for example, atomic weights are getting rounded to three numbers when read from the Excel source, but the SOAP responses return values to four decimal places-- assertions fail as a result). JExcel may work for you if you're not dealing with numbers that go out beyond three decimal places, but I'm looking at POI now to see if it handles this scenario any better.

      Delete
  5. Interesting!
    I'm a beginner in scripting and till now all attempts results is near 0.
    Did not work and by analogy with your example.

    I need to create testcase for ReverseGeocode methode of Bing maps soap geocoding service in SoapUi Free.
    http://dev.virtualearth.net/webservices/v1/metadata/geocodeservice/geocodeservice.wsdl

    I send in request to service 3 parameters: AppID key for working with Bing maps soap service and two coordinates, latitude and longitude. Service returns address.

    Example:







    ${Datasource#AppID}


    ${Datasource#Latitude}
    ${Datasource#Longitude}






    I have made xls and csv files with parameters. Then I think load data from files and assign input values of parameters to variables: AppID, Latitude, Latitude. I'am trying to write a script on groovy, which reads xls or csv file, using examples like yours, but all they didn't work.
    May you council please the steps of reading data by groovy script like in your script:

    "//Create a new filereader object, using the context variable so it can be used between test components
    //Read in the first line of the data file
    //Split the first line into a string array and assign the array elements to various test case properties." and so on.

    ReplyDelete
  6. Sorry, I'm not sure I'm following... I've checked that WSDL URL you provided and couldn't seem to get a valid response within soapUI when trying to generate a new project. Also, your property expansion expressions above look a little odd to me.

    To use property expansion expressions, you're generally referencing properties at some specific level in your script-- the test suite level, the test case level, etc. I'm not sure what "Datasource" is referencing in your example-- is this the name of a specific test step? In my script structure, you'd 1) create AppID, Latitude, and Longitude properties at the test case level, 2) copy values from your source file to those properties (how to do this-- reading in a line, splitting the line, etc.-- is illustrated in the Setup script at the test case level and the ReadNextLine Groovy script step), then 3) reference those properties in your test requests using "${#TestCase#AppID}", etc.

    ReplyDelete
  7. Thank's! I'll сontinue trying with your council.
    Valid response from service comes if request similar like this
    https://docviewer.yandex.ru/?url=ya-disk%3A%2F%2F%2Fdisk%2Frequest.txt&name=request.txt&c=514abd0b91ea

    ReplyDelete
    Replies
    1. valid link is http://yadi.sk/d/X9o4_i1s3QghI

      Delete
  8. Replies
    1. Thanks so much for the positive feedback! I hope the information is helpful for you...

      Delete
  9. Hi
    You can find more easy steps at below link

    http://lgsofttest.blogspot.in/2013/04/datadriven-testing-from-soapui-using.html

    ReplyDelete
    Replies
    1. I believe the link provided by Sunnyboy above illustrates some of the data-driven test features available in the licensed version of soupUI (soapUI Pro)-- worth checking out to see Pro's "out-of-the-box" support for DDT, requiring less Groovy script.

      Delete
  10. Fantastic post, you helped me a lot. Thanks!

    ReplyDelete
  11. more than helpful, exactly what I needed, thank you!

    ReplyDelete
  12. The best blog than any other.

    ReplyDelete
  13. visit this blog

    http://lgsofttest.blogspot.com

    ReplyDelete
  14. Hi Mike,

    You check the response on "boiling point" point in your example. In want to change that and check "polisnummer", but it doesn't work. I have inserted "testCase.setPropertyValue("ns1:Polisnummer",propData[4])" instead of "billing point". Í filled the CSV with a value that doesn't exist, but the testcase is still succesfully. Do you know what happened?
    Thanks for your help!

    ReplyDelete
    Replies
    1. Sorry, I was forgotten to change the assertion. It works fine now!

      Delete
  15. Hi Mike,

    Is this applicable for REST services as well?

    ReplyDelete
  16. I don't want to make a blanket statement about this, but theoretically I think it should be, at least for basic GET-type requests. You have to be able to specify request parameters using property expansion (which you should be able to do), and I believe assertions would still work pretty much the same way, even with REST services. REST services are something I still need to investigate a little bit, but I've found some services I can use for examples, so look for some posts on REST service testing in the future...

    ReplyDelete
    Replies
    1. Great article! It has been very helpful.
      I can confirm that this method works for developing data-driven test cases for REST services. I was successful in developing one for a simple GET-type service.
      In addition to the steps you have mentioned, I need to add the data value variable (e.g. ${#TestCase#uid}) in the "Value" field of each of the parameters in the Request pane of the REST test step.

      Delete
  17. Great post. I am noticing an odd behavior on subsequent runs (but still in the same SoapUI session) - the TestCase Log is not correctly reflecting the Property Name (ie, testCase.getTestStepAt(0).setName("GetAtomicNumber-" + propData[0])). It most often shows data that was from row 2,3 but not row 1 of the data set.
    If I close SoapUI & run it again - all is well.
    Its as if after running, it needs to be reinitalized - is there a way to do this in groovy?

    ReplyDelete
    Replies
    1. I think I see what you're talking about-- after running the script once or twice, it seems like the log gets out of synch somehow-- the third test shows up as "GetElementSymbol-Carbon". It should be reflecting Hydrogen, and if you look at the corresponding request you'll see that it is in fact Hydrogen that's being sent with the request. I also see some test step results reported AFTER their parent test case is reported as finished. It seems like there's an issue with the log getting out of sync with soapUI-- as if execution is getting ahead of the log.

      One solution that seems to work for me-- although it's not ideal-- is adding a Delay step just before the ReadNextLine() step. Even with only one second added it seems to be enough time to allow the log to "catch up" before execution moves on to the next line. As you pointed out, this shouldn't be a problem in cases where you're launching and running a script intermittently; it only seems to happen when running the script repeatedly.

      Delete
  18. This post is exactly what I am looking for. Thank you so much for sharing.

    ReplyDelete
  19. Good stuff, very useful. In your case you are calling up the same 'ElName' for each of your steps. What if you have different data types for each step/request?
    i.e. step/request 1 calls up customer name, booking source, step/request 2 calls up customer address, contact details etc. Should I set up both data sets in the same data file and how do I declare the the data types in the setup script?

    ReplyDelete
    Replies
    1. It's hard to say without knowing the details of your situation, but unless there's some dependency between the data sets (for example, response data from one customer name/booking source request is needed for a subsequent customer address/contact details request), I think the most straightforward way to handle this would be to have two data files corresponding to two test cases. The sample project in this post is set up the way it is because all three of the test steps within that ElementNames test case take an element name as input parameters, so it makes the looping easier-- read a line, use the parameter data from that line in all three tests, read another line, etc. If I had another request or set of requests where I wanted to loop through different inputs, I'd set up a new test case with its own independent loop (including a new source file).

      I think it's possible to combine data for two separate requests into a single file, but it would add some extra complexity, of course. I'd probably set up an extra "field" for each row in my source that would somehow identify which test request each row should target, then add some extra Groovy steps strictly for flow control. For example, after reading in the first line in the setup script, the first step (I'll call it TrafficCop) would look at that first field in the row and execute a "go to step" method to send execution to the first or second request step. After the first request step, I might add another Groovy step to send control to the ReadNextLine step (skipping over the second request where data isn't applicable to the second request); the ReadNextLine step would read the next line and send control back to the TrafficCop step, etc. I haven't tried this, though, and there may be better ways of doing it.

      Delete
  20. HI Mike !
    Great post !
    But I was looking for something different, i would be glad if you can help me out.
    I have a TestCase, which contains around 10 Test steps
    Each test step will import its data from a csv file on per line basis.
    Can we use groovy script to loop through all these steps and run the Test case.
    For example.
    If the csv file contains:
    a,b,c,d
    e,f,g,h
    So the values should be placed for each step
    --
    --
    --
    --A
    --B
    --C
    --D
    --

    --
    --E
    --F
    --G
    --H
    --
    --
    --
    So the values should be imported and test case should run along with its steps.
    Thanx in advance. :)

    ReplyDelete
    Replies
    1. Sorry for the late reply; I'm not sure if you've already figured out a solution on your own.

      If I follow correctly, I think this does go well beyond the techniques covered in this post. As I understand it, you have to plug in data from multiple lines of your csv file into each request-- I imagine each line represents some sort of entity/object you're trying to represent in your request. I'm also assuming you have to do this somewhat dynamically-- in other words, you want to be able to add another line to your file and have the data from this line automatically added to your test requests. To do something like that, you'd probably have to create a Groovy step that would build an XML block containing all your entities and then plug that block into your request(s).

      Am I following correctly? I just want to make sure before I try to look at this further-- if you'd prefer, you can try reaching out to me through the contact form.

      Delete
  21. You are amazing~!! This is exactly what I am looking for.
    Your tips are well explain and easy to follow. It even comes with a SOAP-UI project, which makes it perfect.
    Good job, buddy.

    ReplyDelete
  22. This is the best tutorial on the subject, THANK YOU SO MUCH for putting it together.

    ReplyDelete
  23. Awesome! After trying so many other blogs, its your blog which worked!! Thank you so much!! Keep blogging!!!

    ReplyDelete
  24. Great post. This was very useful!

    ReplyDelete
  25. Very well Explained and self-descriptive article on Parametrization approach for SOAPUI . None of the other article i found on internet was so good.

    Kudos to you !!!

    ReplyDelete
  26. How do I use the same DataDriven Approach to pass Parameter Values for a REST API. I need to loop through 100 values for one parameter from a csv file?

    ReplyDelete
  27. thanku for explaining briefly..it's very interesting..we are offering soup ui online training

    ReplyDelete
  28. Fantastic explanation. Main part is providing the sample projects which helps to understand the best.

    ReplyDelete
  29. Mike you got such a detailed oriented mind to understand what is actually might be the call of the user. Very well written and explained. Hats off!!

    ReplyDelete
  30. This is really helpful, thanks so much for sharing...

    ReplyDelete
  31. Here you are reading data from csv file.Can I write response to csv file.Please share code for that

    ReplyDelete
  32. How too write response to csv files

    ReplyDelete
    Replies
    1. Here's some sample code to write to a csv file (typically, I'd put this in the test suite setup script, assuming you'd want to access it from multiple tests within your suite):

      //First, create log file (note escaped slash characters)
      context.resultLog = new File("C:\\Path\\To\\ResultFile\\results.csv")
      //Write in your header row as comma-separated line of data with system end of line
      context.resultLog.write("TestName,TimeTaken,Result" + System.getProperty("line.separator"))

      So now that you have your log file set up in context, you can write to it in the appropriate places in your test suite; maybe test case tear down scripts or Groovy Script steps.

      In the code to write further lines to your file (beyond the header row) you'll want to use the append method instead of the write method-- the write method will overwrite any existing data; append will simply add to whatever was in the file (in this case, preserving the header row and any previously entered data).

      //Assumes the data items to write have previously been saved to variables name, result, timeTaken
      context.resultLog.append("$name,$result,$timeTaken" + System.getProperty("line.separator"))

      Delete
  33. Hey Mike, This post is very usefule! I have a question though - How do I parameterize the REST URL itself dynamically with values from teststep?

    ReplyDelete
    Replies
    1. Hi, could you clarify your situation using some example URLs, if possible (you can use dummy data for the examples)? You ask about parameterizing a REST URL, but this particular post deals with a SOAP service. If you're actually working with a service URI like http://my.restService.com/users?lastname=Smith you may want to check out the posts on working with REST services from June 2014. SoapUI actually has a built-in way of dealing with those URI parameters when you define a REST service method and/or request. You can plug context/test suite/test case etc. variables into those parameter values.

      In fact, if you have some sort of hybrid situation where you're passing SOAP data to a parameterized URI (via POST), you may still want to try defining the service as a REST service to handle the URI parameters and add in the SOAP/XML content as POST data for the service requests (I think you'd POST it as Media Type "text/xml"). I believe you can even parameterize your SOAP content using the standard ${#TestSuite#Parameter} syntax. I don't think it's as easy (or even possible?) for SoapUI to handle changing URI parameters when you define the service as a "pure" SOAP service using a WSDL.

      Delete

Please be respectful of others (myself included!) when posting comments. Unfortunately, I may not be able to address (or even read) all comments immediately, and I reserve the right to remove comments periodically to keep clutter to a minimum ("clean" posts that aren't disrespectful or off-topic should stay on the site for at least 30 days to give others a chance to read them). If you're looking for a solution to a particular issue, you're free to post your question here, but you may have better luck posting your question on the main forum belonging to your tool's home site (links to these are available on the navigation bar on the right).