Automated Adapter Tests - Specifications

Goal

The goal of the adapter testing framework is to make it easy and comfortable for the community to update existing adapters without fear of breaking past functionality.

What's in a Test?

These are the components of a test. Each test will consist of a folder containing all of these tests files.

recording.xml

All tests should run against recordings, which will mimic a real device at a point in time. This way we can have a controlled test case. This recording file will also contian the connectionPath information used during the generation. This connectionPath element should be stripped out by the testing framework and used as the payload on the SOAP call to the adapters servlet. A sample recording is attached here:
IOS_backup_10.100.4.8_2007-10-12_09-16-55.recording

xpath-tests.properties

This file will be used to verify several pieces of the ZiptieElementDocument^?.xml file that the backup operation generates. Each line of this file will contain and XPath and the value that is expected in that XPath.
xpath-tests.properties

How the Test Program Should Operate

Beginning

• Download the nightly ZipTie server
• Start the server

A Single Test

• load up the test's recording.xml file into the ZipTie device simulator
• Parse out the connectionPath element from the recording.xml file and use it to call the backup operation for the adapter against the adapters SOAP service.
• The return value of the backup call will be the ZiptieElementDocument.xml file
  • Validate the ZiptieElementDocument.xml file against the schema file that it identifies
  • Validate each xpath-tests.properties XPath test

End

• parse the log file created by each individual test
• compose an HTML page with the results and email that page as the email body

Other Test Requirements

• Be able to perform n number of tests simultaneously. Initially this number should be 10, but it should also be configurable.
• The resulting email should have the following metrics
  • The number of overall tests
  • The number of XPath tests
  • The % of passing backups
  • The % of passing schema validation tests
  • The % of passing XPath tests
  • The time the test suite took to run
  • List any failures; backup, schema validation, or xpath tests

Technical Resources

• The ZipTie perl module ZipTie::TestElf has methods to run XPath tests and Schema validation tests. The adapter testing suite should use this module.

-- RyanKruse - 12 Oct 2007