Verification & Testing: SAP HANA Case Study
Anthony J H Simons (University of Sheffield) Wolfgang K R Schwach (SAP Karlsruhe)
Overview
Verification and Testing Toolset
Service Specification
Validation and Verification
Test Generation and Grounding
Test Execution on SAP HANA Cloud App
V&T tools: Anthony Simons, Uni. Sheffield
App & grounding: Wolfgang Schwach, SAP
Verification & Testing Tools
Specification
Why?
Gentle pressure towards common standards
Provide basis for formal service certification
What?
Describe service interfaces and behaviour
State-based model of service API
IOPE model of detailed behaviour
XML language Java metamodel
Holiday Booking Service
XML Specification -FSM
XML Specification -IOPE
Validation & Verification
Why?
Designers write incomplete specifications
Designers write incorrect specifications
How?
Validation tool explores state-transition diagram
Reports unhandled events, missing transitions
Verification tool simulates all input/memory space
Reports blocking/nondeterministic operations
Blocking: input exists which triggers no operation
Nondeterminism: some input triggers many operations
Verification Fault Analysis
Test Generation
Why?
More thorough coverage than manual testing
Assures full compliance to the specification
How?
Generation tool explores the whole specification
Creates all interleaved sequences up to depth k
Reduces test-suite size by optimising
Removes infeasible sequences (blocked in the spec)
Removes redundant sequences (equivalence-classes)
Compresses executable suite (multi-objective tests)
Generated Test Suite
Test Grounding
Why?
High-level test suites are technology-neutral
Platform-specific translations required
How?
Platform providers offer bespoke groundings
[email protected] framework provides hooks
XML test suite parsed as a Java metamodel instance
Visitor Design Pattern is used to generate concrete tests
Bespoke grounding to e.g. Java, SOAP, REST, Selenium
Example Java Grounding
SAP HANA Case Study
User may book up to 5 days vacation
blocks of days can be booked, selected, deleted
prevents overlapping blocks, more than 5 days
App designed using the SAP UI5 toolkit
visually pleasing point-and-click interface
operations nominally match the earlier specification
but a rich-client web app API is hard to test
Deployed on the SAP HANA cloud platform
temporarily available at URI: brokercloud-d050960trial.dispatcher.hanatrial.ondemand.com/
App Running on HANA
Selenium Test Driver
Selenium: a tool for web-testing
normally used to capture-and-record a user’s point-and-click gestures interactively
can replay recorded test scripts, stored as HTML tables, to run web app regression tests
Leverage to test a rich-client cloud app
given the Selenium DOM for the web app
translate high-level test suite into Selenium test scripts
use the Selenium engine to execute the test scripts
Selenium Grounding Rules
Executable Test Scripts
Executing the Tests
Subtle Faults Detected
Functionality
service failed to notify duplicate holiday booking as a separate error case
service failed to distinguish overflow and duplicate holiday booking responses
Security
service executed operations in extra states, when the operations should be ignored
service revealed response information, when the request should be ignored
Beyond State-of-the-Art
Extends leading SXM theory for MBT
Adds reasoning about the full SXM model
Adds validation and full verification capability
Adds test optimisation capability
Exceeds all industrial testing methods
Full coverage of the specification achieved
All positive and negative test cases created
Identifies top 10% uniquely discriminating tests
Questions
Acknowledgement
FP7-ICT-2011-8 grant agreement no: 318392
Project website
www.broker-cloud.eu
References
[email protected] D20.1 State of the art and research baseline, http:www.broker-cloud.eu, 2013.
[email protected] D20.3 Requirements for continuous quality assurance and optimisation in cloud brokerage, http:www.broker-cloud.eu, 2013.
AJH Simons and RE Lefticaru, Specification, verification and testing of brokered software services in the cloud, IEEE ICST 2015, Graz (submitted, under review)
AJH Simons and RE Lefticaru, Stream X-Machine based verification and testing of cloud services, 2nd Int. Workshop on Cloud Service Brokerage, ESOCC, Manchester, 2 Sept., CCIS, 2014.
M Kiran, A Friesen, AJH Simons and WKR Schwach, Model-based testing in cloud brokerage scenarios, 1st Int. Workshop on Cloud Service Brokerage, ICSOC, Berlin, 2 Dec., LNCS, 2013.