Category Archives: test specification

First results of eMRTD Interoperability Test 2016

During 10th Security Document World 2016 an additional Interoperability Test for eMRTD with PACE took place in London. In context of ePassports this was the 14th event starting 2003 on Canberra. This time there were two test labs involved, 17 document providers and twelve inspection system providers. Here I will focus on the conformity test including test labs and document providers and the InteropTest results. The event was organised by the colleagues of secunet. The following document providers delivered in sum 27 samples:

Logo of SDW InteropTest

  • Arjo Systems
  • Atos IT Solutions and Services
  • Bundesdruckerei
  • Canadian Banknote Company
  • cryptovision
  • De La Rue
  • Gemalto
  • ID&Trust
  • Imprimerie Nationale Group
  • Iris Corporation Berhad
  • MaskTech
  • Morpho
  • NXP Semiconductors
  • Oberthur Technologies
  • PAV Card
  • Polygraph combine ‘Ukraina’
  • PWPW

And the following test laboratories performed a subset of tests focusing on PACE (and of course PACE-CAM):

  • Keolabs (France)
  • HJP Consulting / TÜViT (Germany)

The test cases performed during the event based on ICAO’s test specification ‘RF Protocol and Application Test Standard for eMRTD – Part 3‘ Version 2.08 RC2 including some minor adaptions based on the last WG3TF4 meeting in Berlin. The final version 2.08 of this test specification will be released soon and deltas will be listed in an additional blog post here. With focus on PACE-CAM the following test suites were performed by both test labs:

  • Test Unit 7816_O (Security Conditions for PACE-protected eMRTDs)
  • Test Unit 7816_P (Password Authenticated Connection Establishment)
  • Test Unit 7816_Q (Select and Read EF.CardAccess)
  • Test Unit 7816_S (Select and Read EF.CardSecurity)
  • Test Unit LDS_E (Data Group 14)
  • Test Unit LDS_I (EF.CardAccess)
  • Test Unit LDS_K (EF.CardSecurity)

Some statistics concerning the samples:

  • PACE-CAM was supported in the following types:
    • Generic Mapping (GM), Chip Authentication Mapping (CAM): 18 samples
    • Integrated Mapping (IM), CAM: 4 samples
    • GM, IM and CAM: 4 samples
  • LDS:
    • 25 samples used LDS1.7
    • 1 sample used LDS1.8
    • 1 sample used LDS2.0 (with backward compatibility to LDS1.7)

In the preliminary InteropTest results presented by Michael Schlüter during the SDW he mentioned, that 8502 test cases were performed during conformity testing by the test labs and 98% of the relevant test cases were passed by the samples. Additionally, the test results of both labs were fairly consistent. There was one test case that causes the most failures and this test case verifies ChipAuthenticationPublicKey in EF.CardSecurity (LDS_K_2). Here we need some clarification in the specification Doc9303 and finally in the test specification.

During the crossover test there were three problems detected: At first the sequence of PACE, CA and TA was performed correctly while the sequence of PACE-CAM and TA causes some problems during the inspection procedure of the readers. This might be based in the fact, that PACE-CAM is specified in an ICAO document and TA in a BSI document. Some inspection systems had also problems with alternative File IDs for EF.CVCA. The alternative FID can be defined in TerminalAuthenticationInfo (see A.1.1.3 in TR-03110 Part 3) and must be used by the inspection systems to address and read EF.CVCA. But a bad surprise in the InteropTest results was, that around 50% of the inspection systems don’t perform Passive Authentication (PA) correctly. During the preparation of the InteropTest a wrong CSCA certificate was distributed and 50% of the systems have not detected this wrong certificate, this means: 50% of the inspection systems failed in Passive Authentication! During the conference Dr. Uwe Seidel (Bundeskriminalamt, BKA) noticed, that this number mirrors the real world and that in fact PA is currently a real problem in border control.

The InteropTest results can be summed up in two statements:

  1. There is a very good quality of the eMRTD samples.
  2. Reader vendors have still some work to do, especially to implement Passive Authentication correctly.

A detailed test report of this event and the InteropTest results will be published by secunet in June 2016.

Update: The final test report can now be downloaded here (after a short registration at the SDW website).

Maintenance release of BSI TR-03105 Part 5.1

The German BSI has published a maintenance release of technical guideline TR-03105 Part 5.1 Version 1.41 for inspection systems with Extended Access Control (EAC).

Since last release of TR-03105 several (mostly editorial) comments were resolved and integrated in this maintenance release. Part 5.1 describes conformity tests for inspection systems with protocols like PACE, Terminal Authentication and Chip Authentication typically used at (automatic) border control, e.g. eGates.

Maintenance of TR-03105 for inspection systems, http://www.iconarchive.com/artist/oxygen-icons.org.html

Besides some editorial changes the new version 1.41 contains the following modifications:

  • ISO7816_G_36: If a EF.CardAccess contains an invalid OID for PACE-CAM, the inspection system shall use an alternative mapping protocol, that is supported by the chip.
  • ISO7816_G_37: This test case is deleted, because it’s not necessary for an inspection system to check that GM and IM are also supported by the chip besides PACE-CAM.
  • ISO7816_G_41: Curves with parameterID 0, 1 and 2 are based on DH and DH is not supported in context of PACE-CAM. So these curves are deleted.
  • LDS_H_86: Correction in expected result (PASS instead of FAIL).
  • Chapter 7: Relevant algorithms and OIDs for PACE, that must be supported by the inspection system, are added.
  • Chapter 7: Update of hashing algorithms.

For the next major update there should be a discussion how to handle fingerprints (data group 3, EF.DG3) and iris (data group 4, EF.DG4) of people who don’t have a finger or an iris. In this case these data groups should store an empty but valid data structure. Currently there are no test cases specified for these situations in TR-03105 Part 5.1. But inspection systems should be able to handle such cases also, of course.

So you can see, that test specifications in context of eMRTD (ePassports) and inspection systems are always in progress. If you have any comments concerning these test specifications or ideas of test cases, that should also be performed focusing on interoperability, please don’t hesitate to contact me or leave a comment.

Interoperability Test during SDW in May 2016

puzzle - interoperability test

Puzzle of InteropTest

Another interoperability test in context of ePassports (eMRTD) and inspection systems will be performed during SecurityDocumentWorld 2016 in London. The test will be focused on Supplemental Access Control (SAC) respective PACEv2, a security protocol to protect personal data stored in electronic ID documents.

An interoperability test is similar to a plugtest performed e.g. by ETSI. It’s an event during which devices (ePassport, inspection systems and test tools) are tested for interoperability with emerging standards by physically connecting them. This procedure is called crossover testing and allows all vendors to test their devices against other devices. The efforts to perform this kind of test increases very strongly with every ePassport and inspection system. Therefore these kind of tests can be performed only with a small number of devices under test.

Crossover Testing

Crossover Testing

Additionally, there is the opportunity besides this crossover tests to test the devices against conformity test suites implemented in test tools like open source tool GlobalTester. This procedure reduces efforts and allows comprehensive failure analyses of the devices like ePassports or inspection systems. To assure interoperability it is state of the art to set up test specifications. These specifications are implemented by the test labs respectively in the test tools they use.

Conformity Testing

Conformity Testing

There are well established test specifications available, both for ePassports and for inspection systems. Previous interoperability tests took place in Madrid (2014) and London (2013). Both events focused also on SAC/PACE.

If you are interested as a document provider, as a vendor of an inspection system, as a test lab or as an observer, you can register here.

Looking forward to seeing you in London during the InteropTest!

BTW: The EU article 6 group is preparing a document describing how to process an interoperability test and how to prepare such an event.

Update of BSI TR-03105 Part 5.1 available (V1.4)

Introduction

There is an update of BSI technical guideline TR-03105 Part 5.1 available. The new version 1.4 of this test specification for inspection systems with EACv1 is focusing on PACE (including PACE-CAM) and LDS 1.8.

Cover of BSI TR-03105 Part 5.1

Cover of BSI TR-03105 Part 5.1

The new version of TR-03105 is now available in new BSI layout. Additionally, there are some minor editorial changes and updated references (e.g. new Doc9303 is referenced).

New test cases in TR-03105 Part 5.1

The Standard Inspection Procedure (SIP) includes now also PACE and there is a new configuration specified for default PACE passport.

New test cases for PACE/SAC

Here is a list of new test cases, added in TR-03105 5.1 to test PACE, including PACE-CAM:

  • ISO7816_G_01: Correct execution of PACE protocols
  • ISO7816_G_02: Check supported standardized domain parameters with Generic Mapping
  • ISO7816_G_03: Check supported standardized domain parameters with Integrated Mapping
  • ISO7816_G_04: Check supported algorithms
  • ISO7816_G_05: Check PACE with additional entries in SecurityInfos
  • ISO7816_G_06: Check selection of standardized Domain Parameters and algorithms
  • ISO7816_G_07: EF.CardAccess contains two PACEInfo and PACEDomainParameter
  • ISO7816_G_08: Abort PACE because of SW error code during MSE:Set AT
  • ISO7816_G_09: Error on the nonce – Value modifications after first General Authenticate
  • ISO7816_G_10: Error on General Authenticate step 1 command
  • ISO7816_G_11: Error on General Authenticate step 1 command – bad tag (use 90h instead of 80h)
  • ISO7816_G_12: Error on General Authenticate step 2 command
  • ISO7816_G_13: Error on General Authenticate step 2 command bad tag (use 92h instead of 82h)
  • ISO7816_G_14: Abort PACE because of error in GA step 2 (GM)
  • ISO7816_G_15: Abort PACE because of error in GA step 2 (IM)
  • ISO7816_G_16: Error in General Authenticate step 2 command – error on mapping data – all ECDH public key components
  • ISO7816_G_17: Error in General Authenticate step 2 command – error on mapping data – all DH public key components
  • ISO7816_G_18: Error in General Authenticate step 3 command
  • ISO7816_G_19: Error in General Authenticate step 3 command – bad tag (use 94h instead of 84h)
  • ISO7816_G_20: Abort PACE because of error in GA step 3
  • ISO7816_G_21: Error on General Authenticate step 3 command – error on ephemeral public key – all ECDH public key components
  • ISO7816_G_22: Error on General Authenticate step 3 command – error on ephemeral public key – all DH public key components
  • ISO7816_G_23: Abort PACE because of identical ephemeral public keys
  • ISO7816_G_24: Error on General Authenticate step 4 command
  • ISO7816_G_25: Error on General Authenticate step 4 command – bad tag (use 96h instead of 86h)
  • ISO7816_G_26: Abort PACE because of error in GA step 4
  • ISO7816_G_27: Abort PACE because of TLV error on EF.CardAccess
  • ISO7816_G_28: Abort PACE because of incorrect parameterId in PACEInfo
  • ISO7816_G_29: PACE-CAM with missing tag 8Ah but correct ECAD
  • ISO7816_G_30: PACE-CAM with incorrectly encoded tag ECAD (no octet string)
  • ISO7816_G_31: PACE-CAM with wrong tag ECAD
  • ISO7816_G_32: PACE-CAM with wrong tag 8Ah (use 8Bh) but correct ECAD
  • ISO7816_G_33: PACE-CAM with correct tag 8Ah but missing ECAD
  • ISO7816_G_34: PACE-CAM with Passive Authentication
  • ISO7816_G_35: Return additional tag 8Ah during PACE-GM
  • ISO7816_G_36: Use invalid OID for PACE-CAM in EF.CardAccess
  • ISO7816_G_37: Use EF.CardAccess with PACEInfo only for PACE-CAM (no GM or IM)
  • ISO7816_G_38: Use DG14 without SecurityInfo during PACE-CAM
  • ISO7816_G_39: Use EF.CardSecurity with wrong ChipAuthenticationPublicKeyInfo during PACE-CAM
  • ISO7816_G_40: Use EF.CardSecurity without ChipAuthenticationPublicKeyInfo during PACE-CAM
  • ISO7816_G_41: Check supported standardized domain parameters with Chip Authentication Mapping

New test cases for LDS 1.8

Here is a list of new test cases, added in TR-03105 5.1 to test LDS 1.8:

  • LDS_A_10: EF.COM with LDS Version 1.8
  • LDS_H_86: EF.SOD with LDS Version 1.8
  • LDS_H_87: Security Object with LDS Version 1.8 but with wrong version number
  • LDS_H_88: Security Object with LDS Version 1.7 but version number 1
  • LDS_H_89: EF.SOD with future LDS Version 1.9

Adapted test cases in TR-03105

Here is a list of modified test cases in TR-03105 5.1:

  • In chapter 7.1.2 the OIDs for plain signatures are corrected.
  • ISO7816_D_06: Added second public key with key reference FE in EF.DG14
  • ISO7816_D_15: Use configuration of D_06 to assure the use of wrong key reference
  • ISO7816_F_02: Added signature algorithm (ECDSA with SHA1) in EF.DG14 to fulfil requirements
  • ISO7816_F_08: Changed expected results in transfer interface: TA and CA might not be performed
  • LDS_A_06: Correction in EF.COM where Unicode Version 5 must be encoded
  • LDS_D_08: The referenced invalid format owner (0102) is used by JTC1/SC27 IT Security Techniques (see www.ibia.org/base/cbeff/biometric_org.phpx). So the referenced invalid format owner was changed to ’87 02 01 FF’.
  • LDS_E_07: The referenced invalid format owner (0102) is used by JTC1/SC27 IT Security Techniques (see www.ibia.org/base/cbeff/_biometric_org.phpx). So the referenced invalid format owner was changed to ’87 02 01 FF’.
  • LDS_H_04: Correction in EF.SOD where RSASSA-PKCS1_v15 must be used
  • LDS_H_50: The serial number is mandatory, so expected result was changed to “FAIL”

Next steps

The version 1.4 of BSI TR-03105 Part 5.1 is a backport of ISO18745-4. Until the ISO test specification is under construction and not released, TR-03105 can be used as an interims version for testing inspection systems using PACE/SAC.

 

Mapping between protocols and test specifications

Introduction

This posting describes the current relation between test specifications and the protocols used in context of ePassports (eMRTD) and eID cards including their associated readers (terminals) and inspection systems.

This mapping reflects the current(!) status quo of protocols and their test specifications. All these specifications are in intensive editing at present.

Mapping between protocols and test specifications

The following image represents the mapping between protocols and the corresponding test specifications:

Mapping between protocols and test specifications

Mapping between protocols and test specifications in context of eID

You can see all protocols used currently in context of ePassports and eID cards in the rows and in the columns you can find specifications focusing on testing these protocols. For example you can find the test cases for Active Authentcation in the specification ICAO TR Protocol Testing Part 3 for chips and in BSI TR-03105 Part 5.1 for inspection systems.

As soon as there are updates available I will present here in this blog the new structure of these test specifications, including new protocols like Pseudonymous Signatures (PS), Chip Authentication Version 3 (CAv3) or Enhanced Role Authentication (ERA).

Abbreviation of protocols referred here

BAC: Basic Access Control
AA: Active Authentication
PACE: Password Authenticated Connection Establishment
SAC: Supplemental Access Control
CA: Chip Authentication
TA: Terminal Authentication
EAC: Extended Access Control
RI: Restricted Identification
eSign: electronic Signature

Test Specifications referred here

Short Name Title
TR-03105 3.1 BSI Test plan for eMRTD Application Protocol and Logical Data Structure
TR-03105 3.2 BSI Test plan for eMRTDs with EACv1
TR-03105 3.3 BSI Test plan for eID-Cards with Advanced Security Mechanisms EAC 2.0
TR-03105 3.4 BSI Test plan for eID-cards with eSign-application acc. to BSI TR-03117
TR – RF and Protocol Testing Part 3 ICAO TR – RF and Protocol Testing Part 3
TR-03105 5.1 BSI Test plan for ICAO compliant Inspection Systems with EAC
TR-03105 5.2 BSI Test plan for eID and eSign compliant eCard reader systems with EACv2

Update (30.11.2015)

Once again, you can find some discussions concerning this posting at LinkedIn.

Update of RF and Protocol Testing Part 3 V2.07 online

There is an maintenance update of ICAO’s test specification ‘RF and Protocol Testing Part 3‘ available since today. The specification is focusing on conformity testing and protocol testing for ePassports implementing protocols like BAC and Supplemental Access Control (SAC) respective PACE v2.

The Technical Advisory Group (TAG) of ICAO endorsed the release on the ICAO website, so from now on the test specification can be referenced officially. In version 2.07 of the test specification there are no technical or fundamental changes, but editorial changes. The following test cases are modified in the new release 2.07:

  • ISO7816_B_16: Profile corrected
  • ISO7816_B_26: Added version
  • ISO7816_B_34: Profile corrected
  • ISO7816_B_52: Profile corrected
  • ISO7816_D_06: Updated version
  • ISO7816_D_09 – ISO7816_D_22: Profile corrected and version updated
  • ISO7816_E_09 – ISO7816_E_22: Profile corrected and version updated
  • ISO7816_F_20: Profile corrected and version updated
  • ISO7816_G_20: Profile corrected and version updated
  • ISO7816_O_12: Deleted obsolete Test-ID
  • ISO7816_O_13: Corrected sequence tags
  • ISO7816_O_31: Deleted obsolete Test-ID
  • ISO7816_O_35: Added missing caption
  • ISO7816_P_xx:  Deleted sample in description of step 1 (‘i.e. more than one set of
    domain parameters are available for PACE’)
  • ISO7816_P_04: Corrected numbering in expected results
  • ISO7816_P_06: Corrected numbering in expected results
  • ISO7816_P_07: Corrected numbering in expected results
  • ISO7816_P_14: Updated version
  • ISO7816_P_74: In preconditions step 3 concretized concerning PACEInfos in EF.CardAccess
  • ISO7816_Q_03: Added missing reference TR-SAC
  • LDS_D_06: Corrected typos in step 8

 

With the new edition of Doc 9303 several technical reports are now obsolete except the test specifications. These test specifications are still independent documents.

Testing ePassport Readers using TTCN-3

Currently you can find the well-known test tool Titan under the patronage of the Eclipse Foundation (proposal). This tool was developed by Ericsson several years ago to the test internet protocol (IP). Titan bases on TTCN-3, a test language focusing on communication protocols. This keeps me in mind an old project with ETSI where we used TTCN-3 to test ePassport readers concerning BSI TR-3105 part 5.1.

From end of 2009 to middle of 2011 ETSI has conducted a project to develop a test system prototype for conformance testing of ePassport readers. The objective of this project was to design, build and test a TTCN-3 (Testing and Test Control Notation, Version 3) based test system prototype for ePassport reader conformance testing. This project was a joint effort between the European Joint Research Centre (JRC) in Ispra (Italy) and ETSI in Sophia Antipolis (France). The test language TTCN-3 has already been widely used in many testing projects across different protocols and technologies. However, until this project TTCN-3 has not been applied in the area of contactless smart card testing.

The ETSI Specialists Task Force (STF) 400 with experts from the companies / organisations AMB Consulting, ARH, Comprion, ETSI, FSCOM, HJP Consulting, Masaryk University, Soliatis and Testing Technologies operated this project. The work of this STF has been split into three main phases:

  1. Design, implementation, and use of ePassport test system
  2. Development of ePassport testing framework
  3. Writing of the documentation and dissemination material

Scope of this project was to build a test system to test an inspection system typically used to read ePassports. To demonstrate the basic functionality and the feasibility, a subset of BSI TR-03105 Part 5.1 was specified and implemented in the test system.

The following image describes the architecture of the ePassport reader test system developed during this project:

System architecture of prototype to test ePassport Reader with TTCN-3

Architecture of test system based on TTCN-3 for ePassport readers (Source: ETSI White Paper No.7)

The most significant part in the architecture is “TTCN-3 Test Component”. This module simulates the ePassort behaviour  by receiving APDUs, react to these commands and send result in APDUs back to the SUT (here the ePassport reader).

The successful implementation of a TTCN-3 based test system shows that the use of TTCN-3 fits the requirements of conformance testing of eMRTD or other eID systems. The prototype demonstrates the feasibility of using such formal techniques for protocols which would improve the quality and repeatability of tests and reduce room for interpretation.

An overview of this project and the results were summarized by the colleagues Jean-Marc Chareau, Laurent Velez and Zdenek Riha in ETSI White Paper No 7.

 

 

Update of RF and Protocol Testing Part 3 V2.06 online

The MRTD group of ICAO has released an update (version 2.06) with clarifications of their technical report RF and Protocol Testing Part 3 focusing on conformity test and protocol testing for ePassports implementing protocols like BAC and Supplemetal Access Control (SAC) respective PACEv1.

The new version 2.06 of TR-03105 Part 3.2 focusing on protocol testing includes the following changes:

  • General: Several test cases accept now additionally also an Execution Error in expected results.
  • General: Instead of ePassports we are talking now about eMRTD.
  • General: An additional profile was added: “EAC or PACE or AA-ECDSA”.
  • General: The profiles of several test cases were extended.
  • General: Compatibility to both PACE and BAC in most test cases of ISO_D and ISO_E.
  • General: Use CAR from DV certificate during Terminal Authentication instead of reading CAR from file EF.CVCA.
  • ISO7816_C_04: The command GET CHALLENGE must not have been performed.
  • ISO7816_P_10: This test case was deleted.
  • ISO7816_P_73: Allows multiple PACEInfo if just one parameter ID is being used.
  • ISO7816_P_74: Allows multiple PACEInfo if just one parameter ID is being used.
  • ISO7816_P_75: Requires two PACEInfo elements using the same OID and different parameter IDs.
  • LDS_A_03: Now LDS version 1.8 is also accepted.
  • LDS_B_13: Added new assertions on the date (day and month).
  • LDS_D_06: Additional test step checking the LDS info object.

In the past I have missed such a list for every new released version of test specifications, like BSI TR-03105 or ICAO technical reports. You can find a list of modifiied test cases for protocol testing in the last version of BSI TR-03105 Part 3.2 in a previous post.

So I hope, this list of modified test cases is helpful for your work in context of ePassport testing. If you are interested, please leave a comment and I will update this list with every new version of test specifications in context of smart cards used in ePassports and ID cards.

Next generation of ePassport testing

Developing and implementing conformity tests is a time-consuming and fault-prone task. To reduce these efforts a new route must be tackled. The current way of specifying tests and implementing them includes too many manual steps. Based on the experience of testing electronic smart cards in ID documents like ePassports or ID cards, the author describes a new way of saving time to write new test specifications and to get test cases based on these specifications. It is possible, though, to improve the specification and implementation of tests significantly by using new technologies such as model based testing (MBT) and domain specific languages (DSL). I’m describing here my experience in defining a new language for testing smart cards based on DSL and models, and in using this language to generate both documents and test cases that can run in several test tools. The idea of using a DSL to define a test specification goes back to a tutorial of Markus Voelter and Peter Friese, hold during the conference Software Engineering 2010  in Paderborn.

With the introduction of smart cards in ID documents the verification of these electronic parts has become more and more important. The German Federal Office for Information Security (BSI) defines technical guidelines that specify several tests required to fulfill compliance. These guidelines include tests on the electrical and physical layer on the one hand, and tests on the application and data layer on the other hand. In this presentation the author focusses on the tests on the last two layers because these tests can be implemented completely in software.

In TR-03105 the BSI specifies several hundreds of test cases concerning the data format of smart cards and also the commands and protocols used to communicate with the chip.

In the past the typical approach was divided into several separate steps. At first the BSI specified a list of test cases and published them in a document that was written manually by an editor of the technical guideline. Then several test houses and vendors of test tools implemented all the test cases based on the specific guideline into their software solution. All these steps had to be done manually, which means: the software engineer of each institution read the guideline and implemented test case by test case in his special test environment. With every new version of the guideline this procedure had to be repeated again and again. At the beginning, the update cycle of these test specifications was very frequent because all the feedback collected in the field was included in the guideline and new versions were published in short intervals:

figure_1

This way of specifying test specifications is inefficient because of the large number of manual steps. Doing the transformation from the test specification to the implementation is not only inefficient but also fault-prone: every test case in the guideline must be formulated in “prose” by the editor; every engineer must implement the test case in the respective programming language. Also the consistency of the tests must be maintained by the editor manually.

Furthermore, the writing of test specifications is an extensive part of conformity testing. The editor of such a specification in general uses a word processing software that is useful for e.g. writing small letters. But this kind of software is not really convenient for writing technical specifications like TR-03105. A typical problem is versioning of different types. It would be most helpful for developers, if the editor used the track changes mode when he changes test cases. This way the developer can easily detect changes. But this advantage depends on the activated mode. As soon as the editor forgets to activate the track changes mode the implementation of these changes becomes more and more complicated.

Due to an increasing number of new requirements of the applications running on smart cards the complexity of these systems becomes higher and higher. In Walter Fumys “Handbook of eID Security” the history of eID documents from purely visible ones to future versions is illustrated. This complexity in these applications will result in so many test cases that the current approach of writing and implementing test specifications is a blind alley.

With recent results of Model Driven Software Development (MDSD) this blind alley can be avoided. New techniques and tools allow us now to switch from the manual parts to a more automated procedure. The goal is to write only one “text” that can be used as a source for all the test tools. The solution is a model that defines the test cases and a transformation of this model to other platforms or formats.

With this new approach, the process of specifying tests can be reduced to the interesting part where the editor can use his creativity to conceive new tests and not to use his office software to write tests.

Defining a language that describes the test case is the basis for this procedure. This grammar can be used to model test cases, and based on this model all the artifacts needed can be generated. The following figure visualizes this process: there is one Meta test specification that is used to generate not only the human-readable document but also the tool-specific test cases for every test environment.

figure_2

One solution to define a language is Xtext. With Xtext the user gets a complete environment based on Eclipse to develop his own domain specific language (DSL). One of the benefits of Xtext is the editor that is generated automatically by the tool itself. This editor includes code-completion, syntax coloring, code-folding and outline view. This editor is very helpful to write test cases. Every test case that is not compliant with the grammar is marked as faulty. So the editor of the specification can recognize this wrong test case directly like a software developer in Integrated Development Environments (IDE).

Additionally, the user can implement generators to generate code for the scope platform. These generators are called by the Modeling Workflow Engine (MWE). These generators are powerful and productive tools to provide test cases for different platforms.

In the public sector it is more and more important to write barrier-free documents. It takes a lot of time to write a barrier-free document based on a typical technical specification. With a generator that produces a human-readable document the author of the test specifications can use generic templates that produce barrier-free documents in an automatic way because the generator can use rules that fulfill even these standards.

Once the user has generated a new test specification or a new test case based on any test tool, he can modify this document by adding some special features, e.g. a special library to one test case. With a model based test specification it is possible to re-import this modified artifact into the model to assure persistency. The author presented and published his first experience at ICST 2011.

This approach helps to write test specifications in a technical way on a Meta level but it does not focus on the content of the test specification. Thus, the approach helps to write the document but it does not help to produce any content needed. Currently, the quality of a test specification is dependent from the background of the author. With his knowledge of protocols and corresponding pitfalls he can specify interesting test cases. But many test cases contain the same scenarios (wrong length, set a value to zero, use maximum or minimum value and so on). It would be more reasonable and economical if the author could focus on special test cases for the relevant protocols and their pitfalls and “standard” test cases would be generated automatically. On the other hand, test specifications written by humans always run the risk of being inconsistent, error-prone and imprecise. Additionally, it is always rather time-consuming to write test specifications manually.

To focus and solve problems as described above a consortium of BSI, HJP Consulting, s-lab Software Quality Lab (University of Paderborn) and TÜViT started a research project, namely MOTEMRI (Modellbasiertes Testen mit Referenzimplementierung). In MOTEMRI a model is developed that contains all relevant information of the popular protocol PACE. This model is specified in UML so everybody who is interested can read and modify the diagrams easily. In this way it is possible to adapt new protocols into the model like PACE or new versions developed in the future. Based on the model, algorithms generate test cases automatically. Thereby the knowledge of designing test cases is enacted into software, independent from the knowledge of the author of the test specification. By the way, this procedure also allows using various “wrong” values for negative test cases.  Negative test cases are generated automatically and access different “wrong” values. Using random values allows better testing and ensures better chip implementations.