On the Testability of Distributed Real-Time Systems ?

Werner Sch?utz

Institut f. Techn. Informatik

Technical University of Vienna

A-1040 Vienna, Austria

Abstract

In our work on real-time systems we found it useful to distinguish between event-triggered and timetriggered systems. The goal of this paper is to compare them with respect to their testability. To facilitate the comparison, three aspects of testability are introduced: Test Coverage, Observability, and Controllability. After a brief review of the problems of testing concurrent programs, we compare the two system architectures with respect to each of these three aspects.

The results of the comparisons favour time-triggered systems in all three aspects, most notably with respect to Controllability and Test Coverage. Thus, timetriggered systems are inherently more testable than event-triggered systems.

Finally, a series of experiments is summarized that shows that it is in fact easy to achieve reproducible testing of time-triggered systems.

1 Introduction

Testing is a form of verification which involves exercising a system and supplying it with valued inputs [10]. Testing safety-critical real-time systems is becoming ever more complex and expensive. It has been repeatedly reported that testing such a system typically consumes in the order of 50% of the total project costs [13,1]. This is due first to the more stringent demands placed on a system and its verification procedures in the context of safety-critical applications, secondly to the distributed architecture of modern systems.

Testing distributed programs is inherently more complex than testing sequential programs, mainly because of synchronization and communication requirements between cooperating processes, whose actions may depend on the actions of other processes and on

?This work was supported in part by the ESPRIT Basic Research Project 3092 Predictably Dependable Computer Systems"

the time and duration of these actions. Testing realtime programs adds yet another dimension to the problem, because the concern is now not only with the functional correctness, but also with the timeliness of the results produced.

It is therefore important to design a system in such a way as to facilitate its testing as much as possible. In this paper we introduce the notion of system testability. Testability is a qualitative measure of the ease with which a system can be tested. The following aspects of testability have been identified:

ffl Test Coverage: The Test Coverage indicates how many of the anticipated real-world scenarios can be covered by corresponding test scenarios. Ideally, this coverage would be 100%, in reality there are significant limitations due to the combinatorial explosion of possible event combinations.

ffl Observability: Observability is important for determining whether the system under test performs correctly or not. There are two aspects. First, one must be able to observe every significant event generated by the environment and { much more important and difficult { to determine the correct ordering and timing of events. Secondly, it is necessary to observe the actions and outputs of the system during testing, but without disturbing its time behaviour.

ffl Controllability: During testing, one must be able to control the execution of the system so that it is possible to reproduce arbitrary test scenarios such that the system's reaction is deterministic.

Testability is to a great extent determined by the properties of the underlying system architecture. Therefore, in this paper we want to assess the differences in the testability of time-triggered and eventtriggered architectures that are introduced in Section 2. As a starting point of this comparison we briefly review the problems of testing concurrent programs in Section 3. In Sections 4, 5, and 6 the two types of