How many instances of Silent failures were observed in a version of the Mach OS?

92 instances of Catastrophic failures were observed in a version of the Mach OS previous to the version reported here [4].

What is the definition of the Silent class of failure?

The Silent class of failure occurs when invalid parameters are submitted to an OS call, but neither an error return code nor other task failure is generated.

What is the way to measure the robustness of an OS?

If a single OS defect is responsible for a large number of robustness gaps, then the next release of the OS should (if the defect is fixed) have a markedly better robustness rating in return for minimal effort on the part of the vendor.

What is the definition of an Abort failure?

Note that if an error takes place during an OS call and the exception is handled appropriately by the OS before control returns to user code, it is not user-visible, and thus is not considered an Abort failure.

What was the purpose of the previous generation of robustness testing?

This previous generation of robustness testing was accomplished on a single system, and established the viability of invalid value parameter testing as a way of uncovering OS robustness gaps.

What is the definition of the Restart class of failure?

The Restart class of failure occurs when a single task hangs, resulting in the need to kill and restart that task to return to normal execution.

Why is the emphasis on testing the robustness of an OS when application code provides invalid inputs?

Because real-world application software is seldom, if ever, bug-free, the emphasis here is upon testing the robustness of an OS when application code provides invalid inputs to it.

What is the purpose of the portable robustness benchmarks?

A set of portable robustness benchmarks has been developed that exercise several OS calls with combinations of valid and invalid input parameters.

Open AccessProceedings Article10.1109/RELDIS.1997.632800

Comparing operating systems using robustness benchmarks

Philip Koopman, +4 more

- 22 Oct 1997

- pp 72-79

171

TL;DR: The concept of response regions is presented as a way to visualize erroneous system behavior and gain insight into failure mechanisms, and a 5-point "CRASH" scale for grading the severity of robustness vulnerabilities encountered.

Abstract: When creating mission-critical distributed systems using off-the-shelf components, it is important to assess the dependability of not only the hardware, but the software as well. This paper proposes a way to test operating system dependability. The concept of response regions is presented as a way to visualize erroneous system behavior and gain insight into failure mechanisms. A 5-point "CRASH" (catastrophic, restart, abort, silent, hindering) scale is defined for grading the severity of robustness vulnerabilities encountered. Test results from five operating systems are analyzed for robustness vulnerabilities, and exhibit a range of dependability. Robustness benchmarking comparisons of this type may provide important information to both users and designers of off-the-shelf software for dependable systems.

AI Agents for this Paper

Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps

Most frequently asked questions

1. What have the authors contributed in "Comparing operating systems using robustness benchmarks" ?

This paper proposes a way to test operating system dependability.. The concept of response regions is presented as a way to visualize erroneous system behavior and gain insight into failure mechanisms.. Robustness benchmarking comparisons of this type may provide important information to both users and designers of off-the-shelf software for dependable systems.

2. What is the way to approach this area?

One way to approach this area is to focus on the operating system (OS), which is the foundation upon which application software rests.

3. Why was open() not testable under QNX?

The open() function was not testable under QNX due to lack of compiler support for a particular variable argument C function call form that was required by the test methodology used.

4. Why is the vector reported according to the CRASH categories?

The rationale for reporting the vector according to the CRASH categories is that it is hoped, over time, OS vendors will improve their products so that they have zero defects at the higher levels of the scale.

Figure 3. Response regions for write() system call on QNX 4.22. Both Restart and Silent robustness gaps occur in three different File Handle cases.

Table 2. Parameters and values used for exhaustive testing.

Figure 1. The robustness benchmarks consist of three tasks plus the operating system being tested, run using two computers.

Citations

•Journal Article•10.1109/TSE.2019.2946563

The Art, Science, and Engineering of Fuzzing: A Survey

Valentin J. M. Manès, +6 more

- 01 Nov 2021

- IEEE Transactions on Software Engineerin...

TL;DR: In this article, a unified, general-purpose model of fuzzing together with a taxonomy of the current fuzzing literature is presented, and the design decisions at every stage of the model fuzzer by surveying the related literature and innovations in the art, science, and engineering that make modern-day fuzzers effective.

...read moreread less

349

•Posted Content

The Art, Science, and Engineering of Fuzzing: A Survey

Valentin J. M. Manès, +6 more

- 01 Dec 2018

- arXiv: Cryptography and Security

TL;DR: This paper presents a unified, general-purpose model of fuzzing together with a taxonomy of the current fuzzing literature, and methodically explores the design decisions at every stage of the model fuzzer by surveying the related literature and innovations in the art, science, and engineering that make modern-day fuzzers effective.

...read moreread less

313

Journal Article•10.1109/TSE.2006.113

Emulation of Software Faults: A Field Data Study and a Practical Approach

Joao Duraes, +1 more

- 01 Nov 2006

- IEEE Transactions on Software Engineerin...

TL;DR: A new software fault injection technique (G-SWFIT) based on emulation operators derived from the field study is proposed that consists of finding key programming structures at the machine code-level where high-level software faults can be emulated.

...read moreread less

312

•Proceedings Article•10.1109/FTCS.1998.689474

Automated robustness testing of off-the-shelf software components

N.P. Kropp, +2 more

- 23 Jun 1998

TL;DR: The Ballista methodology for scalable, portable, automated robustness testing of component interfaces is described, an object-oriented approach based on parameter data types rather than component functionality essentially eliminates the need for function-specific test scaffolding.

...read moreread less

250

Proceedings Article•10.1109/FTCS.1999.781031

Comparing the robustness of POSIX operating systems

Philip Koopman, +1 more

- 15 Jun 1999

TL;DR: It is suggested that OS implementations are not completely diverse, and that C library junctions are both less diverse and less robust than system calls.

...read moreread less

197

...

Expand

References

Proceedings Article•10.1109/FTCS.1988.5306

FIAT-fault injection based automated testing environment

Zary Segall, +8 more

- 27 Jun 1988

TL;DR: An automated real-time distributed accelerated fault injection environment (FIAT) is presented as an attempt to provide suitable tools for the validation process and an example of fault tolerant systems such as checkpointing and duplicate and match is used to show its usefulness.

...read moreread less

268

Proceedings Article•10.1109/FTCS.1992.243567

FERRARI: a tool for the validation of system dependability properties

G.A. Kanawati, +2 more

- 08 Jul 1992

TL;DR: FERRARI as mentioned in this paper is a fault and error automatic real-time injector, which can evaluate complex systems by emulating most hardware faults in software, including permanent faults and transient errors.

...read moreread less

250

Proceedings Article•10.1109/FTCS.1995.466945

Measuring robustness of a fault tolerant aerospace system

C.P. Dingman, +2 more

- 27 Jun 1995

TL;DR: A system used to measure the robustness of a fault tolerant aerospace system developed at IBM is described and the data collected during the project is presented.

...read moreread less

Proceedings Article•10.2514/6.1988-4436

Results of software error-data experiments

George B. Finelli

- 01 Sep 1988

TL;DR: In order to evaluate existing software reliability models and proposed modeling approaches, a search was conducted for data on the software failure process, which revealed that the data necessary for this evaluation were not available.

...read moreread less

Proceedings Article•10.1109/WWOS.1989.109278

Mach: a foundation for open systems (operating systems)

Richard F. Rashid, +6 more

- 27 Sep 1989

TL;DR: An alternative approach to building an entire operating system (OS) separating those parts of the OS that control the basic hardware resources (the kernel) from those that determine the unique characteristics of an OS environment, is examined, taking the Mach kernel as an example.

...read moreread less

Comparing operating systems using robustness benchmarks

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What have the authors contributed in "Comparing operating systems using robustness benchmarks" ?

2. What is the way to approach this area?

3. Why was open() not testable under QNX?

4. Why is the vector reported according to the CRASH categories?

5. How many instances of Silent failures were observed in a version of the Mach OS?

6. What is the definition of the Silent class of failure?

7. What is the way to measure the robustness of an OS?

8. What is the definition of an Abort failure?

9. What was the purpose of the previous generation of robustness testing?

10. What is the definition of the Restart class of failure?

11. Why is the emphasis on testing the robustness of an OS when application code provides invalid inputs?

12. What is the purpose of the portable robustness benchmarks?

Figures

Citations

The Art, Science, and Engineering of Fuzzing: A Survey

The Art, Science, and Engineering of Fuzzing: A Survey

Emulation of Software Faults: A Field Data Study and a Practical Approach

Automated robustness testing of off-the-shelf software components

Comparing the robustness of POSIX operating systems

References

FIAT-fault injection based automated testing environment

FERRARI: a tool for the validation of system dependability properties

Measuring robustness of a fault tolerant aerospace system

Results of software error-data experiments

Mach: a foundation for open systems (operating systems)

Related Papers (5)

An empirical study of the reliability of UNIX utilities

Comparing the robustness of POSIX operating systems

Automated robustness testing of off-the-shelf software components

Xception: a technique for the experimental evaluation of dependability in modern computers

An approach towards benchmarking of fault-tolerant commercial systems