Open AccessJournal Article10.1103/PHYSREVE.91.013306

Idle waves in high-performance computing

- 20 Jan 2015

- Vol. 91, Iss: 1, pp 013306-013306

TL;DR: This study provides a description of the large number of processes in parallel scientific applications as a continuous medium and identifies the propagation of idle waves through processes in scientific applications with a local information exchange between the two processes.

Abstract: The vast majority of parallel scientific applications distributes computation among processes that are in a busy state when computing and in an idle state when waiting for information from other processes. We identify the propagation of idle waves through processes in scientific applications with a local information exchange between the two processes. Idle waves are nondispersive and have a phase velocity inversely proportional to the average busy time. The physical mechanism enabling the propagation of idle waves is the local synchronization between two processes due to remote data dependency. This study provides a description of the large number of processes in parallel scientific applications as a continuous medium. This work also is a step towards an understanding of how localized idle periods can affect remote processes, leading to the degradation of global performance in parallel scientific applications.

AI Agents for this Paper

Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps

Most frequently asked questions

1. What are the contributions mentioned in the paper "Idle waves in high-performance computing" ?

In this paper, the authors provide a description of a large number of processes in a scientific application as a continuous medium and identify the presence of idle waves propagating through processes via local synchronization.

2. What future works have the authors mentioned in the paper "Idle waves in high-performance computing" ?

As part of future work, idle waves in application with more complex communication patterns will be studied.. The preparation of such model should be addressed in future works.. Ω2 − v2P k2 = 0. ( 3 ) The numerical experiments also suggest the presence of some damping mechanisms ( see Fig. 2 ).. The authors suggest that many cases of unexpected performance degradation of distributed scientific applications can be explained in terms of idle waves.

FIG. 1. (Color online) Physical mechanism for the generation of idle waves in a computation distributed over nine processes. Time is presented in the x axis, while the rank of the process is presented in the y axis. The busy periods are represented by the dark gray (red), idle periods in black, and communication in light gray (blue) lines. An extended busy period in the process with rank four generates two idle periods on nearest-neighbor processes (rank three and five). These idle periods propagate to other processes by local synchronization of nearest-neighbor processes as a wave.

FIG. 4. (Color online) 1/vP dependency on 〈TB〉 (top panel), L (bottom panel black line), and o (bottom panel red line). Linear best fits are superimposed to the data points.

FIG. 5. Interaction between two idle waves in a simulation with 384 processes, N = 100 000, and a driven perturbation of the busy period on rank zero. The plot presents time on the x axis and process rank on the y axis. The white and black colors indicate the busy and idle periods, respectively. The insert in the picture represents an enlargement of the rank-time plot during the idle wave interaction.

FIG. 6. (Color online) Numerical dispersion relation obtained from a simulation with communication with nearest-neighbor communication and a collective operation at each computational cycle with N = 100 000 on 96 processes.

FIG. 7. (Color online) Numerical dispersion relation obtained from a simulation using nonblocking send and receive for communication between nearest-neighbor processes with N = 100 000 on 96 processes. In this case, the propagation speed of idle waves is vP ±0.0027 rank/μs.

FIG. 3. (Color online) Idle waves in a simulation with 2040 processes with N = 100 000 and without a perturbation. In the top panel, a contour plot of busy periods. The insert presents an enlargement of the contour plot including 168 processes. The bottom panel shows the numerical dispersion relation of idle waves.

Citations

Proceedings Article•10.1145/2831129.2831131

A data streaming model in MPI

Ivy Bo Peng, +4 more

- 15 Nov 2015

TL;DR: This work designed and implemented a data streaming library MPIStream atop MPI to allocate data producers and consumers, to stream data continuously or irregularly and to process data at run-time, and developed a parallel stream benchmark to measure data processing rate.

...read moreread less

Proceedings Article•10.1109/CLUSTER.2015.63

The Cost of Synchronizing Imbalanced Processes in Message Passing Systems

Ivy Bo Peng, +2 more

- 08 Sep 2015

TL;DR: In this study, f-way tournament barriers provided the best imbalance absorption rate and convenient communication time and it is identified that synchronization transits from communication-dominated to imbalance-dominated when the standard deviation of imbalance distribution is more than twice the communication time for one message.

...read moreread less

•Proceedings Article•10.1109/CLUSTER.2019.8890995

Propagation and Decay of Injected One-Off Delays on Clusters: A Case Study.

Ayesha Afzal, +2 more

- 25 May 2019

- arXiv: Performance

TL;DR: In this article, the authors investigate various traces collected from synthetic benchmarks that mimic real applications on simulated and real message-passing systems in order to pinpoint the mechanisms behind delay propagation.

...read moreread less

•Book Chapter•10.1007/978-3-030-78713-4_19

Analytic Modeling of Idle Waves in Parallel Programs: Communication, Cluster Topology, and Noise Impact

Ayesha Afzal, +2 more

- 04 Mar 2021

- arXiv: Distributed, Parallel, and Cluste...

TL;DR: In this article, the authors study the propagation mechanisms of idle waves across the ranks of MPI-parallel programs and present a validated analytic model for their propagation velocity with respect to communication parameters and topology.

...read moreread less

•Proceedings Article•10.1109/CLUSTER.2019.8890995

Propagation and Decay of Injected One-Off Delays on Clusters: A Case Study

Ayesha Afzal, +2 more

- 01 Sep 2019

TL;DR: In this paper, the authors investigate various traces collected from synthetic benchmarks that mimic real applications on simulated and real message-passing systems in order to pin-point the mechanisms behind delay propagation.

...read moreread less

...

Expand

References

Book Chapter•10.1017/CBO9781139207249.009

I and J

William Marsden

- 01 Jan 2012

154.7K

•Book

Discrete-Time Signal Processing

Alan V. Oppenheim, +1 more

- 01 Jan 1989

TL;DR: In this paper, the authors provide a thorough treatment of the fundamental theorems and properties of discrete-time linear systems, filtering, sampling, and discrete time Fourier analysis.

...read moreread less

11.8K

Journal Article•10.1119/1.1972241

The Feynman Lectures on Physics; Vol. I

Richard Phillips Feynman, +3 more

- 01 Sep 1965

- American Journal of Physics

1.7K

•Book

Solving Problems on Concurrent Processors

Geoffrey C. Fox

- 01 Mar 1988

TL;DR: Self steering systems for marine craft are described which comprise wind direction sensing and course setting means, servo rudder means operatively connected to a rudder and connecting means linking the servo rubber means to the wind sensing andcourse setting means.

...read moreread less

1.3K

•Proceedings Article

Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis

Barry V. Hess

- 13 Nov 2010

144

Idle waves in high-performance computing

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What are the contributions mentioned in the paper "Idle waves in high-performance computing" ?

2. What future works have the authors mentioned in the paper "Idle waves in high-performance computing" ?

Figures

Citations

A data streaming model in MPI

The Cost of Synchronizing Imbalanced Processes in Message Passing Systems

Propagation and Decay of Injected One-Off Delays on Clusters: A Case Study.

Analytic Modeling of Idle Waves in Parallel Programs: Communication, Cluster Topology, and Noise Impact

Propagation and Decay of Injected One-Off Delays on Clusters: A Case Study

References

I and J

Discrete-Time Signal Processing

The Feynman Lectures on Physics; Vol. I

Solving Problems on Concurrent Processors

Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis

Related Papers (5)

LogGOPSim: simulating large-scale applications in the LogGOPS model

The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q

The Influence of Operating Systems on the Performance of Collective Operations at Extreme Scale

Roofline: an insightful visual performance model for multicore architectures

Characterizing the Influence of System Noise on Large-Scale Applications by Simulation