Spotting code optimizations in data-parallel pipelines through PeriSCOPE
Zhenyu Guo,Xuepeng Fan,Rishan Chen,Jiaxing Zhang,Hucheng Zhou,Sean McDirmid,Chang Liu,Wei Lin,Jingren Zhou,Lidong Zhou +9 more
- 08 Oct 2012
- pp 121-133
TL;DR: PeriScope as mentioned in this paper automatically optimizes a data-parallel program's procedural code in the context of data flow that is reconstructed from the program's pipeline topology, so that less data is transferred between pipeline stages.
read more
Abstract: To minimize the amount of data-shuffling I/O that occurs between the pipeline stages of a distributed data-parallel program, its procedural code must be optimized with full awareness of the pipeline that it executes in. Unfortunately, neither pipeline optimizers nor traditional compilers examine both the pipeline and procedural code of a data-parallel program so programmers must either hand-optimize their program across pipeline stages or live with poor performance. To resolve this tension between performance and programmability, this paper describes PeriSCOPE, which automatically optimizes a data-parallel program's procedural code in the context of data flow that is reconstructed from the program's pipeline topology. Such optimizations eliminate unnecessary code and data, perform early data filtering, and calculate small derived values (e.g., predicates) earlier in the pipeline, so that less data--sometimes much less data--is transferred between pipeline stages. We describe how PeriSCOPE is implemented and evaluate its effectiveness on real production jobs.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Efficient coflow scheduling with Varys
Mosharaf Chowdhury,Yuan Zhong,Ion Stoica +2 more
- 17 Aug 2014
TL;DR: This paper presents Varys, a system that enables data-intensive frameworks to use coflows and the proposed algorithms while maintaining high network utilization and guaranteeing starvation freedom, and outperforms non-preemptive coflow schedulers by more than 5X.
Coflow: a networking abstraction for cluster applications
Mosharaf Chowdhury,Ion Stoica +1 more
- 29 Oct 2012
TL;DR: CoFlow as discussed by the authors is a networking abstraction to express the communication requirements of prevalent data parallel programming paradigms, which makes it easier for the applications to convey their communication semantics to the network, which in turn enables the network to better optimize common communication patterns.
FACADE: A Compiler and Runtime for (Almost) Object-Bounded Big Data Applications
Khanh Nguyen,Kai Wang,Yingyi Bu,Lu Fang,Jianfei Hu,Guoqing Xu +5 more
- 14 Mar 2015
TL;DR: A novel compiler framework, called Facade, that can generate highly-efficient data manipulation code by automatically transforming the data path of an existing Big Data application by leading to significantly reduced memory management cost and improved scalability.
Interruptible tasks: treating memory pressure as interrupts for highly scalable data-parallel programs
Lu Fang,Khanh Nguyen,Guoqing Xu,Brian Demsky,Shan Lu +4 more
- 04 Oct 2015
TL;DR: A thorough evaluation demonstrates the effectiveness of ITask, which has helped real-world Hadoop programs survive 13 out-of-memory problems reported on StackOverflow and shows that the ITask-based versions are 1.5--3x faster and scale to 3--24x larger datasets than their regular counterparts.
62
Implicit Parallelism through Deep Language Embedding
Alexander Alexandrov,Andreas Kunft,Asterios Katsifodimos,Felix Schüler,Lauritz Thamsen,Odej Kao,Tobias Herb,Volker Markl +7 more
- 27 May 2015
TL;DR: This paper proposes a language for complex data analysis embedded in Scala, which allows for declarative specification of dataflows and hides the notion of data-parallelism and distributed runtime behind a suitable intermediate representation.
References
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
- 06 Dec 2004
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
•Book
Compilers: Principles, Techniques, and Tools
Alfred V. Aho,Ravi Sethi,Jeffrey D. Ullman +2 more
- 01 Jan 1986
TL;DR: This book discusses the design of a Code Generator, the role of the Lexical Analyzer, and other topics related to code generation and optimization.
9.7K
Dryad: distributed data-parallel programs from sequential building blocks
Michael Isard,Mihai Budiu,Yuan Yu,Andrew Birrell,Dennis Fetterly +4 more
- 21 Mar 2007
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.
•Book
Advanced Compiler Design and Implementation
Steven S. Muchnick
- 01 Jan 1997
TL;DR: Advanced Compiler Design and Implementation by Steven Muchnick Preface to Advanced Topics
2.6K
Efficiently computing static single assignment form and the control dependence graph
TL;DR: In this article, the authors present new algorithms that efficiently compute static single assignment forms and control dependence graphs for arbitrary control flow graphs using the concept of {\em dominance frontiers} and give analytical and experimental evidence that these data structures are usually linear in the size of the original program.