Book Chapter10.1007/978-3-319-42553-5_24
SGraph: A Distributed Streaming System for Processing Big Graphs
Cheng Chen,Cheng Chen,Hejun Wu,Hejun Wu,Dyce Jing Zhao,Da Yan,James Cheng +6 more
- 29 Jul 2016
- pp 285-294
4
TL;DR: SGraph is a distributed streaming graph processing system built on top of Spark that can process graphs with up to 1.5 billion edges on small clusters with several low-cost commodity PCs, whereas existing systems may require up to tens or hundreds of high-end machines.
read more
Abstract: Big graph processing has been widely used in various computational domains, ranging from language modeling to social networks. Graph-parallel systems have been proposed to process such big graphs on clusters with up to hundreds of nodes. However, the size of a big graph often exceeds the available main memories in a small cluster. As a consequence, task failures happen frequently. To address this problem, we propose SGraph, a distributed streaming graph processing system built on top of Spark. SGraph introduces a streaming data model to avoid loading all of the graph data which may exceed the available RAM space. In addition, SGraph leverages an edge-centric scatter-gather computing model that can be used to conveniently implement graph algorithms. Experiments demonstrate that SGraph can process graphs with up to 1.5 billion edges on small clusters with several low-cost commodity PCs, whereas existing systems may require up to tens or hundreds of high-end machines. Furthermore, SGraph is up to 2.3 times faster than existing systems.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
iTurboGraph: Scaling and Automating Incremental Graph Analytics
Seongyun Ko,Taesung Lee,Kijae Hong,Wonseok Lee,In Seo,Jiwon Seo,Wook-Shin Han +6 more
- 09 Jun 2021
TL;DR: ŁNGA as mentioned in this paper is a domain-specific language for incremental neighbor-centric graph analytics (NGA) for large-scale graph analytics, which can be used to solve the limitations of previous systems: lack of usability due to the difficulties in programming incremental algorithms for NGA and limited scalability and efficiency due to maintaining intermediate results for graph traversals in NGA.
7
•Posted Content
G-Tran: Making Distributed Graph Transactions Fast.
Hongzhi Chen,Changji Li,Chenguang Zheng,Chenghuan Huang,Juncheng Fang,James Cheng,Jian Zhang +6 more
TL;DR: G-Tran as mentioned in this paper is an RDMA-enabled distributed in-memory graph database with serializable and snapshot isolation support, which adopts a fully decentralized architecture that leverages RDMA to process distributed transactions with the MPP model.
1
Journal Article
G-Tran: A High Performance Distributed Graph Database with a Decentralized Architecture
Hongzhi Chen,Changji Li,Che Zheng,Chenghuan Huang,Juncheng Fang,James Cheng,Jian Zhang +6 more
TL;DR: G-Tran is presented, a remote direct memory access (RDMA)-enabled distributed in-memory graph database with serializable and snapshot isolation support, and a new multi-version optimistic concurrency control (MV-OCC) protocol with two optimizations to address the issue of large read/write sets in graph transactions.
G-tran
Hongzhi Chen,Changji Li,Chenguang Zheng,Chenghuan Huang,Juncheng Fang,James Cheng,Jian Zhang +6 more
TL;DR: G-Tran is presented, a remote direct memory access (RDMA)-enabled distributed in-memory graph database with serializable and snapshot isolation support and a new multi-version optimistic concurrency control (MV-OCC) protocol with two optimizations to address the issue of large read/write sets in graph transactions.
References
•Proceedings Article
The PageRank Citation Ranking : Bringing Order to the Web
Lawrence Page,Sergey Brin,Rajeev Motwani,Terry Winograd +3 more
- 11 Nov 1999
TL;DR: This paper describes PageRank, a mathod for rating Web pages objectively and mechanically, effectively measuring the human interest and attention devoted to them, and shows how to efficiently compute PageRank for large numbers of pages.
16.4K
What is Twitter, a social network or a news media?
Haewoon Kwak,Changhyun Lee,Hosung Park,Sue Moon +3 more
- 26 Apr 2010
TL;DR: In this paper, the authors have crawled the entire Twittersphere and found a non-power-law follower distribution, a short effective diameter, and low reciprocity, which all mark a deviation from known characteristics of human social networks.
7.5K
•Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
- 25 Apr 2012
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
- 06 Jun 2010
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
PowerGraph: distributed graph-parallel computation on natural graphs
Joseph E. Gonzalez,Yucheng Low,Haijie Gu,Danny Bickson,Carlos Guestrin +4 more
- 08 Oct 2012
TL;DR: This paper describes the challenges of computation on natural graphs in the context of existing graph-parallel abstractions and introduces the PowerGraph abstraction which exploits the internal structure of graph programs to address these challenges.