Book Chapter10.1007/978-3-540-74469-6_41
Integrating a stream processing engine and databases for persistent streaming data management
Yousuke Watanabe,Shinichi Yamada,Hiroyuki Kitagawa,Toshiyuki Amagasa +3 more
- 03 Sep 2007
- pp 414-423
13
TL;DR: This paper describes the data stream management system, which employs an architecture combining a stream processing engine and DBMS, and a proposed query language that supports not only filtering, join, and projection over data streams, but also continuous persistence requirements for stream data.
read more
Abstract: Because of increased stream data, managing stream data has become quite important. This paper describes our data stream management system, which employs an architecture combining a stream processing engine and DBMS. Based on the architecture, the system processes both continuous queries and traditional one-shot queries. Our proposed query language supports not only filtering, join, and projection over data streams, but also continuous persistence requirements for stream data. Users can also specify continuous queries that integrate streaming data and historical data stored in DBMS. Another contribution of this paper is feasibility validation of queries. Processing queries on streams with frequent inputs may cause the system to overflow its capacity. Specifically, the maximum writing rate to DBMS is a significant bottleneck when we try to store stream data into DBMS. Our system detects infeasible queries in advance.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Distributed Join Processing Between Streaming and Stored Big Data Under the Micro-Batch Model
TL;DR: The proposed DS-join optimizes the join operation by minimizing the data shuffling, managing a cache in a distributed SPE, parallelizing the join processing, and balancing the load between the SPE and the external database system.
Lineage-based Probabilistic Event Stream Processing
Zhitao Shen,Hideyuki Kawashima,Hiroyuki Kitagawa +2 more
- 27 Apr 2008
TL;DR: A query language to support probabilistic queries for composite event stream matching that allows users to express Kleene closure patterns for complex event detection in physical world is proposed and a performance evaluation of the method comparing with naive approach is conducted.
17
IoT-enabled directed acyclic graph in spark cluster
TL;DR: This paper presents an IoT-enabled Directed Acyclic Graph (I-DAG) technique that labels micro-batches at the stage of building a stream event and arranges stream elements with event labels, and provides an effective stream event heterogeneous solution for IoT- enabled datasets in spark clusters.
Meet me halfway: split maintenance of continuous views
Christian Winter,Tobias Schmidt,Thomas Neumann,Alfons Kemper +3 more
- 01 Jul 2020
TL;DR: This paper proposes a new type of view specialized for queries involving high-velocity inputs, called continuous view, and shows that split maintenance can outperform even dedicated stream processing engines on analytical workloads, all while still offering similar insert rates.
A video stream management system for heterogeneous information integration environments
Yousuke Watanabe,Ryo Akiyama,Kousuke Ohki,Hiroyuki Kitagawa +3 more
- 31 Jan 2008
TL;DR: This work proposes a video stream management system that provides a SQL-like query interface for heterogeneous information sources including video streams, and proposes a dynamic source selection scheme for some applications, like moving object tracking with video streams.
6
References
Aurora: a new model and architecture for data stream management
Daniel J. Abadi,Don Carney,Uğur Çetintemel,Mitch Cherniack,Christian Convey,Sangdon Lee,Michael Stonebraker,Nesime Tatbul,Stan Zdonik +8 more
- 01 Aug 2003
TL;DR: The basic processing model and architecture of Aurora, a new system to manage data streams for monitoring applications, are described and a stream-oriented set of operators are described.
•Proceedings Article
The Design of the Borealis Stream Processing Engine
Daniel J. Abadi,Yanif Ahmad,Magdalena Balazinska,Mitch Cherniack,Jeong-Hyon Hwang,Wolfgang Lindner,Anurag S. Maskey,Alexander Rasin,Esther Ryvkina,Nesime Tatbul,Ying Xing,Stan Zdonik +11 more
- 01 Jan 2005
TL;DR: This paper outlines the basic design and functionality of Borealis, and presents a highly flexible and scalable QoS-based optimization model that operates across server and sensor networks and a new fault-tolerance model with flexible consistency-availability trade-offs.
The CQL continuous query language: semantic foundations and query execution
Arvind Arasu,Shivnath Babu,Jennifer Widom +2 more
- 01 Jun 2006
TL;DR: This paper presents the structure of CQL's query execution plans as well as details of the most important components: operators, interoperator queues, synopses, and sharing of components among multiple operators and queries.
•Proceedings Article
TelegraphCQ: Continuous Dataflow Processing for an Uncertain World.
Sirish Chandrasekaran,Owen Cooper,Amol Deshpande,Michael J. Franklin,Joseph M. Hellerstein,Wei Hong,Sailesh Krishnamurthy,Samuel Madden,Vijayshankar Raman,Frederick Reiss,Mehul A. Shah +10 more
- 01 Jan 2003
TL;DR: The next generation Telegraph system, called TelegraphCQ, is focused on meeting the challenges that arise in handling large streams of continuous queries over high-volume, highly-variable data streams and leverages the PostgreSQL open source code base.
1.2K