Optimizing Multiple Distributed Stream Queries Using Hierarchical Network Partitions

TL;DR: This paper proposes two algorithms - top-down and bottom-up which utilize hierarchical network partitions to provide scalable query optimization and establishes the bounds on the search-space and shows the sub-optimality of these algorithms.

Abstract: We consider the problem of query optimization in distributed data stream systems where multiple continuous queries may be executing simultaneously. In order to achieve the best performance, query planning (such as join ordering) must be considered in conjunction with deployment planning (e.g., assigning operators to physical nodes with optimal ordering). However, such a combination involves not only a large number of network nodes but also many query operators, resulting in an extremely large search space for optimal solutions. Our paper aims at addressing this problem by utilizing hierarchical network partitions. We propose two algorithms - top-down and bottom-up which utilize hierarchical network partitions to provide scalable query optimization. Formal analysis is presented to establish the bounds on the search-space and to show the sub-optimality of our algorithms. Through simulations and experiments using a prototype deployed on Emulab we demonstrate the effectiveness of our algorithms.