A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream
TL;DR: A simple, message-optimal algorithm for maintaining a random sample from a large data stream whose input elements are distributed across multiple sites that communicate via a central coordinator is presented.
read more
Abstract: We present a simple, message-optimal algorithm for maintaining a random sample from a large data stream whose input elements are distributed across multiple sites that communicate via a central coordinator. At any point in time, the set of elements held by the coordinator represent a uniform random sample from the set of all the elements observed so far. When compared with prior work, our algorithms asymptotically improve the total number of messages sent in the system. We present a matching lower bound, showing that our protocol sends the optimal number of messages up to a constant factor with large probability. We also consider the important case when the distribution of elements across different sites is non-uniform, and show that for such inputs, our algorithm significantly outperforms prior solutions.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Communication-Efficient Probabilistic Algorithms: Selection, Sampling, and Checking
Lorenz Hübschle-Schneider
- 01 Jan 2020
TL;DR: Diese Arbeit ist durch einen wachsenden Bedarf an Kommunikationseffizienz motiviert, dass der auf das Netzwerk and seine Nutzung zuruckzufuhrende Anteil sowohl der Anschaffungskosten als auch des Energieverbrauchs von Supercomputern and der Laufzeit verteilter Anwendung
References
•Book
The Art of Computer Programming, Volume 2: Seminumerical Algorithms
Donald E. Knuth
- 01 Jan 1981
4.4K
Random sampling with a reservoir
TL;DR: Theoretical and empirical results indicate that Algorithm Z outperforms current methods by a significant margin, and an efficient Pascal-like implementation is given that incorporates these modifications and that is suitable for general use.
BlinkDB: queries with bounded errors and bounded response times on very large data
Sameer Agarwal,Barzan Mozafari,Aurojit Panda,Henry Milner,Samuel Madden,Ion Stoica +5 more
- 15 Apr 2013
TL;DR: BlinkDB allows users to trade-off query accuracy for response time, enabling interactive queries over massive data by running queries on data samples and presenting results annotated with meaningful error bars.
Applications and explanations of Zipf's law
David M. W. Powers
- 11 Jan 1998
TL;DR: It is demonstrated how Zipf's analysis can be extended to include some of these phenomena, and closer examination uncovers systematic deviations from its normative form.
455
Sampling from a moving window over streaming data
Brian Babcock,Mayur Datar,Rajeev Motwani +2 more
- 06 Jan 2002
TL;DR: This work introduces the problem of sampling from a moving window of recent items from a data stream and develops two algorithms, the first of which, "chain-sample", extends reservoir sampling to deal with the expiration of data elements from the sample and the second, "priority- sample", works even when the number of elements in the window can vary dynamically over time.
425