Pattern queries are widely used in complex event processing (CEP) systems. Existing pattern matching techniques, however, can provide only limited performance for expensive queries in real-world applications, which may involve Kleene closure patterns, flexible event selection strategies, and events with imprecise timestamps. To support these expensive queries with high performance, we begin our study by analyzing the complexity of pattern queries, with a focus on the fundamental understanding of which features make pattern queries more expressive and at the same time more computationally expensive. This analysis allows us to identify performance bottlenecks in processing those expensive queries, and provides key insights for us to develop a series of optimizations to mitigate those bottlenecks. Microbenchmark results show superior performance of our system for expensive pattern queries while most state-of-the-art systems suffer from poor performance. A thorough case study on Hadoop cluster monitoring further demonstrates the efficiency and effectiveness of our proposed techniques.

/pdf/on-complexity-and-optimization-of-expensive-queries-in-4508gfo0sq.pdf

On complexity and optimization of expensive queries in complex event processing

A computer is programmed to accept queries over streams of, data structured as per a predetermined syntax (e.g. defined in XML). The computer is further programmed to execute such queries continually (or periodically) on data streams of tuples containing structured data that conform to the same predetermined syntax. In many embodiments, the computer includes an engine that exclusively processes only structured data, quickly and efficiently. The computer invokes the structured data engine in two different ways depending on the embodiment: (a) directly on encountering a structured data operator, or (b) indirectly by parsing operands within the structured data operator which contain path expressions, creating a new source to supply scalar data extracted from structured data, and generating additional trees of operators that are natively supported, followed by invoking the structured data engine only when the structured data operator in the query cannot be fully implemented by natively supported operators.

Processing XML data stream(s) using continuous queries in a data stream management system

A data flow is received from a mobile network relating to a mobile subscriber. Subscriber data is received for the subscriber identifying a service path corresponding to the subscriber and at least one service policy corresponding to the subscriber, the service path including a set of network service nodes in a plurality of network service nodes. Packets of the data flow are routed according to the service path, the packets corresponding to a request for a resource. At least one packet is appended with service header data identifying the service policy. Each service node performs at least one service based on received request data, each service node in the set of service nodes performing a service defined in a service subscription of the subscriber. At least one particular service node in the set of network service nodes performs a particular service based at least in part on the service policy.

Mobile service routing in a network environment

Current pattern-detection proposals for streaming data recognize the need to move beyond a simple regular-expression model over strictly ordered input. We continue in this direction, relaxing restrictions present in some models, removing the requirement for ordered input, and permitting stream revisions (modification of prior events). Further, recognizing that patterns of interest in modern applications may change frequently over the lifetime of a query, we support updating of a pattern specification without blocking input or restarting the operator. Our new pattern operator (called AFA) is a streaming adaptation of a non-deterministic finite automaton (NFA) where additional schema-based user-defined information, called a register, is accessible to NFA transitions during execution. AFAs support dynamic patterns, where the pattern itself can change over time. We propose clean order-agnostic pattern-detection semantics for AFAs, with new algorithms that allow a very efficient implementation, while retaining significant expressiveness and supporting native handling of out-of-order input, stream revisions, dynamic patterns, and several optimizations. Experiments on Microsoft StreamInsight show that we achieve event rates of more than 200K events/sec (up to 5x better than simpler schemes). Our dynamic patterns give up to orders-of-magnitude better throughput than solutions such as operator restart, and our other optimizations are very effective, incurring low memory and latency.

/pdf/high-performance-dynamic-pattern-matching-over-disordered-4zykbt5yqb.pdf

High-performance dynamic pattern matching over disordered streams

Techniques for facilitating communication between one or more client applications and an event processing server. In one set of embodiments, a framework is provided that enables a client application to communicate with an event processing server via a standard database connectivity API, such as Java Database Connectivity (JDBC).

Standardized database connectivity support for an event processing server

Data stream management systems are useful when large volumes of data need to be processed in real time. Examples include monitoring network traffic, monitoring financial transactions, and analyzing large scale scientific data feeds. These applications have varying data rates and often show bursts of high activity that overload the system, often during the most critical instants (e.g., network attacks, financial spikes) for analysis. Therefore, load shedding is necessary to preserve the stability of the system, gracefully degrade its performance and extract answers. Existing methods for load shedding in a general purpose data stream query system use random sampling of tuples, essentially independent of the query. While this technique is acceptable for some queries, the results may be meaningless or even incorrect for other queries, lit principle, a number of different query-dependent sampling methods exist, but they work only for particular queries. In this paper, we show how to perform query-aware sampling (semantic sampling) which works in general. We present methods for analyzing any given query, choosing sampling methods judiciously, and reconciling the sampling methods required by different queries in a query set. We conclude with experiments on a highspeed data stream that demonstrate with different query sets that our method produces accurate results while decreasing the load significantly.

Query-Aware Sampling for Data Streams

We present an efficient algorithm for regular expression matching on streams with out of order data, while maintaining a small state and without complete stream reconstruction. We have implemented three versions of the algorithm - sequential, parallel and mixed - and show by experimental study that the algorithms are highly effective in matching regular expressions on IP packet streams.

Monitoring Regular Expressions on Out-of-Order Streams

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

T. Johnson

Author Tools

Chat about Author

Papers

Query-Aware Sampling for Data Streams

Monitoring Regular Expressions on Out-of-Order Streams