An elastic and traffic-aware scheduler for distributed data stream processing in heterogeneous clusters

Precision medicine (PM) is an innovative medical approach that considers differences in the individuals’ omics, medical histories, lifestyles, and environmental information in treating diseases. To fully achieve the envisaged gains of PM, various contemporary and future technologies have to be employed, among which are nanotechnology, sensor network, big data, and artificial intelligence. These technologies and other applications require a communication network that will enable them to work in tandem for the benefit of PM. Hence, communication technology serves as the nervous system of PM, without which the entire system collapses. Therefore, it is essential to explore and determine the candidate communication technology requirements that can guarantee the envisioned gains of PM. To the best of our knowledge, no work exploring how communication technology directly impacts the development and deployment of PM solutions exists. This survey paper is designed to stimulate discussions on PM from the communication engineering perspective. We introduce the fundamentals of PM and the demands in terms of quality of service that each of the enabling technologies of PM places on the communication network. We explore the information in the literature to suggest the ideal metric values of the key performance indicators for the implementation of the different components of PM. The comparative analysis of the suitability of the contemporary and future communication technologies for PM implementation is discussed. Finally, some open research challenges for the candidate communication technologies that will enable the full implementation of PM solutions are highlighted. 

Enabling Precision Medicine via Contemporary and Future Communication Technologies: A Survey

With the start of big data era, data stream computing has emerged as a well-known approach to optimize data-intensive workflows. Apache STORM is an open-source real-time distributed computation system for processing data streams and has been opted by famous organizations such as Twitter, Yahoo, Alibaba, Baidu, Groupon. The workflows are implemented as topologies in STORM. The main aspect that controls the execution performance of a workflow in STORM is the strategy of scheduling the topology components (spout and bolts). In this paper, we evaluate and analyze the performance of our algorithm Partition-based Data-intensive Workflow optimization Algorithm (PDWA) in Apache STORM using a use case workflow, EURExpressII. It is a real-world application-based workflow that builds a transcriptome-wide atlas of gene expression for the developing mouse embryo established by ribonucleic acid (RNA) in situ hybridization. Our proposed algorithm, PDWA, partitions the application task graph so that the data movement between partitions is minimum. Each partition is then mapped on one machine for the execution of tasks of that partition. It provides minimum execution time for that particular partition. Partial task duplication is also part of this algorithm that enhances the performance. A STORM-based computing cluster is developed in OpenStack cloud which is used as a computing environment. The performance of PDWA-based optimizer is evaluated with the data sets of different sizes. The achieved results show that PDWA performs with 21% improved average execution time for different sizes of data sets and varying execution nodes. In addition, the comparative results show that on average the efficiency of PDWA is 20.4% higher as compared to STORM default scheduler (SDS).

Use case-based evaluation of workflow optimization strategy in real-time computation system

Cost-efficient Scheduling of Streaming Applications in Apache Flink on Cloud

Stream processing is a new memory computing paradigm that deals with dynamic data streams efficiently. Storm is one of the stream processing frameworks, but the default stream processing scheduler of storm also has some problems. For example, it does not consider reducing the cost in the cloud environment while ensuring the performance requirements. In this paper, a cost-efficient scheduling algorithm for Storm framework (CE-Storm) is proposed to reduce cost while satisfying deadline constrain. First, a new cost-efficient model (including resources usage cost, energy cost and communication cost) based on Storm framework is built. Then, based on the cost model, a cost-efficient scheduling algorithm which integrated resource monitoring module and communication detection module is designed. The nodes in the cluster are prioritized according to the cost-efficient information, and the nodes with higher priority are assigned tasks first to minimize the total cost of the cluster. Furthermore, this algorithm also reduces the communication cost between nodes and improves the cost effectiveness of the Storm cluster. We have performed extensive experiments on Storm clusters using Hibench’s workloads in cloud environment. The result shows that the cost consumption of Storm clusters in cloud environment is reduced by 19.25% on average compared with the traditional scheduling algorithm. In others words, the proposed algorithms effectively improve the cost efficiency of Storm cluster in the cloud environment while satisfying the performance constrains.

A cost-efficient scheduling algorithm for streaming processing applications on cloud

With the massive growth of big data applications, the requirement for data processing speed is getting higher and higher in stream computing systems. The Storm, as one of the most popular distributed stream computing systems, has received more attention. However, the Storm's traditional scheduling strategy is not ideal for processing a large volume of streaming data. The resource scheduling in a distributed stream computing system should consider not only node allocation status but also fluctuating input rates of data stream. To address this problem, this paper has completed the following work: (1) A performance model La-Stream (latency-sensitive elastic adaptive scheduling) is proposed and built by adopting a quantitative method for calculating the amount of computation required between task map nodes and node communication. (2) A La-Stream based algorithm is proposed. The algorithm dynamically plans a resource allocation scheme with minimal data processing latency among available resources to achieve optimal allocation. (3) Three functional modules of La-steam are proposed and implemented: module Monitor, module Optimizer and module Scheduler. The three modules are integrated into the Storm platform with minimal overhead. Several sets of experiments are conducted, verifying the feasibility and effectiveness of La-Stream.

A Latency-Sensitive Elastic Adaptive Scheduling in Distributed Stream Computing Systems

High accuracy of multi-dimensional feature forecasting is very important for online scheduling in big data stream computing environments. Currently, most stream computing systems only consider historical features, with future features ignored. In this paper, a strategy to improve accuracy of multi-dimensional feature forecasting for online data stream is proposed. It includes the following contributions. (1) Profiling principles of accurate future feature forecasting objectives from multi-dimensional big data streams. (2) Extracting future features from multi-dimensional historical features of data stream via an improved hybrid IGA-BP (Immune Genetic Algorithm and Back Propagation) algorithm. (3) Evaluating accuracy of future feature forecasting and acceptable latency objectives in big data stream computing environments. Experimental results conclusively demonstrate the efficiency and effectiveness of the proposed strategy.

A Strategy to Improve Accuracy of Multi-dimensional Feature Forecasting in Big Data Stream Computing Environments

State and runtime-aware scheduling is one of the problems that is hard to resolve in elastic big data stream computing systems, as the state of each vertex is different, and the arrival rate of data streams fluctuates over time. A state and runtime-aware scheduling framework should be able to dynamically adapt to the fluctuation of the arrival rate of data streams and be aware of vertex states and resource availability. Currently, there is an increasing number of research work focusing on application scheduling in stream computing systems, however, this problem is still far from being completely solved. In this paper, we focus on the state of vertex in applications and the runtime feature of resources in a data center, and propose a state and runtime-aware scheduling framework (Sra-Stream) for elastic streaming computing systems, which incorporates the following features: (1) Profiling mathematical relationships between the system response time and the arrival rate of data streams, and identifying relevant resource constraints to meet the low response time and high throughput objectives. (2) Classifying vertex into stateless vertex or stateful vertex from a quantitative perspective, and achieving vertex parallelization by considering the state of the vertex. (3) Demonstrating a proposed stream application scheduling scheme consisting of a modified first-fit based runtime-aware data tuple scheduling strategy at the initial stage, and a maximum latency-sensitive based runtime-aware data stream scheduling strategy at the online stage, by considering the current scheduling status of the application. (4) Evaluating the achievement levels of low response time and high throughput objectives in a real-world elastic stream computing system. Experimental results conclusively demonstrate that the proposed Sra-Stream provides significant performance improvements on achieving the low system response time and high system throughput.

State and runtime-aware scheduling in elastic stream computing systems

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Dawei Sun

Author Tools

Chat about Author

Papers

A Latency-Sensitive Elastic Adaptive Scheduling in Distributed Stream Computing Systems

A Strategy to Improve Accuracy of Multi-dimensional Feature Forecasting in Big Data Stream Computing Environments

State and runtime-aware scheduling in elastic stream computing systems