We are currently living in the big data era, in which it has become more necessary than ever to develop “smart” schedulers. It is common knowledge that the default Storm scheduler, as well as a large number of static schemes, has presented certain deficiencies. One of the most important of these deficiencies is the weakness in handling cases in which system changes occur. In such a scenario, some type of re-scheduling is necessary to keep the system working in the most efficient way. In this paper, we present a pipeline-based dynamic modular arithmetic-based scheduler (PMOD scheduler), which can be used to re-schedule the streams distributed among a set of nodes and their tasks, when the system parameters (number of tasks, executors or nodes) change. The PMOD scheduler organizes all the required operations in a pipeline scheme, thus reducing the overall processing time.

Pipelined Dynamic Scheduling of Big Data Streams

An important as well as challenging task in modern applications is the management and processing with very short delays of large data volumes. It is quite often, that the capabilities of individual machines are exceeded when trying to manage such large data volumes. In this regard, it is important to develop efficient task scheduling algorithms, which reduce the stream processing costs. What makes the situation more difficult is the fact that the applications as well as the processing systems are prone to changes during runtime: processing nodes may be down, temporarily or permanently, more resources may be needed by an application, and so on. Therefore, it is necessary to develop dynamic schedulers, which can effectively deal with these changes during runtime. In this work, we provide a fast and fair task migration policy while maintaining load balancing and low latency times. The experimental results have shown that our scheme offers better load balancing and reduces the overall latency compared to the state of the art strategies, due to the stepwise communication and the pipeline based processing it employs.

More on Pipelined Dynamic Scheduling of Big Data Streams

In a Big data computing, the processing of data requires a large amount of CPU cycles and network bandwidth and disk I/O. Dataflow is a programming model for processing Big data which consists of tasks organized in a graph structure. Scheduling these tasks is one of the key active research areas which mainly aims to place the tasks on available resources. It is essential to effectively schedule the tasks, in a manner that minimizes task completion time and increases utilization of resources. In recent years, researchers have discussed and presented different task scheduling algorithms. In this research study, we have investigated the state-of-art of various task scheduling algorithms, scheduling considerations for batch and streaming processing, and task scheduling algorithms in the wellknown open-source big data platforms. Furthermore, this study proposes a new task scheduling system to alleviate the problems persists in the existing task scheduling for big data.

Task Scheduling in Big Data - Review, Research Challenges, and Prospects

The development of the Human Genome Project provides the important technical guarantee for people's health. The gene sequencing service plays an important role in the disease prevention or treatment. However, faced with the enormous, unstructured and quick updata human genome, the data mining technology in the traditional computer is helpless. In the paper, we introduced a new kind of the cloud platform Spark. Spark has more advantages over Hadoop in the performance and scalability. In addition, it is becoming the preferred even the only cloud computing platform in the integration and diversification of large data processing center, such as the genetic engineering field. This article is an overview for the cloud computing platform Spark in the human genome data mining.

An overview on cloud computing platform spark for Human Genome mining

Traditional Internet of Things (IoT) sensor nodes transmit raw data to centralized servers for analysis and storage. At higher data rates this architecture is no longer feasible. Instead data must be processed at the lowest level possible to include the sensor node itself. The Joule framework enables decentralized data processing by distributing computation into modules which are dynamically linked at run time by a process supervisor. Modules may be collocated on a single machine or distributed across a network. Data is stored where it is processed creating a fully decentralized architecture. Processed data can be queried by centralized servers or delivered directly to the end user. This allows high bandwidth sensors to be incorporated seamlessly into existing IoT deployments without depleting network bandwidth or server resources. This paper presents the Joule framework as well as performance benchmarks and deployment case studies.

Joule: A Real-Time Framework for Decentralized Sensor Networks

To solve two main problems in task scheduling algorithms for real-time stream processing: the lack of consideration for links between tasks and the lack of support for dynamic scenes, this paper presents a new task scheduling approach. First, it concludes common scheduling scenarios on platforms for real-time stream processing. Second, it proposes matrix model of describing the real-time task scheduling problem. Third, it describes the specific scheduling approach and the algorithm set in it, and finally it implements the scheduling approach on SpeedStream, a real-time stream processing platform. Results prove that the approach can not only effectively balance the load of cluster nodes but also reduce communication traffic passing through switches. This approach is suitable for deployment and use in large-scale clusters.

A Task Scheduling Approach for Real-Time Stream Processing

For two main problems of the existing stream processing model: lack of support for iterative computing and tight coupling, the paper proposes a real-time stream processing model based on directed graph with sources and sinks, which supports both general DAG computing by means of directed circle detection and iteration protection as well as iterative feedback stream computing of directed circles, tow-way arcs and ring arcs. On this basis, the paper has implemented a set of flexible and loosely-coupled interfaces, which supports functional programming and runs executable program directly to achieve language independence without performance loss. This computing model is applicable to a wider range with more flexible interfaces and provides an effective approach to complicated stream computing.

A Computing Model for Real-Time Stream Processing

SpeedStream is a universal distributed platform that can handle with massive data flows with the features of low coupling, high availability, low latency and high scalability Focusing on the core technologies of real-time stream computing platform in cloud environment, this paper conducts a series of researches and implementation of the system First of all, aiming at the availability of real-time streaming computing platform, we design a high availability framework based on Zookeeper It ensures fault detection and recovery of process level and node level timely by monitoring heartbreak of each modules and strategy of fault migration Secondly, in order to increase the application types of the platform, by means of directed cycle detection and iteration protection, we design a real-time streaming computing model that based on directed graph with sources and sinks, which can not only satisfy the needs of common DAG computing services, but also support iteration computing services including directed cycle, bidirectional arcs and annular arcs In addition, the platform can realize personalized task scheduling strategy for users by establishing task allocation matrix and optimize task allocation model Finally, in order to solve the many-to-many dynamic load-balancing between tasks, we apply scheduler with status and distributed session table It overcomes the difficulty of maintaining consistency of session without global session table We also testified the convergence of this method The experiment indicates that the throughput and data processing delay of SpeedStream are superior to other alternatives in dealing with the businesses of iteration applications, high traffic fluctuation applications, and high demand of load-balancing applications This platform provides reliable, universal, and real-time solutions to process massive data flows, such as to process the real-time trading data in e-commerce, to analyze sensing flow in internet of things, and monitor traffics of the Internet

SpeedStream: A real-time stream data processing platform in the cloud

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Li Zhao

Author Tools

Chat about Author

Papers

A Task Scheduling Approach for Real-Time Stream Processing

A Computing Model for Real-Time Stream Processing

SpeedStream: A real-time stream data processing platform in the cloud