https://bura.brunel.ac.uk/bitstream/2438/13152/4/Fulltext.pdf

Critical analysis of Big Data challenges and analytical methods

Internet of Things (IoT) related applications have emerged as an important field for both engineers and researchers, reflecting the magnitude and impact of data-related problems to be solved in contemporary business organizations especially in cloud computing. This paper first provides a functional framework that identifies the acquisition, management, processing and mining areas of IoT big data, and several associated technical modules are defined and described in terms of their key characteristics and capabilities. Then current research in IoT application is analyzed, moreover, the challenges and opportunities associated with IoT big data research are identified. We also report a study of critical IoT application publications and research topics based on related academic and industry publications. Finally, some open issues and some typical examples are given under the proposed IoT-related research framework.

IoT-Based Big Data Storage Systems in Cloud Computing: Perspectives and Challenges

Software developers are increasingly required to write big data code. However, they find big data software development challenging. To help these developers it is necessary to understand big data topics that they are interested in and the difficulty of finding answers for questions in these topics. In this work, we conduct a large-scale study on Stackoverflow to understand the interest and difficulties of big data developers. To conduct the study, we develop a set of big data tags to extract big data posts from Stackoverflow; use topic modeling to group these posts into big data topics; group similar topics into categories to construct a topic hierarchy; analyze popularity and difficulty of topics and their correlations; and discuss implications of our findings for practice, research and education of big data software development and investigate their coincidence with the findings of previous work.

/pdf/going-big-a-large-scale-study-on-what-big-data-developers-3cfwe92e09.pdf

Going big: a large-scale study on what big data developers ask

As the growth of Internet, Cloud Computing, Mobile Network and Internet of Things is increasing rapidly, Big Data is becoming a hot-spot in recent years. Big Data Processing is involved in our daily life such as mobile devices, RFID and wireless sensors, which aims at dealing with billions of users' interactive data. At the same time, real-time processing is eagerly needed in integrated system. In this paper, several technologies associated with real-time big data processing are introduced, among which the core technology called Storm is emphasized. An entire system is built based on Storm, associated with RabbitMQ, NoSQL and JSP. To ensure the practical applicability and high efficiency, a simulation system is established and shows acceptable performance in various expressions using data sheet and Ganglia. It is proved that the big data real-time processing based on Storm can be widely used in various computing environment.

Big Data Real-Time Processing Based on Storm

A novel adaptive e-learning model based on Big Data by using competence-based knowledge and social learner activities

HighlightsWe explain in detail the architecture of prefetching module in Section 4.4.We detail the framework of HPSO by example in Section 4.1.We modify the scheduling algorithm based on prefetching to fully exploit the potential map tasks with data locality in Section 4.3.1. This method has the advantages of reducing network transmission. Furthermore, we consider part of nodes, whose remaining time is less then threshold Tunder to avoid invalid data prefetching.We conduct a serial of experiments to evaluate performance of the proposed system using different 5 applications (Section 5).A survey on the state-of-the-art method for improving data locality is conducted in Section 6. Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes for future map tasks based on current pending tasks and then preload the needed data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

/pdf/scheduling-algorithm-based-on-prefetching-in-mapreduce-5e105a0ixy.pdf

Scheduling algorithm based on prefetching in MapReduce clusters

The invention discloses a distributed monitoring system based on a Hadoop cluster and a distributed monitoring method thereof. The system comprises a client, a master computer and a slave computer. The system is characterized in that: the master computer and the slave computer in the system are constructed based on a MapReduce framework of the Hadoop cluster; the client is used for submitting a monitoring work request to the master computer; after responding to the monitoring work request, the master computer divides monitoring work and distributes the divided monitoring work to the salve computer for independent completion; and the slave computer is used for completing an independent monitoring work task, integrating a result through the master computer and returning data to the client. By the invention, data such as the task schedule and the like of a distributed computing network MapReduce can be effectively monitored, the monitoring result can be precisely obtained in real time, and independent dynamic switching is realized under the condition of not influencing work execution.

Distributed monitoring system based on Hadoop cluster and monitoring method thereof

Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes to which future map tasks should be assigned and then preload the input data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

HPSO: Prefetching Based Scheduling to Improve Data Locality for MapReduce Clusters

Comparing with the traditional disk based distributed storage system, RAM based storage has been proven to be an effective way to accelerate real time applications processing speed. In this paper, we propose a memory based distributed Cloud storage system called Sedna. Managing 'big data' across lots of commodity servers, Sedna provides high scalability, simple effective data access APIs with data consistency and persistency, and a new trigger based APIs for real time applications. To guarantee the scalability with low latency, we design and implement a hierarchical structure to manage huge size data center which is simple and effective. Except the high speed provided by memory based storage, Sedna absorbs advantages from state-of-art cloud programming frameworks, and gives programmers a new way to write massive data real time applications. These data read/write triggers APIs are necessary but are missing parts of modern distributed storage system. Experiments and examples show Sedna achieve comparable speed to widely-used distributed cache system, and provide a more efficient way to use distributed storage.

Sedna: A Memory Based Key-Value Storage System for Realtime Processing in Cloud

The invention discloses an active response system based on intrusion detection for Android mobile phones. The active response system and an intrusion detection system of Android (IDSA) share the same system architecture, namely a client side-server side mode, and the system is simple in structure and easy to implement. The client side is responsible for acquiring application program information installed by users from users' mobile phones, primarily estimating the application program according to abnormal behaviors when receiving the abnormal information detected by IDSA, sending the uncertain suspicious program information to the server side, waiting for receiving feedback information after being analyzed by the service side and taking appropriate active response measurements according to the final feedback results. The server side is mainly responsible for receiving request information about reanalyzing the suspicious programs of the client side, performing static analysis to executable files in the information, providing the final analysis results according to data and methods and finally sending the results to the client side.

Active response system based on intrusion detection for Android mobile phones

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Mingming Sun

Author Tools

Chat about Author