Hadoop and Apache Spark have become popular frameworks for distributed big data processing. This research aims to configure Hadoop and Spark for conducting training and testing on big data using distributed machine learning methods with MLlib, including linear regression and multi-linear regression. Additionally, an external library, LSTM, is used for experimentation. The experiments utilize three desktop devices to represent a series of tests on single and multi-node networks. Three datasets, namely bitcoin (3,613,767 rows), gold-price (5,585 rows), and housing-price (23,613 rows), are employed as case studies. The distributed computation tests are conducted by allocating uniform core processors on all three devices and measuring execution times, as well as RMSE and MAPE values. The results of the single-node tests using MLlib (both linear and multi-linear regression) with variations of core utilization ranging from 2 to 16 cores, show that the overall dataset performs optimally using 12 cores, with an execution time of 532.328 seconds. However, in the LSTM method, core allocation variations do not yield significant results and require longer program execution times. On the other hand, in the multinode (2) tests, optimal performance is achieved using 8 cores, with an execution time of 924.711 seconds, while in the multi-node (3) tests, the ideal configuration is 6 cores with an execution time of 881.495 seconds. In conclusion, without the involvement of HDFS, distributed MLlib programs cannot be processed, and core allocation depends on the number of nodes used and the size of the dataset.

https://www.e3s-conferences.org/articles/e3sconf/pdf/2023/102/e3sconf_icimece2023_02058.pdf

Distributed Machine Learning using HDFS and Apache Spark for Big Data Challenges

Apache Spark is a popular distributed computing platform at present for its excellent capacity in big data computing, and the large-scale parallel execution in Apache Spark relies on the divide-and-conquer strategy. However, the irregular algorithms cannot be divided into small-scale problems and processed parallelly on Apache Spark, because the complex dependencies inside the input data stop the input from being divided. To remedy this, based on Thread-Level Speculation technique, this paper proposes a speculative parallel execution model for Apache Spark. With the proposed model, the complex dependencies inside the input data is conquered by a speculative strategy, thus the input could be divided into small chunks. Accordingly, the irregular algorithms could also be partitioned into a series of small-scale tasks and be executed in parallel. After parallel execution, the proposed model raises an evaluation method to eliminate incorrect speculations and ensure the correctness of the final output. At last, to verify the practicability and scalability of the proposed model, the intrusion prevention system (IPS) is implemented on the proposed model, and the achievement is encouraging. Experiments show that through the proposed model, the execution time for big data intrusion prevention could be markedly reduced. All in all, by adopting our novel model, the efficiency of the irregular algorithms on Apache Spark can be enhanced significantly.

A Speculative Parallel Execution Model for Apache Spark

Popular Tasks used by Researchers

How can I help with your research?

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

We encountered an error while loading the panel. Please try refreshing the page.

Something went wrong

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

Anyone with the link can view this chat. You can switch it back to private anytime.

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Zhoukai Wang

Author Tools

Chat about Author

Papers

A Speculative Parallel Execution Model for Apache Spark