Modeling Big Data Processing Programs

Question

1. What are the future works in "Modeling big data processing programs" ?

2. What are the contributions in "Modeling big data processing programs" ?

3. What is the function groupBy used to represent the aggregations?

4. What is the use of lambda expressions in the aggregation of a?

Accepted Answer

As future work, the authors intend to study the mapping of their model into the input notation of CPN Tools for model simulation and analysis.. As a further goal, the authors can also envision the use of this specification for code generation in those similar systems.

Accepted Answer

The authors propose a new model for data processing programs.. The authors use Monoid Algebra to model operations over distributed, partitioned datasets and Petri Nets to represent the data/control flow.

Accepted Answer

The authors represent these aggregations with the transformations reduce, which operates on the whole set, and reduceByKey, which operates on values grouped by key.

Accepted Answer

In addition, monoid algebra also supports the use of lambda expressions (λx.e), if-then-else, and the union operation on bags (]).

Accepted Answer

Large scale data processing systems can be classified according to their purpose into general-purpose, SQL, graph processing, and stream processing systems [2].

Accepted Answer

According to the programming model adopted for processing data, general-purpose systems can be based on control flow (like Apache Hadoop) or data flow (like Apache Spark).

Accepted Answer

Dx selects all the elements of Dx that differ from every element of Dy.Aggregation Transformations collapses elements of a dataset into a single element.

Accepted Answer

The analysis focuses the kind of tools and strategies used for formalizing either control/data flows and data processing operations.

Accepted Answer

An edge may either be a pair in D × T , representing an input dataset of a transformation, or a pair in T ×D, representing an output dataset of a transformation.

Accepted Answer

The definition of flatMap in their model is just the monoid algebra operation defined in Section 2:flatMap :: (α→ Bag[β])→ Bag[α]→ Bag[β] flatMap(f,D) = flatmap(f,D)The map transformation derives data of type Bag[β] when given a function f : α → β.

Accepted Answer

In general, the authors define three types of modifications in the data flow: replacement of one transformation with another (both existing in the program); swap the calling order of two transformations, and delete the call of a transformation in the data flow.

Accepted Answer

This can be importantfor comparing infrastructures, for pipelines for testing parallel data processing programs, and eventually verifying programs properties (like correctness, completeness, concurrent access to data).

Accepted Answer

A transformation t in the transformation set T of a program P is characterized by the operation it carries out, the types of its input and output datasets, and its input parameters.

Accepted Answer

To define the DAG that represents the data flow of a data processing program, the authors rely on the data flow graph model presented in [14], which was formalized using Petri Nets [17].

Accepted Answer

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.data.

Accepted Answer

The authors can also identify the application of three transformations in P , thus the set T in their example is defined as T = {t1, t2, t3}, where the transformations in T are t1 = union(d1, d2), t2 = distinct(d3), and t3 = filter((line: String) => !(line.startsWith(“host”) && line.contains(“bytes”)), d4).

Modeling Big Data Processing Programs

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What are the future works in "Modeling big data processing programs" ?

2. What are the contributions in "Modeling big data processing programs" ?

3. What is the function groupBy used to represent the aggregations?

4. What is the use of lambda expressions in the aggregation of a?

5. What is the purpose of large scale data processing systems?

6. What is the programming model used for processing data?

7. What is the function that is used to group elements of a dataset?

8. What is the purpose of the analysis?

9. What is the definition of an edge in the data flow model?

10. What is the definition of flatMap in the model?

11. What are the types of modifications in the data flow?

12. What can be important for comparing infrastructures?

13. What is the definition of a transformation in a program?

14. What is the DAG that represents the data flow of a data processing program?

15. What was the funding for this study?

16. How can the authors identify the application of three transformations in P?

Figures

Citations

TRANSMUT‐Spark: Transformation mutation for Apache Spark

TRANSMUT-SPARK: Transformation Mutation for Apache Spark

An Abstract View of Big Data Processing Programs.

TRANSMUT-SPARK: Transformation Mutation for Apache Spark-Long Version

Open-Source Digital Infrastructure Capacity Prediction System for Production Company

References

Petri nets: Properties, analysis and applications

Spark: cluster computing with working sets

Dryad: distributed data-parallel programs from sequential building blocks

Kommunikation mit Automaten

Apache flink : Stream and batch processing in a single engine

Related Papers (5)

Operational Semantics of a Data Flow Language.

Computer automated multi-paradigm modelling for analysis and design of traffic networks

Analysis of Real-Time Systems by Data Flowgraphs

Control flow graphs as a representation language

Streamlining Control Flow Graph Construction with DCFlow