What is the way to solve this problem?

Solving this problem requires a composable mechanism (e.g., multi-dimensional associative arrays) for creating operations of increasing complexity without increasing their relative effort.

What is the standard approach for handling the increasing size of data?

The standard approach for handling the increasing size of data is to increase the storage capacity at the cost of increasing the time it takes to access any particular data item.

What is the standard approach for handling the increasing diversity of data?

The standard approach for handling the increasing diversity of data is to increase the number of databases/tables at the cost of increasing the effort required to make the data available to users.

What is the standard approach for handling the increasing complexity of operations?

The standard approach for handling the increasing complexity of operations is to increase the size of the functions at the cost of increasing the effort required to build these functions.

What is the key innovation of the LLGrid 11?

MIT Lincoln Laboratory LLGrid 11Multi-Dimensional Associative Arrays• Extends associative arrays to 2D and mixed data types A('alice ','bob ') = 'cited ' !or A('alice ','bob ') = 47.0•

What is the advantage of storing the table and its transpose in the database?

Storing both the table and its logical transpose in the database allows for all rows and columns to be searched efficiently without the need to build specialized indexes.

What are the main areas of the D4M project?

Modern database analysis in the areas of healthcare, internet search, finance, and network security are outgrowing the capabilities of current technologies.

What is the first implementation of multidimensional numeric associative arrays?

D4M is the first implementation of multidimensional numeric associative arrays and the first implementation of composable associative arrays.

What are the primary benefits of making a serial program a parallel program?

The primary benefits of this approach are simplicity (making a serial program a parallel program can be done with just a few lines of code) and performance (distributing arrays across processorsallows each processor to easily work on data that is in its own memory).

What are the challenges of a database?

The increasing size of the data (doubling every year), the increasing diversity of the data (e.g., web pages, documents, audio, images, and video), and the increasing complexity of the operations (e.g., ingestion, scanning, link analysis, and importance scoring) all present tremendous challenges.

Open AccessProceedings Article10.1109/ICASSP.2012.6289129

Dynamic distributed dimensional data model (D4M) database and computation system

Jeremy Kepner, +15 more

- 25 Mar 2012

- pp 5349-5352

168

TL;DR: D4M (Dynamic Distributed Dimensional Data Model) has been developed to provide a mathematically rich interface to tuple stores (and structured query language “SQL” databases) and it is possible to create composable analytics with significantly less effort than using traditional approaches.

Abstract: A crucial element of large web companies is their ability to collect and analyze massive amounts of data. Tuple store databases are a key enabling technology employed by many of these companies (e.g., Google Big Table and Amazon Dynamo). Tuple stores are highly scalable and run on commodity clusters, but lack interfaces to support efficient development of mathematically based analytics. D4M (Dynamic Distributed Dimensional Data Model) has been developed to provide a mathematically rich interface to tuple stores (and structured query language “SQL” databases). D4M allows linear algebra to be readily applied to databases. Using D4M, it is possible to create composable analytics with significantly less effort than using traditional approaches. This work describes the D4M technology and its application and performance.

AI Agents for this Paper

Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps

Most frequently asked questions

1. What is the middle layer of D4M?

The middle layer consists of several parallel computation technologies (e.g., pMatlab [1,2], MatlabMPI [3], and gridMatlab [4]) that allow associative arrays to bedistributed efficiently across a parallel computer.

2. What are the contributions in this paper?

This work describes the D4M technology and its application and performance.

3. What is the top layer of D4M?

The top layer consists of composable associative arrays that provide a one-to-one correspondence between database queries and linear algebra.

4. What are the properties of fuzzy algebra?

In addition, many of the linear algebraic properties of fuzzy algebra can also be directly applied: linear independence [7], strong regularity [8], and uniqueness [9].

Table 4. Technology Comparison. D4M uniquely supports composable multi-dimensional associative arrays on parallel computers and tuple store databases. Key: SQL (System Query Language), MPI (Message Passing Interface), HPF (High Performance Fortran), UPC (Universal Parallel C), VSIPL++ (Vector, Signal and Image Processing Library).

Fig. 1. D4M Matlab prototype architecture. At the top is the user application consisting of a series of query and analysis steps. In the middle is the parallel library that hides the parallel mapping of the operations. On the bottom are the databases (typically tuple stores) running on parallel computing hardware.

Fig. 4. Single process D4M and D4M+AccumuloDB insert rates (top) versus the number of table entries. Curves are shown for standalone D4M (in memory) and D4M connected to the Accumulo database (in storage).

Fig. 5. Parallel D4M and D4M+AccumuloDB insert rates (top) versus number of insert processes on a single node system. Curves are shown for standalone D4M (in memory) and D4M connected to the Accumulo database (in storage).

Citations

Journal Article•10.1016/J.CIE.2016.07.013

Big Data for supply chain management in the service and manufacturing sectors

Ray Y. Zhong, +3 more

- 01 Nov 2016

- Computers & Industrial Engineering

TL;DR: This paper investigates representative Big Data applications from typical services like finance & economics, healthcare, Supply Chain Management (SCM), and manufacturing sector and discusses current movements on the Big Data for SCM in service and manufacturing world-wide including North America, Europe, and Asia Pacific region.

...read moreread less

558

•Journal Article•10.1145/2814710.2814713

The BigDAWG Polystore System

Jennie Duggan, +9 more

- 12 Aug 2015

TL;DR: In this paper, a new view of federated databases is presented to address the growing need for managing information that spans multiple data models. And the authors propose a polystore architecture, which is designed to unify querying over multiple models.

...read moreread less

305

•Proceedings Article•10.1109/HPEC.2016.7761646

Mathematical foundations of the GraphBLAS

Jeremy Kepner, +15 more

- 18 Jun 2016

TL;DR: The GraphBLAS standard as discussed by the authors defines a core set of matrix-based graph operations that can be used to implement a wide class of graph algorithms in a wide range of programming environments.

...read moreread less

230

•Proceedings Article•10.1109/HPEC.2016.7761646

Mathematical Foundations of the GraphBLAS

Jeremy Kepner, +15 more

- 18 Jun 2016

- arXiv: Mathematical Software

TL;DR: This paper provides an introduction to the mathematics of the GraphBLAS, a core set of matrix-based graph operations that can be used to implement a wide class of graph algorithms in a wide range of programming environments.

...read moreread less

204

...

Expand

References

Proceedings Article•10.1109/MSST.2010.5496972

The Hadoop Distributed File System

Konstantin Shvachko, +3 more

- 03 May 2010

TL;DR: The architecture of HDFS is described and experience using HDFS to manage 25 petabytes of enterprise data at Yahoo! is reported on.

...read moreread less

5.8K

Journal Article•10.1145/1365815.1365816

Bigtable: A Distributed Storage System for Structured Data

Fay W. Chang, +8 more

- 01 Jun 2008

- ACM Transactions on Computer Systems

TL;DR: The simple data model provided by Bigtable is described, which gives clients dynamic control over data layout and format, and the design and implementation of Bigtable are described.

...read moreread less

3.5K

•Book

Parallel MATLAB for Multicore and Multinode Computers

Jeremy Kepner

- 23 Jul 2009

TL;DR: Parallel MATLAB for Multicore and Multinode Computers covers more parallel algorithms and parallel programming models than any other parallel programming book due to the succinctness of MATLAB.

...read moreread less

116

•Journal Article•10.1177/1094342007078446

pMatlab Parallel Matlab Library

Nadya T. Bliss, +1 more

- 20 Jun 2006

- arXiv: Astrophysics

TL;DR: pMatlab as discussed by the authors is a distributed numerical array whose distribution onto multiple processors is specified with a map construct, which can support redistribution between any block-cyclic-overlapped distributions up to four dimensions.

...read moreread less

Pmatlab:parallel matlab libraryfor signalprocessingapplications'

Jeremy Kepner, +1 more

- 01 Jan 2007

TL;DR: The pMaltab project as discussed by the authors implements partitioned global address space (PGAS) support for overloading techniques, which simplifies parallel programming by removing the need for decoding message passing.

...read moreread less

Dynamic distributed dimensional data model (D4M) database and computation system

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What is the middle layer of D4M?

2. What are the contributions in this paper?

3. What is the top layer of D4M?

4. What are the properties of fuzzy algebra?

5. What is the way to solve this problem?

6. What is the standard approach for handling the increasing size of data?

7. What is the standard approach for handling the increasing diversity of data?

8. What is the standard approach for handling the increasing complexity of operations?

9. What is the key innovation of the LLGrid 11?

10. What is the advantage of storing the table and its transpose in the database?

11. What are the main areas of the D4M project?

12. What is the simplest way to perform a facet search in a Tuple store?

13. What is the first implementation of multidimensional numeric associative arrays?

14. What are the primary benefits of making a serial program a parallel program?

15. What are the challenges of a database?

Figures

Citations

Big Data for supply chain management in the service and manufacturing sectors

The BigDAWG Polystore System

Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis

Mathematical foundations of the GraphBLAS

Mathematical Foundations of the GraphBLAS

References

The Hadoop Distributed File System

Bigtable: A Distributed Storage System for Structured Data

Parallel MATLAB for Multicore and Multinode Computers

pMatlab Parallel Matlab Library

Pmatlab:parallel matlab libraryfor signalprocessingapplications'

Related Papers (5)

D4M 2.0 schema: A general purpose high performance schema for the Accumulo database

D4M: Bringing associative arrays to database engines

Bigtable: A Distributed Storage System for Structured Data

The Combinatorial BLAS: design, implementation, and applications

Parallel MATLAB for Multicore and Multinode Computers