A huge amount of data is constantly being produced in the world. Data coming from the IoT, from scientific simulations, or from any other field of the eScience, are accumulated over historical data sets and set up the seed for future Big Data processing, with the final goal to generate added value and discover knowledge. In such computing processes, data are the main resource, however, organizing and managing data during their entire life cycle becomes a complex research topic. As part of this, Data LifeCycle (DLC) models have been proposed to efficiently organize large and complex data sets, from creation to consumption, in any field, and any scale, for an effective data usage and big data exploitation. 2. Several DLC frameworks can be found in the literature, each one defined for specific environments and scenarios. However, we realized that there is no global and comprehensive DLC model to be easily adapted to different scientific areas. For this reason, in this paper we describe the Comprehensive Scenario Agnostic Data LifeCycle (COSA-DLC) model, a DLC model which: i) is proved to be comprehensive as it addresses the 6Vs challenges (namely Value, Volume, Variety, Velocity, Variability and Veracity, and ii), it can be easily adapted to any particular scenario and, therefore, fit the requirements of a specific scientific field. In this paper we also include two use cases to illustrate the ease of the adaptation in different scenarios. We conclude that the comprehensive scenario agnostic DLC model provides several advantages, such as facilitating global data management, organization and integration, easing the adaptation to any kind of scenario, guaranteeing good data quality levels and, therefore, saving design time and efforts for the scientific and industrial communities.

/pdf/towards-a-comprehensive-data-lifecycle-model-for-big-data-37mvzmsfsb.pdf

Towards a comprehensive data lifecycle model for big data environments

A vast amount of data is constantly being produced in the world from several sources, including the IoT at smart cities. Such complex universe of digital data is being set up through these daily accumulated fresh data over other historical repositories. This data can then be used in different forms and then be reused in further processes, hence, drawing the life cycle of data. Conventionally, the smart city resources management are based on cloud models where sensors data are collected to prepare a centralized and rich set of open data. Cloud-based frameworks provide several benefits, consisting of their ubiquity and (almost) unlimited resources capacity. But accessing data from the cloud results in large network traffics, high latencies and are not recommended for real-time solutions. In order to solve this, fog computing has been defined as a promising technology. It uses devices at the edge to perform closer computation and, therefore, reduces network traffic, decreases latencies, while enhancing security levels. In this paper we present the Data Preservation block, as part of our hierarchical fog-to-cloud F2C data management architecture. This model has the advantages of both fog and cloud technologies, as it allows reduced latencies for critical applications while being able to use the high computing capabilities of cloud technology. Furthermore, we have estimated the amount of collected data to be stored in the storage media, from fog to cloud layers, at the Barcelona smart city, and shown the tremendous advantages of such data organization.

Data Preservation through Fog-to-Cloud (F2C) Data Management in Smart Cities

Traditional smart city resources management rely on cloud based solutions to provide a centralized and rich set of open data. The advantages of cloud based frameworks are their ubiquity, (almost) unlimited resources capacity, cost efficiency, as well as elasticity. However, accessing data from the cloud implies large network traffic, high data latencies, and higher security risks. Alternatively, fog computing emerges as a promising technology to absorb these inconveniences. The use of devices at the edge provides closer computing facilities, reduces network traffic and latencies, and improves security. We have defined a new framework for data management in the context of smart city through a global fog to cloud management architecture; in this paper we present the data acquisition block. As a first experiment we estimate the network traffic during data collection, and compare it with a traditional real system. We also show the effectiveness of some basic data aggregation techniques in the model, such as redundant data elimination and data compression.

/pdf/a-novel-architecture-for-efficient-fog-to-cloud-data-ln1z3itspn.pdf

A Novel Architecture for Efficient Fog to Cloud Data Management in Smart Cities

Smart cities are the current technological solutions to handle the challenges and complexity of the growing urban density. Traditionally, smart city resources management rely on cloud based solutions where sensors data are collected to provide a centralized and rich set of open data. The advantages of cloud-based frameworks are their ubiquity, as well as an (almost) unlimited resources capacity. However, accessing data from the cloud implies large network traffic, high latencies usually not appropriate for real-time or critical solutions, as well as higher security risks. Alternatively, fog computing emerges as a promising technology to absorb these inconveniences. It proposes the use of devices at the edge to provide closer computing facilities and, therefore, reducing network traffic, reducing latencies drastically while improving security. We have defined a new framework for data management in the context of a smart city through a global fog to cloud resources management architecture. This model has the advantages of both, fog and cloud technologies, as it allows reduced latencies for critical applications while being able to use the high computing capabilities of cloud technology. In this paper, we present the data acquisition block of our framework and discuss the advantages. As a first experiment, we estimate the network traffic in this model during data collection and compare it with a traditional real system. Keywords—Smart City; Fog-to-Cloud (F2C) computing; Data Management; Data Lifecycle Model (DLC); Data Aggregation

Fog-to-Cloud (F2C) Data Management for Smart Cities

Identifying and appropriately resolving system tradeoffs or effectively evaluating alternatives is a key deliverable of systems engineering. Without proper resolution, system performance is hindered, or suboptimal technologies are chosen. TRIZ, the Theory of Inventive Problem Solving, offers tools and methods to identify and resolve tradeoffs (which it terms contradictions or conflicts). TRIZ recognizes that fundamental performance limits arise when one or more unresolved tradeoffs exist in a system. According to TRIZ, eliminating or reducing the effects of the conflicts is necessary to move to improved system performance. This paper presents a TRIZ Trade Study framework that is useful to identify system conflicts, both across alternatives and within a technology, and at various levels of requirements decomposition, to compare options and optimize how the system performs. The framework was developed to perform a trade study between alternative pharmaceutical production systems, and merges traditional trade study methodologies with classical TRIZ. Specifically, traditional pharmaceutical manufacturing development and manufacturing were traded against an emerging and proposed approach, QbD (Quality by Design). Given TRIZ is scalable and useful to a variety of systems, the application of the TRIZ Trade Study Framework is also broadly applicable across system scale and domains. © 2012 Wiley Periodicals, Inc. Syst Eng © 2012 Wiley Periodicals, Inc.

Using a TRIZ framework for systems engineering trade studies

The widespread need within corporate information systems (IS) divisions to migrate large quantities of data between data stores has spawned a family of commercial products, which are commonly referred to as extract-transform-load (ETL) tools. The focus of this paper is the development of engineering trade studies to be used for ETL tool evaluation. This approach: (1) Identifies selection criteria that are essential in the evaluation of an ETL tool, (2) Develops scenarios that examine each criterion, and (3) Develops quantitative measures useful for evaluating the various aspects of ETL usage. This approach generates replicable evaluation methods that can be used and modified by companies to address their own ETL product needs. With the results generated through such evaluations, companies will be able to make informed decisions and choose the best ETL tool for their purposes.

Engineering trade study: extract, transform, load tools for data migration

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

D. Lee

Author Tools

Chat about Author

Papers

Engineering trade study: extract, transform, load tools for data migration