Near real-time satellite data is very useful in the research and development activities for land, ocean, and atmospheric processes, environmental monitoring and assessment, weather prediction, and so on. The quality, completeness, and timeliness of this satellite data are great concerns to researchers. How to monitor and collect this information presents them with technical challenges. This paper describes the practical solutions to address these challenges in an enterprise level satellite data repository, which acquires, manipulates, and disseminates various types of near real time satellite data in different formats from multiple data providers. The technical implementations are presented in detail. In addition, the approach of collecting and displaying quality information is also discussed.

Near real-time satellite data quality monitoring and control

A wide array of time-sensitive satellite data is required in the research and development activities for natural hazard assessment, storms and weather prediction, hurricane tracking, disaster and emergency response, and so on. Identifying and analyzing the latencies of large volumes of real-time and near real-time satellite data is very useful and helpful for detecting transmission issues, managing IT resources, and configuring and optimizing data management systems. This paper introduces how to monitor and collect important timestamps of data transmissions, organize them in a NoSQL database, and explore data latency via a user-friendly dashboard. Taking Sentinel series satellite data as an example, data transmission issues are illustrated and investigated further. Latency analysis and explorations help data providers and managers improve data transmission and enhance data management.

Latency analysis of large volume satellite data transmissions

STAR Central Data Repository (SCDR) collects a large volume of real-time and near real-time satellite data to improve data management and facilitate research and development activities across the organization. How to track and showcase the usage of this system using formal metrics is still challenging, and is very useful and helpful to better manage data, monitor system status, evaluate program output, and plan future infrastructure. This paper proposes design and implementation of a metrics analysis system to collect metrics information on user data requests, manage and organize request information, and represent metrics information in an intuitive and interactive manner. The system offers detailed information on satellite data product requests from users on various hosts and robust capabilities of generating insight charts and periodical report.

Assessing a Central Satellite Data Repository and its Usage Statistics

High volumes of satellite data management within an organization is still challenging and daunting in the era of big data. The increasing information technology costs and limited budgets, growing satellite data needs, data availability across multiple teams and projects, strategic goals of organization, and expected project outcomes require better satellite data management mechanism and system to facilitate research and development activities. An organization level centralized satellite data repository is a practical solution to satisfy these requirements. This paper describes the best practices and experiences from building such a central satellite data repository within our organization, including data management strategy and policy, scalable and extensible system infrastructure, comprehensive data management system, and technical support and user assistance. These practices can be borrowed and applied in other organizations with similar requirements.

Practices and Experiences in High Volumes of Satellite Data Management

In the big data era, innovative technologies like cloud computing, artificial intelligence, and machine learning are increasingly utilized in the large-scale data management systems of many industry sectors to make them more scalable and intelligent. Applying them to automate and optimize earth observation data management is a hot topic. To improve data quality control mechanisms, a machine learning method in combination with built-in quality rules is presented in this paper to evolve processes around data quality and enhance management of earth observation data. The rules of quality check are set up to detect the common issues, including data completeness, data latency, bad data, and data duplication, and the machine learning model is trained, tested, and deployed to address these quality issues automatically and reduce manual efforts.

A Machine Learning Approach for Data Quality Control of Earth Observation Data Management System

Central Satellite Data Repository Supporting Research and Development

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

J. Brust

Author Tools

Chat about Author

Papers

Central Satellite Data Repository Supporting Research and Development