Computational micromagnetics has become an essential tool in academia and industry to support fundamental research and the design and development of devices. Consequently, computational micromagnetics is widely used in the community, and the fraction of time researchers spend performing computational studies is growing. We focus on reducing this time by improving the interface between the numerical simulation and the researcher. We have designed and developed a human-centred research environment called Ubermag. With Ubermag, scientists can control an existing micromagnetic simulation package, such as OOMMF, from Jupyter notebooks. The complete simulation workflow, including definition, execution, and data analysis of simulation runs, can be performed within the same notebook environment. Numerical libraries, co-developed by the computational and data science community, can immediately be used for micromagnetic data analysis within this Python-based environment. By design, it is possible to extend Ubermag to drive other micromagnetic packages from the same environment.

Ubermag: Towards more effective micromagnetic workflows

More than ninety percent of published Jupyter notebooks do not state dependencies on external packages. This makes them non-executable and thus hinders reproducibility of scientific results. We present SnifferDog, an approach that 1) collects the APIs of Python packages and versions, creating a database of APIs; 2) analyzes notebooks to determine candidates for required packages and versions; and 3) checks which packages are required to make the notebook executable (and ideally, reproduce its stored results). In its evaluation, we show that SnifferDog precisely restores execution environments for the largest majority of notebooks, making them immediately executable for end users.

pdf/restoring-execution-environments-of-jupyter-notebooks-388807ergw.pdf

Restoring Execution Environments of Jupyter Notebooks

Abstract. With the increase in computational power, ocean models with kilometer-scale resolution have emerged over the last decade. These models have been used for quantifying the energetic exchanges between spatial scales, informing the design of eddy parametrizations, and preparing observing networks. The increase in resolution, however, has drastically increased the size of model outputs, making it difficult to transfer and analyze the data. It remains, nonetheless, of primary importance to assess more systematically the realism of these models.
Here, we showcase a cloud-based analysis framework proposed by the Pangeo project that aims to tackle such distribution and analysis challenges. We analyze the output of eight submesoscale-permitting simulations, all on the cloud, for a crossover region of the upcoming Surface Water and Ocean Topography (SWOT) altimeter mission near the Gulf Stream separation.
The cloud-based analysis framework (i) minimizes the cost of duplicating and storing ghost copies of data and (ii) allows for seamless sharing of analysis results amongst collaborators. We describe the framework and provide example analyses (e.g., sea-surface height variability, submesoscale vertical buoyancy fluxes, and comparison to predictions from the mixed-layer instability parametrization). Basin- to global-scale, submesoscale-permitting models are still at their early stage of development; their cost and carbon footprints are also rather large. It would, therefore, benefit the community to document the different model configurations for future best practices. We also argue that an emphasis on data analysis strategies would be crucial for improving the models themselves.


Cloud-based framework for inter-comparing submesoscale-permitting realistic ocean models

Software development for data reduction and analysis at large research facilities is increasingly professionalized, and internationally coordinated. To foster software quality and sustainability, and to facilitate collaboration, representatives from software groups of European neutron and muon facilities have agreed on a set of guidelines for development practices, infrastructure, and functional and non-functional product properties. These guidelines have been derived from actual practices in software projects from the EU funded consortium ‘Science and Innovation with Neutrons in Europe in 2020’ (SINE2020), and have been enriched through extensive literature review. Besides guiding the work of the professional software engineers in our computing groups, we hope to influence scientists who are willing to contribute their own data treatment software to our community. Moreover, this work may also provide inspiration to scientific software development beyond the neutron and muon field.

Guidelines for collaborative development of sustainable data treatment software

The XrayFeed device of Karabo [1, 2] is designed to provide spatial X-ray beam stability in terms of drift compensation utilizing different diagnostic components at the European XFEL (EuXFEL). Our feedback systems proved to be indispensable in cutting-edge pump-probe experiments at EuXFEL. The feedback mechanism is based on a closed loop PID control algorithm [3] to steer the beam position measured by the so-called diagnostic devices to the desired centred position via defined actuator adjusting the alignment of Xray optical elements, in our case a flat X-ray mirror system. Several diagnostic devices and actuators can be selected according to the specific experimental area where a beam position feedback is needed. In this contribution, we analyze the improvement of pointing stability of X-rays using different diagnostic devices as an input source for our feedback system. Different types of photon diagnostic devices such as gas-based X-ray monitors [4], quadrant detectors based on avalanche photo diodes [5] and optical cameras imaging the X-ray footprint on scintillator screens have been evaluated in our pointing stability studies.

Beam Position Feedback System supported by Karabo at European XFEL

Jupyter notebooks are executable documents that are displayed in a web browser. The notebook elements consist of human-authored contextual elements and computer code, and computer-generated output from executing the computer code. Such outputs can include tables and plots. The notebook elements can be executed interactively, and the whole notebook can be saved, re-loaded and re-executed, or converted to read-only formats such as HTML, LaTeX and PDF. Exploiting these characteristics, Jupyter notebooks can be used to improve the effectiveness of computational and data exploration, documentation, communication, reproducibility and re-usability of scientific research results. They also serve as building blocks of remote data access and analysis as is required for facilities hosting large data sets and initiatives such as the European Open Science Cloud (EOSC). In this contribution we report from our experience of using Jupyter notebooks for data analysis at research facilities, and outline opportunities and future plans.

/pdf/data-exploration-and-analysis-with-jupyter-notebooks-5fmd07metf.pdf

Data Exploration and Analysis with Jupyter Notebooks

To achieve a tight integration of instrument control and (online) data analysis, the European XFEL decided in 2011 to develop Karabo, a custom control and data processing system. Karabo provides control via event-driven communication. Signal/slot and request/reply patterns are implemented via a central message broker. Data pipelines for e.g. scientific workflows or detector calibration are implemented as direct TCP/IP connections. The core elements of Karabo are self-describing devices written in C++ or Python. They represent hardware, orchestrate other devices, or provide system services like data logging and configuration storage. To operate Karabo, a Python command line interface and a generic GUI written in PyQt are provided. Control and data widgets compose Karabo scenes that are provided by devices or are manually customized and stored together with device configurations in a central database. Since 2016, Karabo is used to commission and operate the currently three photon beam lines and six scientific instruments at the European XFEL. This contribution summarizes the status of Karabo, highlights achievements and lessons learned, and gives an outlook for future directions.

Status of the Karabo Control and Data Processing Framework

Agile Project Management (Agile PM), coupled with the DevOps concept, has been worked out as a fundamental approach in a highly uncertain and unpredictable environment to achieve mature software development and to efficiently support concurrent operation [1]. At the European XFEL[2], Agile PM and DevOps have been applied to provide adaptability and efficiency in the development and operation of its control system: Karabo[3,4]. In this context, the Control and Analysis Software Group (CAS) has developed in-house a management platform composed of the following macro-artefacts: (1) Agile Process; (2) Release Planning; (3) Testing Infrastructure; (4) Roll-out and Deployment Strategy; (5) Automated tools for Monitoring Control Points (i.e. Configuration Items[5]) and; (6) Incident Management[6]. The software engineering management platform is also integrated with User Relationship Management to establish and maintain a proper feedback loop with our scientists who set up the requirements. This article aims to briefly describe the above points and show how agile project management has guided the software strategy, development and operation of the Karabo control system at the European XFEL.

Control and Analysis Software Development at the European XFEL

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

D.B. Rück

Author Tools

Chat about Author

Papers

Data Exploration and Analysis with Jupyter Notebooks

Status of the Karabo Control and Data Processing Framework

Control and Analysis Software Development at the European XFEL