The focus during the last year was on the continuation of software development for in situ visual analysis of simulation results, including a redesign of the in situ pipeline that connects simulation and analysis tools, and on the implementation of a backend that enables visualisation and analysis tools to run on the new Fenix infrastructure. The back end has been successfully deployed at CINECA in a close co-development effort with Michele Migliore (CNR) and his group. The visualisation software catalogue has been released through the Knowledge Graph of the HBP and additionally includes metadata on general software relevant for HBP and EBRAINS. The latter has been developed in close collaboration with the Knowledge Graph, Collaboratory, and data curation teams.
In situ Pipeline
The connection of the visualisation tools with the in situ pipeline enables a direct view into a running simulation and, for instance, shortening the time to detect errors. It is the first step towards interactive simulator steering.
A middleware has been developed that can read data coming from running simulations through the in situ pipeline. This middleware has been integrated into the ViSimpl tool. The development has been carried out in close collaboration with the developers of the in situ pipeline to have a fast feedback loop for a quick integration as well as to deal with spikes in compartment-based simulations (see Figure below).
Restructured setup of the in situ pipeline
This figure shows the restructured setup of the in situ pipeline. A client connects to a single access point of the pipeline using standardised HTTP requests. The requested data is gathered from simulator-specific modules integrated into the simulation engine running on multiple compute nodes.
The in situ pipeline underwent a major rework in the second year of SGA2 to reduce the integration effort into visualisation tools. The rework was done in close collaboration with the users from URJC / UPM and included a 3-month research stay of Simon Oehrl at the URJC Madrid. In addition, Sebastian Spreizer, developer of NEST Desktop was included in the design decisions for the pipeline. All the necessary software is available inside Docker containers for easy deployment. Computing resources on the CSCS cluster have been granted for intensive testing of the pipeline.
| Component | Content | URL |
| ViSimpl | Software Repository | https://github.com/gmrvvis/visimpl |
| Technical Documentation | https://github.com/gmrvvis/visimpl | |
| User Documentation | https://github.com/gmrvvis/visimpl | |
| In Situ Pipeline | Software Repository | https://devhub.vr.rwth-aachen.de/VR-Group/in-situ-pipeline |
| Technical Documentation | https://devhub.vr.rwth-aachen.de/VR-Group/in-situ- pipeline/insite/tree/develop | |
| User Documentation | https://devhub.vr.rwth-aachen.de/VR-Group/in-situ- pipeline/insite/-/wikis/api/api_reference |
Deployment and extension of visualisation tools at CINECA
In collaboration with Michele Migliore’s group, two of the visualisation tools (ViSimpl and NeuroTeshMess) were deployed on GALILEO, a supercomputer at CINECA (see SGA2 Deliverable D7.6.1 for details). The evaluation of the difficulties encountered, as well as the lessons learned during this experience, led this work towards the creation of Docker containers that will make it easy to deploy these tools on any supercomputer, in EBRAINS and also outside of the HBP. The team of Colin McMurtrie at CSCS is currently testing these containers.
Release of a prototype for remote visualisation and interaction
The developed remote rendering middleware allows running any application that has been built on the machines where the middleware is deployed (see SGA2 Deliverable D7.6.1 for details). The entry point for the backend can be a web page or a Jupyter Notebook that allows users to create and manage sessions. The middleware is fully usable from the HBP Collaboratory and also allows to retrieve data from the Collaboratory and make it available in the machine where the backend is running. The connection with the archival database repository using swiftfs could, due to the Covid-19 crisis, not be finished during SGA2 and is therefore still work in progress.
Software repository, technical and user documentation: https://github.com/gmrvvis/ReMo
Visualisation software catalogue design and first version
As outlined in SGA2 Deliverable D7.6.1, the identified shift from a standalone, newly developed semantic search engine towards adding searchable software metadata into the EBRAINS Knowledge Graph (KG) has been finalised. In collaboration with Andrew Davison and the Data Curation Team, partner UT was able to develop a software metadata schema based on the previous investigation and developments that were conducted in the first half of SGA2. All details on the metadata schema can be found in the catalogue’s documentation on GitHub and in SGA2 Deliverable D7.4.1. With the help of UT, the schema has been implemented by the KG development team (led by Oliver Schmid) in the KG as well as in the KG editor. The release of the first version of the software metadata card took place together with the release of the new front end at the start of the HBP Summit in February 2020 in Athens, Greece.
In close collaboration (weekly data curation videoconferences, in which UT started to participate) and with support by the Data Curation Team, UT established curation processes for the step-wise research and integration of software metadata into the Knowledge Graph from various sources. All processes are documented in the GitHub repository and described in detail in SGA2 Deliverable D7.4.1. The curation processes are transparently documented in the GitHub repository’s issue tracker (https://github.com/bweyers/HBPVisCatalogue/issues, 15 fully integrated into Knowledge Graph and released, 40 data sets in curation, status of 2020-02-11). UT has started to integrate software metadata from the Collaboratory software catalogue, and of this HPAC Guidebook following suggestions from the project. However, the catalogue will not only consider software developed within the HBP but also list software relevant for neuroscience and/or provided on EBRAINS resources.
For broader coordination and face-to-face discussions, UT organised together with Andrew Davison a meeting in Heidelberg during the CodeJam#10. During this meeting, a subset of the PLUS, Collaboratory, KG, and Data Curation Team members were present to discuss overlaps and potential integration. In addition, UT participated in the data curation helpdesk during the HBP Summit to advertise the software metadata curation to the Project Consortium.
Curation Repository: https://github.com/bweyers/HBPVisCatalogue/issues
Technical and user documentation: https://github.com/bweyers/HBPVisCatalogue/wiki