The Virtual Brain (TVB) is a large-scale brain simulator. With a community of thousands of users around the world, TVB has become a validated, popular and standard choice for the simulation of whole brain activity. TVB users can create simulations using neural mass models which can produce outputs for different analysis and modalities. TVB allows scientists to explore and analyze both simulated and experimental data. It contains analytic tools for evaluating relevant scientific parameters in light of that data. The current implementation of TVB is written in Python, with limited large-scale parallelization over different parameters. The objective of TVB-HPC is to enable large-scale parallelization of TVB simulations by making use of high performance computing to explore large parameter spaces for the models. With this approach, neuroscientists can define their models in a domain specific language based on NeuroML and automatically generate code which can run either on GPUs or on CPUs with different architectures and optimizations. The result is a framework that hides the complexity of writing robust parallel code and offers neuroscientists a fast and efficient access to high performance computing. TVB-HPC is publicly available on GitHub and, at the end of HBP project phase SGA2, it will be possible to launch large parameter simulations using code automatically generated with this framework via the HBP Collaboratory.
The in situ pipeline consists of a set of libraries that can be integrated into neuronal network simulators developed by the HBP to enable live visual analysis during the runtime of the simulation. The library called ‘nesci’ (neuronal simulator conduit interface) stores the raw simulation data into a common conduit format and the library called ‘contra’ (conduit transport) transports the serialized data from one endpoint to another using a variety of different (network) protocols. The pipeline currently works with NEST and Arbor. Support for TVB is currently in development.
Prototypical implementation into the HPAC Platform finalised in February 2019.
Date of release
First released in July 2018 with continuous updates (see also above)
Science has driven the development of the NEST simulator for the past 20 years. Originally created to simulate the propagation of synfire chains using single-processor workstations, we have pushed NEST’s capabilities continuously to address new scientific questions and computer architectures. Prominent examples include studies on spike-timing dependent plasticity in large simulations of cortical networks, the verification of mean-field models, models of Alzheimer’s and Parkinson’s disease and tinnitus. Recent developments include a significant reduction in memory requirements, as demonstrated by a record-breaking simulation of 1.86 billion neurons connected by 11.1 trillion synapses on the Japanese K supercomputer, paving the way for brain-scale simulations.
Running on everything from laptops to the world’s largest supercomputers, NEST is configured and controlled by high-level Python scripts, while harnessing the power of C++ under the hood. An extensive testsuite and systematic quality assurance ensure the reliability of NEST.
The development of NEST is driven by the demands of neuroscience and carried out in a collaborative fashion at many institutions around the world, coordinated by the non-profit member-based NEST Initiative. NEST is released under GNU Public License version 2 or later.
How NEST has been improved in HBP
Continuous dynamics
The continuous dynamics code in NEST enables simulations of rate- based model neurons in the event-based simulation scheme of the spiking simulator NEST. The technology was included and released with NEST 2.14.0.
Furthermore, additional rate-based models for the Co-Design Project “Visuo-Motor Integration” (CDP4) have been implemented and scheduled for NEST release 2.16.0.
NESTML is a domain-specific language that supports the specification of neuron models in a precise and concise syntax, based on the syntax of Python. Model equations can either be given as a simple string of mathematical notation or as an algorithm written in the built-in procedural language. The equations are analyzed by NESTML to compute an exact solution if possible, or use an appropriate numeric solver otherwise.
This technology couples the simulation software NEST and UG4 by means of the MUSIC library. NEST can only send spike trains where spiking occurs; UG4 receives those in form of events arriving at synapses (timestamps). The time course of the extracellular potential in a cube (representing a piece of tissue) is simulated based on the arriving spike data.The evolution of the membrane potential in space and time is described by the Xylouris-Wittum model.
Link to this release (2017): https://github.com/UG4
The development of Monsteer was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
Monsteer is a library for Interactive Supercomputing in the neuroscience domain. Monsteer facilitates the coupling of running simulations (currently NEST) with interactive visualization and analysis applications. Monsteer supports streaming of simulation data to clients (currenty only spikes) as well as control of the simulator from the clients (also kown as computational steering). Monsteer’s main components are a C++ library, an MUSIC-based application and Python helpers.
Minimum configuration to configure using cmake, compile and run Monsteer:
A Linux box,
GCC compiler 4.8+,
CMake 2.8+,
Boost 1.54,
MPI (OpenMPI, mvapich2, etc),
NEST simulator 2.4.2,
MUSIC 1.0.7,
Python 2.6,
See also: http://bluebrain.github.io/Monsteer-0.3/_user__guide.html#Compilation
MSPViz is a visualization tool for the Model of Structural Plasticity. It uses a visualisation technique based on the representation of the neuronal information through the use of abstract levels and a set of schematic representations into each level. The multilevel structure and the design of the representations constitutes an approach that provides organized views that facilitates visual analysis tasks.
Each view has been enhanced adding line and bar charts to analyse trends in simulation data. Filtering and sorting capabilities can be applied on each view to ease the analysis. Other views, such as connectivity matrices and force-directed layouts, have been incorporated, enriching the already existing views and improving the analysis process. This tool has been optimised to lower render and data loading times, even from remote sources such as WebDav servers.
Screenshot of MSPVizScreenshot of MSPVizView of MSPViz to investigate structural plasticity models on different levels of abstraction: connectivity of a single neuronView of MSPViz to investigate structural plasticity models on different levels of abstraction: full network connectivity
The development of VIOLA was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
VIOLA (VIsualizer Of Layer Activity) is a tool to visualize activity in multiple 2D layers in an interactive and efficient way. It gives an insight into spatially resolved time series such as simulation results of neural networks with 2D geometry. The usage example shows how VIOLA can be used to visualize spike data from a NEST simulation (http://nest-simulator.org/) of an excitatory and an inhibitory neuron population with distance-dependent connectivity.
The development of neuroFiReS was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
neuroFiReS is a library for performing search and filtering operations using both data contents and metadata. These search operations will be tightly coupled with visualization in order to improve insight gaining from complex data. A first prototype (named spineRet) for searching and filtering over segmented spine data has been developed.
NeuroScheme is a tool that allows users to navigate through circuit data at different levels of abstraction using schematic representations for a fast and precise interpretation of data. It also allows filtering, sorting and selections at the different levels of abstraction. Finally it can be coupled with realistic visualization or other applications using the ZeroEQ event library developed in WP 7.3.
This application allows analyses based on a side-by-side comparison using its multi-panel views, and it also provides focus-and-context. Its different layouts enable arranging data in different ways: grid, 3D, camera-based, scatterplot-based or circular. It provides editing capabilities, to create a scene from scratch or to modify an existing one.
ViSimpl, part of the NeuroScheme framework, is a prototype developed to analyse simulation data, using both abstract and schematic visualisations. This analysis can be done visually from temporal, spatial and structural perspectives, with the additional capability of exploring the correlations between input patterns and produced activity.
NeuroScheme screenshotNeuroScheme screenshotNeuroScheme screenshotNeuroScheme screenshotNeuroScheme screenshotOverview of various neuronsUser interface of ViSimpl visualising activity data emerging from a simulation of a neural network model
Required: Qt4, nsol
Optional: Brion/BBPSDK (to access BBP data), ZeroEQ (to couple with other software)
Supported OS: Windows 7, Windows 8.1, Linux (tested on Ubuntu 14.04) and Mac OSX
NeuroLOTs is a set of tools and libraries that allow creating neuronal meshes from a minimal skeletal description. It generates soma meshes using FEM deformation and allows to interactively adapt the tessellation level using different criteria (user-defined, camera distance, etc.)
NeuroTessMesh provides a visual environment for the generation of 3D polygonal meshes that approximate the membrane of neuronal cells, starting from the morphological tracings that describe neuronal morphologies. The 3D models can be tessellated at different levels of detail, providing either a homogeneous or an adaptive resolution of the model. The soma shape is recovered from the incomplete information of the tracings, applying a physical deformation model that can be interactively adjusted. The adaptive refinement process performed in the GPU generates meshes, that allow good visual quality geometries at an affordable computational cost, both in terms of memory and rendering time. NeuroTessMesh is the front-end GUI to the NeuroLOTs framework.
The development of DLB was co-funded by the HBP during the second project phase (SGA1). This page is kept for reference but will no longer be updated.
DLB is a library devoted to speedup hybrid parallel applications. And at the same time DLB improves the efficient use of the computational resources inside a computing node. The DLB library will improve the load balance of the outer level of parallelism by redistributing the computational resources at the inner level of parallelism. This readjustment of resources will be done at dynamically at runtime. This dynamism allows DLB to react to different sources of imbalance: Algorithm, data, hardware architecture and resource availability among others.
The first version that was integrated in the HPAC Platform was v1.1.
Used on MareNostrum IV supercomputer for some applications
The development of MonetDB was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
When a database grows into millions of records spread over many tables and business intelligence or science becomes the prevalent application domain, a column-store database management system (DBMS) is called for. Unlike traditional row-stores, such as MySQL and PostgreSQL, a column-store provides a modern and scalable solution without calling for substantial hardware investments.
MonetDB pioneered column-store solutions for high-performance data warehouses for business intelligence and eScience since 1993. It achieves its goal by innovations at all layers of a DBMS, e.g. a storage model based on vertical fragmentation, modern CPU-tuned query execution architecture, automatic and adaptive indices, run-time query optimization, and a modular software architecture. It is based on the SQL 2003 standard with full support of foreign keys, joins, views, triggers, and stored procedures. It is fully ACID compliant and supports a rich spectrum of programming interfaces (JDBC, ODBC, PHP, Python, RoR, C/C++, Perl).
The current version provides the following new features as compared to the version that was part of the HBP-internal Platform Release in M18:
The development of PyCOMPSs was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated, apart from release notes.
PyCOMPSs is the Python binding of COMPSs, (COMP Superscalar) a coarse-grained programming model oriented to distributed environments, with a powerful runtime that leverages low-level APIs (e.g. Amazon EC2) and manages data dependencies (objects and files). From a sequential Python code, it is able to run in parallel and distributed.
COMPSs screenshot
Releases
PyCOMPSs is based on COMPSs. COMPSs version 1.3 was released in November 2015, version 1.4 in May 2016 and version 2.0 in November 2016.
New features in COMPSs v1.3
Runtime
Persistent workers: workers can be deployed on computing nodes and persist during all the application lifetime, thus reducing the runtime overhead. The previous implementation of workers based on a per task process is still supported.
Enhanced logging system
Interoperable communication layer: different inter-nodes communication protocol is supported by implementing the Adaptor interface (JavaGAT and NIO implementations already included)
Simplified cloud connectors interface
JClouds connector
Python/PyCOMPSs
Added constraints support
Enhanced methods support
Lists accepted as a tasks’ parameter type
Support for user decorators
Tools
New monitoring tool: with new views, as workload and possibility of visualizing information about previous runs
Enhanced tracing mechanism
Simplified execution scripts
Simplified installation on supercomputers through better scripts
New features in COMPSs v1.4
Runtime
Added support for Docker
Added support for Chameleon Cloud
Object cache for persistent workers
Improved error management
Added connector for submitting tasks to MN supercomputer from external COMPSs applications
Bug-fixes
Python/PyCOMPSs
General bug-fixes
Tools
Enhanced Tracing mechanism:
Reduced overhead using native Java API
Added support for communications instrumentation added
Added support for PAPI hardware counters
Known Limitations
When executing Python applications with constraints in the cloud the initial VMs must be set to 0
New features in COMPSs v2.0 (released November 2016)
Runtime:
Upgrade to Java 8
Support to remote input files (input files already at workers)
Integration with Persistent Objects
Elasticity with Docker and Mesos
Multi-processor support (CPUs, GPUs, FPGAs)
Dynamic constraints with environment variables
Scheduling taking into account the full tasks graph (not only ready tasks)
Support for SLURM clusters
Initial COMPSs/OmpSs integration
Replicated tasks: Tasks executed in all the workers
Explicit Barrier
Python:
Python user events and HW counters tracing
Improved PyCOMPSs serialization. Added support for lambda and generator parameters.
C:
Constraints support
Tools:
Improved current graph visualization on COMPSs Monitor
Improvements:
Simplified Resource and Project files (NO retrocompatibility)
Improved binding workers execution (use pipes instead of Java Process Builders)
Simplifies cluster job scripts and supercomputers configuration
Several bug fixes
Known Limitations:
When executing python applications with constraints in the cloud the initial VMs must be set to 0
New features in PyCOMPSs/COMPSs v2.1 (released June 2017)
New features:
Runtime:
New annotations to simplify tasks that call external binaries
Integration with other programming models (MPI, OmpSs,..)
Support for Singularity containers in Clusters
Extension of the scheduling to support multi-node tasks (MPI apps as tasks)
Support for Grid Engine job scheduler in clusters
Language flag automatically inferred in runcompss script
New schedulers based on tasks’ generation order
Core affinity and over-subscribing thread management in multi-core cluster queue scripts (used with MKL libraries, for example)
Python:
@local annotation to support simpler data synchronizations in master (requires to install guppy)
Support for args and kwargs parameters as task dependencies
Task versioning support in Python (multiple behaviors of the same task)
New Python persistent workers that reduce overhead of Python tasks
Support for task-thread affinity
Tracing extended to support for Python user events and HW counters (with known issues)
C:
Extension of file management API (compss_fopen, compss_ifstream, compss_ofstream, compss_delete_file)
Support for task-thread affinity
Tools:
Visualization of not-running tasks in current graph of the COMPSs Monitor
Improvements
Improved PyCOMPSs serialization
Improvements in cluster job scripts and supercomputers configuration
Several bug fixes
Known Limitations
When executing Python applications with constraints in the cloud the <InitialVMs> property must be set to 0
Tasks that invoke Numpy and MKL may experience issues if tasks use a different number of MKL threads. This is due to the fact that MKL reuses threads in the different calls and it does not change the number of threads from one call to another.
New features in PyCOMPSs/COMPSs v2.3 (released June 2018)
Runtime
Persistent storage API implementation based on Redis (distributed as default implementation with COMPSs)
Support for FPGA constraints and reconfiguration scripts
Support for PBS Job Scheduler and the Archer Supercomputer
Java
New API call to delete objects in order to reduce application memory usage
Python
Support for Python 3
Support for Python virtual environments (venv)
Support for running PyCOMPSs as a Python module
Support for tasks returning multiple elements (returns=#)
Automatic import of dummy PyCOMPSs AP
C
Persistent worker with Memory-to-memory transfers
Support for arrays (no serialization required)
Improvements
Distribution with docker images
Source Code and example applications distribution on Github
Automatic inference of task return
Improved obsolete object cleanup
Improved tracing support for applications using persistent memory
Improved finalization process to reduce zombie processes
Several bug fixes
Known limitations
Tasks that invoke Numpy and MKL may experience issues if a different MKL threads count is used in different tasks. This is due to the fact that MKL reuses threads in the different calls and it does not change the number of threads from one call to another.
New features in PyCOMPSs/COMPSs v2.5 (released June 2019)
Runtime:
New Concurrent direction type for task parameter.
Multi-node tasks support for native (Java, Python) tasks. Previously, multi-node tasks were only posible with @mpi or @decaf tasks.
@Compss decorator for executing compss applications as tasks.
New runtime api to synchronize files without opening them.
Customizable task failure management with the “onFailure” task property.
Enabled master node to execute tasks.
Python:
Partial support of numba in tasks.
Support for collection as task parameter.
Supported task inheritance.
New persistent MPI worker mode (alternative to subprocess).
Support to ARM MAP and DDT tools (with MPI worker mode).
C:
Support for task without parameters and applications without src folder.
Improvements:
New task property “targetDirection” to indicate direction of the target object in object methods. Substitutes the “isModifier” task property.
Warnings for deprecated or incorrect task parameters.
Improvements in Jupyter for Supercomputers.
Upgrade of runcompss_docker script to docker stack interface.
Several bug fixes.
Known Limitations:
Tasks that invoke Numpy and MKL may experience issues if a different MKL threads count is used in different tasks. This is due to the fact that MKL reuses threads in the different calls and it does not change the number of threads from one call to another.
C++ Objects declared as arguments in a coarse-grain tasks must be passed in the task methods as object pointers in order to have a proper dependency management.
Master as worker is not working for executions with persistent worker in C++.
Coherence and concurrent writing in parameters annotated with the “Concurrent” direction must be managed by the underlaying distributed storage system.
Delete file calls for files used as input can produce a significant synchronization of the main code.
PyCOMPSs/COMPSs PIP installation package
This is a new feature available since January 2017.
Installation:
Check the dependencies in the PIP section of the PyCOMPSs installation manual (available at the documentation section of compss.bsc.es). Be sure that the target machine satisfies the mentioned dependencies.
The installation can be done in various alternative ways:
Use PIP to install the official PyCOMPSs version from the pypi live repository:
sudo -E python2.7 -m pip install pycompss -v
Use PIP to install PyCOMPSs from a pycompss.tar.gz
The development of OmpSs was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
OmpSs is a fine-grained programming model oriented to shared memory environments, with a powerful runtime that leverages low-level APIs (e.g. CUDA/OpenCL) and manages data dependencies (memory regions). It exploits task level parallelism and supports asynchronicity, heterogeneity and data movement.
The new version 15.06 provides the following new features as compared to version 15.04 that was part of the HBP-internal Platform Release in M18:
Socket aware (scheduling taking into account processor socket)
Reductions (mechanism to accumulate results of tasks more efficiently)
Work sharing (persistence of data in the worker) mechanisms
The development of Deflect Client Library was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
Deflect is a C++ library to develop applications that can send and receive pixel streams from other Deflect-based applications, for example DisplayCluster. The following applications are provided which make use of the streaming API:
DesktopStreamer: A small utility that allows the user to stream the desktop.
SimpleStreamer: A simple example to demonstrate streaming of an OpenGL application.
The development of SCOUT was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
SCOUT is a structure-aware method for prefetching data along interactive spatial query sequences. Given the user input, which is a spatial range query sequence representing the structure explored (interactively) by the user, and the spatial dataset to be queried, SCOUT reduces the query response time by prefetching the data along the query sequence.
Similarly to FLAT, both the query ranges in the query sequence and the spatial objects should be represented using a minimum bounding rectangle.
SCOUT outperforms the related prefetching techniques (e.g., Straight Line Extrapolation or Hilbert prefetching) with high prefetching accuracy, which is translated to one order of magnitude speedup.
Date of release
March 2015
Version of software
1.0
Version of documentation
1.0
Software available
Collaboratory, integrated in and part of BBP SDK tool set
The development of RTNeuron in the HPAC Platform was co-funded by the HBP during the second project phase (SGA1). This page is kept for reference but will no longer be updated.
RTNeuron is a scalable real-time rendering tool for the visualisation of neuronal simulations based on cable models. Its main utility is twofold: the interactive visual inspection of structural and functional features of the cortical column model and the generation of high quality movies and images for presentations and publications. The package provides three main components:
A high level C++ library.
A Python module that wraps the C++ library and provides additional tools.
The Python application script rtneuron-app.py
A wide variety of scenarios is covered by rtneuron-app.py. In case the user needs a finer control of the rendering, such as in movie production or to speed up the exploration of different data sets, the Python wrapping is the way to go. The Python wrapping can be used through an IPython shell started directly from rtneuron-app.py or importing the module rtneuron into own Python programs. GUI overlays can be created for specific use cases using PyQt and QML.
RTNeuron is available on the pilot system JULIA and on JURECA as environment module.
RTNeuron in aixCAVENeuron rendered by RTNeuronVisual representation of cell dyesSimulation playbackInteractive circuit slicingConnection browsing
Date of release
February 2018
Version of software
2.13.0
Version of documentation
2.13.0
Software available
https://developer.humanbrainproject.eu/docs/projects/RTNeuron/2.11/index.html; Open sourcing scheduled for June 2018
The development of Remote Connection Manager (RCM) was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
The Remote Connection Manager (RCM) is an application that allows HPC users to perform remote visualisation on Cineca HPC clusters.
The “Remote Connection Manager” works on the following operating systems: Windows, Linux, Mac OSX
(OSX Mountain Lion users need to install XQuartz: http://xquartz.macosforge.org/landing/)
The development of FLAT was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated.
FLAT is a spatial indexing tool, which enables scalable range queries on (3D) spatial datasets. Given the user input, which should be a query range, and the dataset to be queried, FLAT returns all the objects that intersect with the query range.
In particular, both the query ranges and the spatial objects should be represented using minimum bounding rectangle, which is the geometry approximation bounding the underlying spatial object.
FLAT outperforms the state-of-the-art spatial indexing techniques (e.g. R-trees, grid file) on extremely dense datasets.
Date of release
March 2015
Version of software
1.0
Version of documentation
1.0
Software available
Collaboratory, integrated and part of BBP SDK tool set
This website describes the results of the “High Performance Analytics and Computing” (HPAC) Platform of the Human Brain Project (HBP) from the first three project phases (Ramp-up Phase 10/2013-03/2016, SGA1 04/2016-03/2018 and SGA2 04/2018-03/2020).
Due to a major project-internal reorganisation, this website will no longer be updated after March 2020.
More recent information can be found on humanbrainproject.eu and ebrains.eu.
Information about the Fenix Research Infrastructure and the ICEI project, including resource access, are available on their website.
Follow EBRAINS Computing Services@HBPHighPerfComp and Fenix RI@Fenix_RI_eu on Twitter to learn about the most recent developments and to get to know about upcoming opportunities for calls and collaborations!
The Virtual Brain (TVB) is a large-scale brain simulator. With a community of thousands of users around the world, TVB has become a validated, popular and standard choice for the simulation of whole brain activity. TVB users can create simulations using neural mass models which can produce outputs for different analysis and modalities. TVB allows scientists to explore and analyze both simulated and experimental data. It contains analytic tools for evaluating relevant scientific parameters in light of that data. The current implementation of TVB is written in Python, with limited large-scale parallelization over different parameters. The objective of TVB-HPC is to enable large-scale parallelization of TVB simulations by making use of high performance computing to explore large parameter spaces for the models. With this approach, neuroscientists can define their models in a domain specific language based on NeuroML and automatically generate code which can run either on GPUs or on CPUs with different architectures and optimizations. The result is a framework that hides the complexity of writing robust parallel code and offers neuroscientists a fast and efficient access to high performance computing. TVB-HPC is publicly available on GitHub and, at the end of HBP project phase SGA2, it will be possible to launch large parameter simulations using code automatically generated with this framework via the HBP Collaboratory.
Science has driven the development of the NEST simulator for the past 20 years. Originally created to simulate the propagation of synfire chains using single-processor workstations, we have pushed NEST’s capabilities continuously to address new scientific questions and computer architectures. Prominent examples include studies on spike-timing dependent plasticity in large simulations of cortical networks, the verification of mean-field models, models of Alzheimer’s and Parkinson’s disease and tinnitus. Recent developments include a significant reduction in memory requirements, as demonstrated by a record-breaking simulation of 1.86 billion neurons connected by 11.1 trillion synapses on the Japanese K supercomputer, paving the way for brain-scale simulations.
