The development of PyCOMPSs was co-funded by the HBP during the Ramp-up Phase. This page is kept for reference but will no longer be updated, apart from release notes.
PyCOMPSs is the Python binding of COMPSs, (COMP Superscalar) a coarse-grained programming model oriented to distributed environments, with a powerful runtime that leverages low-level APIs (e.g. Amazon EC2) and manages data dependencies (objects and files). From a sequential Python code, it is able to run in parallel and distributed.
Releases
PyCOMPSs is based on COMPSs. COMPSs version 1.3 was released in November 2015, version 1.4 in May 2016 and version 2.0 in November 2016.
New features in COMPSs v1.3
- Runtime
- Persistent workers: workers can be deployed on computing nodes and persist during all the application lifetime, thus reducing the runtime overhead. The previous implementation of workers based on a per task process is still supported.
- Enhanced logging system
- Interoperable communication layer: different inter-nodes communication protocol is supported by implementing the Adaptor interface (JavaGAT and NIO implementations already included)
- Simplified cloud connectors interface
- JClouds connector
- Python/PyCOMPSs
- Added constraints support
- Enhanced methods support
- Lists accepted as a tasks’ parameter type
- Support for user decorators
- Tools
- New monitoring tool: with new views, as workload and possibility of visualizing information about previous runs
- Enhanced tracing mechanism
- Simplified execution scripts
- Simplified installation on supercomputers through better scripts
New features in COMPSs v1.4
- Runtime
- Added support for Docker
- Added support for Chameleon Cloud
- Object cache for persistent workers
- Improved error management
- Added connector for submitting tasks to MN supercomputer from external COMPSs applications
- Bug-fixes
- Python/PyCOMPSs
- General bug-fixes
- Tools
- Enhanced Tracing mechanism:
- Reduced overhead using native Java API
- Added support for communications instrumentation added
- Added support for PAPI hardware counters
- Known Limitations
- When executing Python applications with constraints in the cloud the initial VMs must be set to 0
New features in COMPSs v2.0 (released November 2016)
- Runtime:
- Upgrade to Java 8
- Support to remote input files (input files already at workers)
- Integration with Persistent Objects
- Elasticity with Docker and Mesos
- Multi-processor support (CPUs, GPUs, FPGAs)
- Dynamic constraints with environment variables
- Scheduling taking into account the full tasks graph (not only ready tasks)
- Support for SLURM clusters
- Initial COMPSs/OmpSs integration
- Replicated tasks: Tasks executed in all the workers
- Explicit Barrier
- Python:
- Python user events and HW counters tracing
- Improved PyCOMPSs serialization. Added support for lambda and generator parameters.
- C:
- Constraints support
- Tools:
- Improved current graph visualization on COMPSs Monitor
- Improvements:
- Simplified Resource and Project files (NO retrocompatibility)
- Improved binding workers execution (use pipes instead of Java Process Builders)
- Simplifies cluster job scripts and supercomputers configuration
- Several bug fixes
- Known Limitations:
- When executing python applications with constraints in the cloud the initial VMs must be set to 0
New features in PyCOMPSs/COMPSs v2.1 (released June 2017)
- New features:
- Runtime:
- New annotations to simplify tasks that call external binaries
- Integration with other programming models (MPI, OmpSs,..)
- Support for Singularity containers in Clusters
- Extension of the scheduling to support multi-node tasks (MPI apps as tasks)
- Support for Grid Engine job scheduler in clusters
- Language flag automatically inferred in runcompss script
- New schedulers based on tasks’ generation order
- Core affinity and over-subscribing thread management in multi-core cluster queue scripts (used with MKL libraries, for example)
- Python:
- @local annotation to support simpler data synchronizations in master (requires to install guppy)
- Support for args and kwargs parameters as task dependencies
- Task versioning support in Python (multiple behaviors of the same task)
- New Python persistent workers that reduce overhead of Python tasks
- Support for task-thread affinity
- Tracing extended to support for Python user events and HW counters (with known issues)
- C:
- Extension of file management API (compss_fopen, compss_ifstream, compss_ofstream, compss_delete_file)
- Support for task-thread affinity
- Tools:
- Visualization of not-running tasks in current graph of the COMPSs Monitor
- Runtime:
- Improvements
- Improved PyCOMPSs serialization
- Improvements in cluster job scripts and supercomputers configuration
- Several bug fixes
- Known Limitations
- When executing Python applications with constraints in the cloud the <InitialVMs> property must be set to 0
- Tasks that invoke Numpy and MKL may experience issues if tasks use a different number of MKL threads. This is due to the fact that MKL reuses threads in the different calls and it does not change the number of threads from one call to another.
New features in PyCOMPSs/COMPSs v2.3 (released June 2018)
- Runtime
- Persistent storage API implementation based on Redis (distributed as default implementation with COMPSs)
- Support for FPGA constraints and reconfiguration scripts
- Support for PBS Job Scheduler and the Archer Supercomputer
- Java
- New API call to delete objects in order to reduce application memory usage
- Python
- Support for Python 3
- Support for Python virtual environments (venv)
- Support for running PyCOMPSs as a Python module
- Support for tasks returning multiple elements (returns=#)
- Automatic import of dummy PyCOMPSs AP
- C
- Persistent worker with Memory-to-memory transfers
- Support for arrays (no serialization required)
- Improvements
- Distribution with docker images
- Source Code and example applications distribution on Github
- Automatic inference of task return
- Improved obsolete object cleanup
- Improved tracing support for applications using persistent memory
- Improved finalization process to reduce zombie processes
- Several bug fixes
- Known limitations
- Tasks that invoke Numpy and MKL may experience issues if a different MKL threads count is used in different tasks. This is due to the fact that MKL reuses threads in the different calls and it does not change the number of threads from one call to another.
New features in PyCOMPSs/COMPSs v2.5 (released June 2019)
- Runtime:
- New Concurrent direction type for task parameter.
- Multi-node tasks support for native (Java, Python) tasks. Previously, multi-node tasks were only posible with @mpi or @decaf tasks.
- @Compss decorator for executing compss applications as tasks.
- New runtime api to synchronize files without opening them.
- Customizable task failure management with the “onFailure” task property.
- Enabled master node to execute tasks.
- Python:
- Partial support of numba in tasks.
- Support for collection as task parameter.
- Supported task inheritance.
- New persistent MPI worker mode (alternative to subprocess).
- Support to ARM MAP and DDT tools (with MPI worker mode).
- C:
- Support for task without parameters and applications without src folder.
- Improvements:
- New task property “targetDirection” to indicate direction of the target object in object methods. Substitutes the “isModifier” task property.
- Warnings for deprecated or incorrect task parameters.
- Improvements in Jupyter for Supercomputers.
- Upgrade of runcompss_docker script to docker stack interface.
- Several bug fixes.
- Known Limitations:
- Tasks that invoke Numpy and MKL may experience issues if a different MKL threads count is used in different tasks. This is due to the fact that MKL reuses threads in the different calls and it does not change the number of threads from one call to another.
- C++ Objects declared as arguments in a coarse-grain tasks must be passed in the task methods as object pointers in order to have a proper dependency management.
- Master as worker is not working for executions with persistent worker in C++.
- Coherence and concurrent writing in parameters annotated with the “Concurrent” direction must be managed by the underlaying distributed storage system.
- Delete file calls for files used as input can produce a significant synchronization of the main code.
PyCOMPSs/COMPSs PIP installation package
This is a new feature available since January 2017.
Installation:
- Check the dependencies in the PIP section of the PyCOMPSs installation manual (available at the documentation section of compss.bsc.es). Be sure that the target machine satisfies the mentioned dependencies.
- The installation can be done in various alternative ways:
- Use PIP to install the official PyCOMPSs version from the pypi live repository:
sudo -E python2.7 -m pip install pycompss -v
- Use PIP to install PyCOMPSs from a pycompss.tar.gz
sudo -E python2.7 -m pip install pycompss-version.tar.gz -v
- Use the setup.py script
sudo -E python2.7 setup.py install
- Use PIP to install the official PyCOMPSs version from the pypi live repository:
Internal report
How multi-scale applications can be developed using PyCOMPSs (accessible by HBP members only):
| Date of release | June 2019 |
| Version of software | 2.5 |
| Version of documentation | 2.5 |
| Software available | http://compss.bsc.es |
| Documentation | https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/documentation |
| Responsible | BSC Workflows and Distributed Computing Group: support-compss@bsc.es |
| Requirements & dependencies | http://compss.bsc.es/releases/compss/latest/docs/COMPSs_Installation_Manual.pdf?tracked=true |
| Target system(s) | Supercomputers or clusters with different nodes, distributed computers, grid and cloud architectures |