Transform DataStore to DataInterface to optimize RAM consumption
Current Implementation
Currently the DataStore is used as intermediate storage between components/core of the simulation and the observers.
openPASS core / components -> DataStore -> Observer
All data that might be relevant for logging/output of the simulation is sent to the DataStore by the the openPASS core and components. The DataStore saves all received data and makes it available for observers. An observer can query any information from the DataStore at any time.
Problem
We observed, that with increasing size of the simulation, the RAM consumption grows rapidly. Having a few houndred agents or running a scenario with more than 30 seconds duration can require more than 50Gb RAM.
Proposed Solution
Transform the DataStore to a DataInterface
This means, that data won't be stored in future but the DataInterface serves as a proxy to forward information to all observers. The architectural design of the setup won't change. The simulation core and all components will send data in the same way as they do now. Also querying information by the observers will stay the same. However, it will have to happen in each time step, as data won't be stored by the DataStore anymore.
Cons:
- Observers are responsible of storing data that's needed later on.
- Observers have to be triggered in each timestep and have to run synchronously with the simulation
Pros:
- Optimized storing of only relevant information in observers
- Lean and generic DataInterface component in the platform and customized observers that can be optimized individually to their application
- No need for a new component with external dependency (e.g. database)
Next Steps
We conducted some preliminary tests proving that the RAM consumption can be significantly reduced by the proposed solution. In an exemplary simulation we reduced the RAM consumption by a factor of 7.
@all Please comment in the issue if you support the solution, have questions or concerns.
@rparisha2 @rbiegel Feel free to add, if I missed something.
Note: This issue has been discussed in the AC on 20th Mai 2021.