Commit c943e052 authored by René Paris's avatar René Paris
Browse files

Merge branch '29-transform-datastore-to-datainterface-to-optimize-ram-consumption' into 'servant'

Rename DataStore to DataBuffer

See merge request eclipse/simopenpass/simopenpass!19
parents feedb72f 6506deae
......@@ -53,7 +53,7 @@ Each simulation run splits up into 3 phases: Bootstrap tasks, Stepping and Final
2. After this Spawning, PreAgent, NonRecurring, Recurring and Syncronize Tasks are processed for each timestep.
First all Runtime Spawners are triggered.
Whenever a new agent is spawned, all its components are parsed to *Trigger* and *Update* tasks, grouped and sorted by execution time as non-recurring and recurring tasks.
Next all PreAgents task are executed, which includes publishing the world state to the DataStore in triggering all EventDetectors and Manipulators
Next all PreAgents task are executed, which includes publishing the world state to the DataBuffer in triggering all EventDetectors and Manipulators
Non-recurring tasks are executed just once and will be deleted afterwards (*init* flag in ComponentRepository).
Right after execution of all agent-based tasks, the Syncronize phase synchronizes all changes to the world as world state for the next timestep.
......
......@@ -57,7 +57,7 @@ Please refer to the individual components, for information about their published
.. todo::
The concept Cyclics and the DataStore needs further explanation.
The concept Cyclics and the DataBuffer needs further explanation.
We also need a way to better communicate, who is publishing what.
This should directly come out of the source code, to keep the documentation up to date.
......
......@@ -35,13 +35,13 @@ Then the AgentType with the components and channels is build by the DynamicAgent
![AgentBlueprintProviderSequenceDiagram](AgentBlueprintProviderSequence.svg)
\section dev_framework_modules_datastore DataStore
\section dev_framework_modules_databuffer DataBuffer
The DataStore accepts arbitrary values from different components (from framework as well as agents).
The stored values if cyclic and acyclic type are associated with an timestamp, entity id and a key (topic).
Static values are only associated with a key.
The DataBuffer accepts arbitrary values from different components (from framework as well as agents).
The stored values if cyclic and acyclic type are associated with an entity id and a key (topic) and stored for one timestep.
Static values are only associated with a key but stored for the entire run.
Typically, Observers are using the DataStore to retrieve information about the simulation.
Typically, Observers are using the DataBuffer to retrieve information about the simulation.
This can happen during the simulation run and/or at the end of a run.
For the keys, a generic datatype is used (strings).
For the values, a set of predefined datatypes is used (bool, char, int, size_t, float, double, string).
......
......@@ -53,7 +53,7 @@ Each simulation run splits up into 3 phases: Bootstrap tasks, Stepping and Final
2. After this Spawning, PreAgent, NonRecurring, Recurring and Syncronize Tasks are processed for each timestep.
First all Runtime Spawners are triggered.
Whenever a new agent is spawned, all its components are parsed to *Trigger* and *Update* tasks, grouped and sorted by execution time as non-recurring and recurring tasks.
Next all PreAgents task are executed, which includes publishing the world state to the DataStore in triggering all EventDetectors and Manipulators
Next all PreAgents task are executed, which includes publishing the world state to the DataBuffer in triggering all EventDetectors and Manipulators
Non-recurring tasks are executed just once and will be deleted afterwards (*init* flag in ComponentRepository).
Right after execution of all agent-based tasks, the Syncronize phase synchronizes all changes to the world as world state for the next timestep.
......
......@@ -57,7 +57,7 @@ Please refer to the individual components, for information about their published
.. todo::
The concept Cyclics and the DataStore needs further explanation.
The concept Cyclics and the DataBuffer needs further explanation.
We also need a way to better communicate, who is publishing what.
This should directly come out of the source code, to keep the documentation up to date.
......
/*******************************************************************************
* Copyright (c) 2020 in-tech GmbH
* Copyright (c) 2020, 2021 in-tech GmbH
*
* This program and the accompanying materials are made
* available under the terms of the Eclipse Public License 2.0
......@@ -23,15 +23,15 @@
#include "include/callbackInterface.h"
#include "common/commonTools.h"
namespace openpass::datastore {
namespace openpass::databuffer {
using Key = openpass::type::FlatParameterKey;
using Value = openpass::type::FlatParameterValue;
using Parameter = openpass::type::FlatParameter;
using Tokens = std::vector<Key>;
static const std::string WILDCARD = "*"; //!< Wildcard to match any token inside a DataStore key string. Length of 1 is mandatory.
static constexpr char SEPARATOR = '/'; //!< Separator for hierarchical DataStore key strings. Length of 1 is mandatory.
static const std::string WILDCARD = "*"; //!< Wildcard to match any token inside a key string. Length of 1 is mandatory.
static constexpr char SEPARATOR = '/'; //!< Separator for hierarchical key strings. Length of 1 is mandatory.
/*!
* \brief Representation of an component event
......@@ -85,12 +85,11 @@ public:
};
/*!
* \brief Representation of an entry in the DataStore acyclics
* \brief Representation of an entry in the acyclics
*/
struct AcyclicRow
{
AcyclicRow(openpass::type::Timestamp ts, openpass::type::EntityId id, Key k, Acyclic data) :
timestamp{ts},
AcyclicRow(openpass::type::EntityId id, Key k, Acyclic data) :
entityId{id},
key{k},
data{data}
......@@ -99,25 +98,22 @@ struct AcyclicRow
bool operator==(const AcyclicRow &other) const
{
return timestamp == other.timestamp &&
entityId == other.entityId &&
return entityId == other.entityId &&
key == other.key &&
data == other.data;
}
openpass::type::Timestamp timestamp; //!< Simulation time timestamp [ms]
openpass::type::EntityId entityId; //!< Id of the entity (agent or object)
Key key; //!< Key (topic) associated with the data
Acyclic data; //!< Acyclic data container
};
/*!
* \brief Representation of an entry in the DataStore cyclics
* \brief Representation of an entry in the cyclics
*/
struct CyclicRow
{
CyclicRow(openpass::type::Timestamp ts, openpass::type::EntityId id, Key k, Value v) :
timestamp{ts},
CyclicRow(openpass::type::EntityId id, Key k, Value v) :
entityId{id},
key{k},
tokens{CommonHelper::TokenizeString(key, SEPARATOR)},
......@@ -127,33 +123,31 @@ struct CyclicRow
bool operator==(const CyclicRow &other) const
{
return timestamp == other.timestamp &&
entityId == other.entityId &&
return entityId == other.entityId &&
key == other.key &&
value == other.value;
}
openpass::type::Timestamp timestamp; //!< Simulation time timestamp [ms]
openpass::type::EntityId entityId; //!< Id of the entity (agent or object)
Key key; //!< Key (topic) associated with the data
Tokens tokens; //!< Tokenized representation of key
Value value; //!< Data value
};
using Keys = std::vector<Key>; //!< List of keys used by the DataStore
using Values = std::vector<Value>; //!< List of values used by the DataStore
using CyclicRows = std::vector<CyclicRow>; //!< List of data rows used by the DataStore
using Keys = std::vector<Key>; //!< List of keys
using Values = std::vector<Value>; //!< List of values
using CyclicRows = std::vector<CyclicRow>; //!< List of data rows
using CyclicRowRefs = std::vector<std::reference_wrapper<const CyclicRow>>; //!< List of references to rows inside the DataStore
using AcyclicRowRefs = std::vector<std::reference_wrapper<const AcyclicRow>>; //!< List of references to acyclic rows inside the DataStore
using CyclicRowRefs = std::vector<std::reference_wrapper<const CyclicRow>>; //!< List of references to rows
using AcyclicRowRefs = std::vector<std::reference_wrapper<const AcyclicRow>>; //!< List of references to acyclic rows
/*!
* \brief A set of cyclic data elements representing a DataStore query result
* \brief A set of cyclic data elements representing a DataInterface query result
*
* Basic forward iterator properties are provided for convenient result iteration.
*
* \code{.cpp}
* const auto cyclicResult = dataStore->GetCyclic(std::nullopt, std::nullopt, "*");
* const auto cyclicResult = dataBuffer->GetCyclic(std::nullopt, "*");
*
* for (const CyclicRow& row : *cyclicResult)
* {
......@@ -174,12 +168,12 @@ public:
};
/*!
* \brief A set of acyclic data elements representing a DataStore query result
* \brief A set of acyclic data elements representing a DataInterface query result
*
* Basic forward iterator properties are provided for convenient result iteration.
*
* \code{.cpp}
* const auto acyclicResult = dataStore->GetAcyclic(std::nullopt, std::nullopt, "*");
* const auto acyclicResult = dataBuffer->GetAcyclic(std::nullopt, "*");
*
* for (const AcyclicRow& row : *acyclicResult)
* {
......@@ -198,38 +192,38 @@ public:
virtual AcyclicRowRefs::const_iterator begin() const = 0;
virtual AcyclicRowRefs::const_iterator end() const = 0;
};
} // namespace openpass::datastore
} // namespace openpass::databuffer
using namespace openpass::datastore;
using namespace openpass::databuffer;
/*!
* \brief The DataStoreReadInterface provides read-only access to an underlying DataStore implementation
* \brief The DataReadInterface provides read-only access to the data
*
* Topics (see Get* methods) for cyclics, acyclics and statics are independent of each other.
*/
class DataStoreReadInterface
class DataBufferReadInterface
{
public:
DataStoreReadInterface() = default;
DataStoreReadInterface(const DataStoreReadInterface &) = delete;
DataStoreReadInterface(DataStoreReadInterface &&) = delete;
DataStoreReadInterface &operator=(const DataStoreReadInterface &) = delete;
DataStoreReadInterface &operator=(DataStoreReadInterface &&) = delete;
virtual ~DataStoreReadInterface() = default;
DataBufferReadInterface() = default;
DataBufferReadInterface(const DataBufferReadInterface &) = delete;
DataBufferReadInterface(DataBufferReadInterface &&) = delete;
DataBufferReadInterface &operator=(const DataBufferReadInterface &) = delete;
DataBufferReadInterface &operator=(DataBufferReadInterface &&) = delete;
virtual ~DataBufferReadInterface() = default;
static constexpr bool NO_DESCEND = false;
/*!
* \brief Retrieves stored cyclic values from the data store
* \brief Retrieves stored cyclic values
*
* \param[in] time Timestamp of interest
* \param[in] entityId Entity's id
* \param[in] key Unique topic identification
*/
virtual std::unique_ptr<CyclicResultInterface> GetCyclic(const std::optional<openpass::type::Timestamp> time, const std::optional<openpass::type::EntityId> entityId, const Key &key) const = 0;
virtual std::unique_ptr<CyclicResultInterface> GetCyclic(const std::optional<openpass::type::EntityId> entityId, const Key &key) const = 0;
/*!
* \brief Retrieves stored acyclic values from the data store
* \brief Retrieves stored acyclic values
*
* \param[in] time Timestamp of interest
* \param[in] entityId Entity's id
......@@ -237,24 +231,24 @@ public:
*
* \note Current implementation ignores time and entityId
*/
virtual std::unique_ptr<AcyclicResultInterface> GetAcyclic(const std::optional<openpass::type::Timestamp> time, const std::optional<openpass::type::EntityId> entityId, const Key &key) const = 0;
virtual std::unique_ptr<AcyclicResultInterface> GetAcyclic(const std::optional<openpass::type::EntityId> entityId, const Key &key) const = 0;
/*!
* \brief Retrieves stored static values from the data store
* \brief Retrieves stored static values
*
* \param[in] key Unique topic identification
*/
virtual Values GetStatic(const Key &key) const = 0;
/*!
* \brief Retrieves keys at a specific node in the DataStore hierarchy.
* \brief Retrieves keys at a specific node in the hierarchy.
*
* The key parameter has to be prefixed with "Cyclics/", "Acyclics/" or "Statics/" to
* get access to the different types of stored elements.
*
* The following example will retrieve the list of agent ids participating in the current simulation run:
* \code{.cpp}
* const auto agentIds = dataStore.GetKeys("Statics/Agents");
* const auto agentIds = dataBuffer.GetKeys("Statics/Agents");
* \endcode
*
* The following example will retrieve a list of instantiated sensors for agent 3:
......@@ -278,42 +272,42 @@ public:
};
/*!
* \brief The DataStoreWriteInterface provides write-only access to an underlying DataStore implementation
* \brief The DataWriteInterface provides write-only access to the data
*
* Topics (see Put* methods) for cyclics, acyclics and statics are independent of each other.
*/
class DataStoreWriteInterface
class DataBufferWriteInterface
{
public:
DataStoreWriteInterface() = default;
DataStoreWriteInterface(const DataStoreWriteInterface &) = delete;
DataStoreWriteInterface(DataStoreWriteInterface &&) = delete;
DataStoreWriteInterface &operator=(const DataStoreWriteInterface &) = delete;
DataStoreWriteInterface &operator=(DataStoreWriteInterface &&) = delete;
virtual ~DataStoreWriteInterface() = default;
DataBufferWriteInterface() = default;
DataBufferWriteInterface(const DataBufferWriteInterface &) = delete;
DataBufferWriteInterface(DataBufferWriteInterface &&) = delete;
DataBufferWriteInterface &operator=(const DataBufferWriteInterface &) = delete;
DataBufferWriteInterface &operator=(DataBufferWriteInterface &&) = delete;
virtual ~DataBufferWriteInterface() = default;
/*!
* \brief Writes cyclic information into the data store
* \brief Writes cyclic information
*
* \param[in] time Timestamp associated with the provided key/value [ms]
* \param[in] entityId Id of the associated agent or object
* \param[in] key Unique topic identification
* \param[in] value Value to be written
*/
virtual void PutCyclic(const openpass::type::Timestamp time, const openpass::type::EntityId entityId, const Key &key, const Value &value) = 0;
virtual void PutCyclic(const openpass::type::EntityId entityId, const Key &key, const Value &value) = 0;
/*!
* \brief Writes acyclic information into the data store
* \brief Writes acyclic information
*
* \param[in] time Timestamp associated with the provided key/value [ms]
* \param[in] entityId Id of the associated agent
* \param[in] key Unique topic identification
* \param[in] acyclic The acyclic element to be written
*/
virtual void PutAcyclic(const openpass::type::Timestamp time, const openpass::type::EntityId entityId, const Key &key, const openpass::datastore::Acyclic &acyclic) = 0;
virtual void PutAcyclic(const openpass::type::EntityId entityId, const Key &key, const openpass::databuffer::Acyclic &acyclic) = 0;
/*!
* \brief Writes static information into the data store
* \brief Writes static information
*
* \param[in] key Unique topic identification
* \param[in] value Value to be written
......@@ -322,9 +316,9 @@ public:
virtual void PutStatic(const Key &key, const Value &value, bool persist = false) = 0;
/*!
* \brief Clears the datastore contents, except persistent static data
* \brief Clears the data contents, except persistent static data
*/
virtual void Clear() = 0;
virtual void ClearRun() = 0;
/*!
* \brief Provides callback to LOG() macro
......@@ -338,29 +332,29 @@ public:
};
/*!
* \brief The DataStoreInterface provides read/write access to an underlying DataStore implementation
* \brief The DataInterface provides read/write access to the data
*
* This interface combines DataStoreReadInterface and DataStoreWriteInterface and adds some additional
* This interface combines DataReadInterface and DataWriteInterface and adds some additional
* methods required for instantiation by the framework.
*/
class DataStoreInterface : public DataStoreReadInterface, public DataStoreWriteInterface
class DataBufferInterface : public DataBufferReadInterface, public DataBufferWriteInterface
{
public:
DataStoreInterface() = default;
DataStoreInterface(const openpass::common::RuntimeInformation *runtimeInformation, const CallbackInterface *callbacks) :
DataBufferInterface() = default;
DataBufferInterface(const openpass::common::RuntimeInformation *runtimeInformation, const CallbackInterface *callbacks) :
runtimeInformation(runtimeInformation),
callbacks(callbacks)
{
}
DataStoreInterface(const DataStoreInterface &) = delete;
DataStoreInterface(DataStoreInterface &&) = delete;
DataStoreInterface &operator=(const DataStoreInterface &) = delete;
DataStoreInterface &operator=(DataStoreInterface &&) = delete;
virtual ~DataStoreInterface() override = default;
DataBufferInterface(const DataBufferInterface &) = delete;
DataBufferInterface(DataBufferInterface &&) = delete;
DataBufferInterface &operator=(const DataBufferInterface &) = delete;
DataBufferInterface &operator=(DataBufferInterface &&) = delete;
virtual ~DataBufferInterface() override = default;
/*!
* \brief Instantiates the data store
* \brief Instantiates the data buffer
*
* \return true if instantiation was successful, false otherwise
*/
......@@ -372,13 +366,16 @@ public:
/*!
* \brief Determines the instantiation status
*
* \return true if data store is instantiated, false otherwise
* \return true if data buffer is instantiated, false otherwise
*/
virtual bool isInstantiated() const
{
return false;
}
//! \brief Clears the data of the last timestep, but keeps static data
virtual void ClearTimeStep() = 0;
void Log(CbkLogLevel logLevel, const char *file, int line, const std::string &message) const override
{
if (callbacks)
......
/*******************************************************************************
* Copyright (c) 2017, 2018, 2019 in-tech GmbH
* Copyright (c) 2017, 2018, 2019, 2021 in-tech GmbH
*
* This program and the accompanying materials are made
* available under the terms of the Eclipse Public License 2.0
......@@ -14,7 +14,7 @@
#include "include/stochasticsInterface.h"
class AgentBlueprintProviderInterface;
class DataStoreInterface;
class DataBufferInterface;
class WorldInterface;
namespace SimulationSlave {
......@@ -46,11 +46,11 @@ public:
virtual AgentFactoryInterface* GetAgentFactory() = 0;
/*!
* \brief Returns a pointer to the data store
* \brief Returns a pointer to the data buffer
*
* @return data store pointer
* @return data buffer pointer
*/
virtual DataStoreInterface* GetDataStore() = 0;
virtual DataBufferInterface* GetDataBuffer() = 0;
/*!
* \brief Returns a pointer to the EventDetectorNetwork
......
/*******************************************************************************
* Copyright (c) 2017, 2018, 2019, 2020 in-tech GmbH
* Copyright (c) 2017, 2018, 2019, 2020, 2021 in-tech GmbH
* 2016, 2017, 2018 ITK Engineering GmbH
*
* This program and the accompanying materials are made
......@@ -30,7 +30,7 @@
#include "include/runResultInterface.h"
class CallbackInterface;
class DataStoreReadInterface;
class DataBufferReadInterface;
class ParameterInterface;
class StochasticsInterface;
class WorldInterface;
......@@ -44,7 +44,7 @@ public:
WorldInterface* world,
const ParameterInterface* parameters,
const CallbackInterface* callbacks,
[[maybe_unused]] DataStoreReadInterface* dataStore) :
[[maybe_unused]] DataBufferReadInterface* dataBuffer) :
stochastics(stochastics),
world(world),
parameters(parameters),
......
/*******************************************************************************
* Copyright (c) 2017, 2018, 2019, 2020 in-tech GmbH
* Copyright (c) 2017, 2018, 2019, 2020, 2021 in-tech GmbH
* 2016, 2017, 2018 ITK Engineering GmbH
*
* This program and the accompanying materials are made
......@@ -60,7 +60,7 @@ public:
WorldInterface* world,
EventNetworkInterface* eventNetwork,
const std::string& sceneryPath,
DataStoreReadInterface* dataStore) = 0;
DataBufferReadInterface* dataBuffer) = 0;
//-----------------------------------------------------------------------------
//! Returns the observation module mapping.
......
/*******************************************************************************
* Copyright (c) 2020 in-tech GmbH
* Copyright (c) 2020, 2021 in-tech GmbH
*
* This program and the accompanying materials are made
* available under the terms of the Eclipse Public License 2.0
......@@ -24,7 +24,7 @@
#include <vector>
#include "common/logEntryBase.h"
#include "include/dataStoreInterface.h"
#include "include/dataBufferInterface.h"
/**
* @brief Interface which has to be provided by observation modules
......@@ -32,8 +32,8 @@
class PublisherInterface
{
public:
PublisherInterface(DataStoreWriteInterface *const dataStore) :
dataStore(dataStore)
PublisherInterface(DataBufferWriteInterface *const dataBuffer) :
dataBuffer(dataBuffer)
{
}
......@@ -46,33 +46,33 @@ public:
virtual ~PublisherInterface() = default;
/*!
* \brief Writes information into a data store backend
* \brief Writes information into a data buffer backend
*
* \param key[in] Unique topic identification
* \param value[in] Value to be written
*/
virtual void Publish(const openpass::datastore::Key &key, const openpass::datastore::Value &value)
virtual void Publish(const openpass::databuffer::Key &key, const openpass::databuffer::Value &value)
{
}
/*!
* \brief Writes acyclic information into a data store backend
* \brief Writes acyclic information into a data buffer backend
*
* \param value[in] The acyclic event
*/
virtual void Publish(const openpass::datastore::Key &key, const openpass::publisher::LogEntryBase &event)
virtual void Publish(const openpass::databuffer::Key &key, const openpass::publisher::LogEntryBase &event)
{
}
/*!
* \brief Writes acyclic information into a data store backend
* \brief Writes acyclic information into a data buffer backend
*
* \param value[in] The acyclic event
*/
virtual void Publish(const openpass::datastore::Key &key, const openpass::datastore::ComponentEvent &event)
virtual void Publish(const openpass::databuffer::Key &key, const openpass::databuffer::ComponentEvent &event)
{
}
protected:
DataStoreWriteInterface *const dataStore; //!< References the datastore backend
DataBufferWriteInterface *const dataBuffer; //!< References the dataBuffer backend
};
/*******************************************************************************
* Copyright (c) 2020 in-tech GmbH
* Copyright (c) 2020, 2021 in-tech GmbH
*
* This program and the accompanying materials are made
* available under the terms of the Eclipse Public License 2.0
......@@ -11,12 +11,12 @@
#pragma once
#include "common/openPassTypes.h"
#include "include/dataStoreInterface.h"
#include "include/dataBufferInterface.h"
namespace openpass::publisher {
/**
* @brief Base class for logging of acyclic events in the datastore
* @brief Base class for logging of acyclic events in the dataBuffer
* @see openpass::publisher::coreDataPublisher
* @see openpass::publisher::LogEntry
*/
......@@ -40,7 +40,7 @@ public:
openpass::type::TriggeringEntities triggeringEntities;
openpass::type::AffectedEntities affectedEntities;
virtual operator openpass::datastore::Acyclic() const = 0;
virtual operator openpass::databuffer::Acyclic() const = 0;
// We do not allow copy to force move semantics
LogEntryBase(const LogEntryBase &) = delete;
......
/*******************************************************************************
* Copyright (c) 2020 in-tech GmbH
* Copyright (c) 2020, 2021 in-tech GmbH
*
* This program and the accompanying materials are made
* available under the terms of the Eclipse Public License 2.0
......@@ -12,7 +12,7 @@
#include <string>
#include <vector>
#include "common/parameter.h"
#include "include/dataStoreInterface.h"
#include "include/dataBufferInterface.h"
/**
* @brief Wrapper for building an ObservationInstance
......
......@@ -6,7 +6,6 @@ add_openpass_target(
HEADERS
callbacks.h
coreDataPublisher.h
timeKeeper.h
../../common/xmlParser.h
cephesMIT/mconf.h
......
/*******************************************************************************
* Copyright (c) 2020 in-tech GmbH
* Copyright (c) 2020, 2021 in-tech GmbH
*
* This program and the accompanying materials are made
* available under the terms of the Eclipse Public License 2.0
......@@ -10,13 +10,11 @@
#include "coreDataPublisher.h"
#include "timeKeeper.h"
namespace openpass::publisher {
void CoreDataPublisher::Publish(const openpass::datastore::Key &key, const openpass::publisher::LogEntryBase &event)
void CoreDataPublisher::Publish(const openpass::databuffer::Key &key, const openpass::publisher::LogEntryBase &event)
{
dataStore->PutAcyclic(openpass::scheduling::TimeKeeper::Now(), CORE_ID, key, event);
dataBuffer->PutAcyclic(CORE_ID, key, event);
}
} // namespace openpass::publisher
\ No newline at end of file
} // namespace openpass::publisher