Skip to content
Snippets Groups Projects
Code owners
Assign users and groups as approvers for specific file changes. Learn more.

OpenScenarioEngine

Generator Documentation

The behavior tree of the OpenScenarioEngine is created for the most part through the utilization of a Python code generator. In the following the background, basic usage, and technical details, such as integration with manually written code or schema processing will be described in detail.

:point_up: Hint

For convenience, the generated files are already checked into the repository (engine/gen), so calling the generator is optional.

Background

Parallel to the development of OpenSCENARIO support for parsing and importing of OpenSCENARIO XML conform descriptions into C/C++ objects have been developed, and are available through the OpenSCENARIO API and OpenScenarioLib. Both are based on the official ASAM OpenSCENARIO XML specification, ensuring full compliance with the standard, and generated from a well defined UML model, also available in the sources.

When an OpenSCENARIO description is being read by the parser component of the OpenScenarioLib, it maps each individual tag onto an object that implements the OpenSCENARIO API, such as the Storyboard or SpeedAction class. While this provides access to validated scenario file information, it does not provide the logical relationship between objects.

The objective of the OpenScenarioEngine generator is to extract maximum logical relationship data from the XML model and augment it with metadata to create a functional behavior tree. In order to do so, the UML model for both the OpenSCENARIO API and OpenScenarioLib was further used to generate the basis for the OpenScenarioEngine. This means that the generator uses the emitted json representation of the OpenSCENARIO API, also stored here: generator/model/v1_3.json.

Usage

cd generator
python3 generate.py

Command Line Arguments

usage: generate.py [-h] [--clang-format CLANG_FORMAT] [-l LOG_FILE] [-d] [-c CLASSES [CLASSES ...]] [-f]

OpenScenarioEngine CodeGenerator

optional arguments:
  -h, --help            show this help message and exit
  --clang-format CLANG_FORMAT
                        Specifies executable for clang-format
  -l LOG_FILE, --log_file LOG_FILE
                        Name of log file
  -d, --debug           Additionally print debug information
  -c CLASSES [CLASSES ...], --classes CLASSES [CLASSES ...]
                        Restricts generation scope to given class (e.g. TeleportAction)
  -f, --force           Forces generation even for detected user specializations

:point_up: Hint

The generator applies clang-format on every C++ source file.
The path can be specified (mandatory under Windows), e.g. by calling python3 generate.py --clang-format="c:\clang-format.exe".
Be aware that generation under Windows might mix up line endings w.r.t the files already checked-in.

Dependencies

  • Python 3.10
  • jinja2
  • case-converter

Technical Details

:warning: Attention

If you are not familiar with the underlying behavior tree, it's strongly recommended to read the Architecture Documentation first.

Before generation begins, the UML model generator/model/v1_3.json is converted into a tree:

Storyboard (1)
├── Init
│   └── InitActions
│       ├── GlobalAction
│       │   ├── EnvironmentAction (2)
│       │   └── ...
│       ├── UserDefinedAction
│       │   └── ...
│       └── Private
│           └── ...
├── Stories
│   └── ...
└── StopTrigger
    └── ConditionGroup
        └── Condition
             ├── ByEntityCondition
             │   ├── TriggeringEntities
             │   └── EntityCondition
             │       └── EndOfRoadCondition (2)
             └── ByValueCondition

Legend

  1. Hardcoded root of the tree
  2. Leaves of the tree = Actual condition or action, which need to be implemented manually.

In general, the generator distinguishes between leaves and "everything else" (aka nodes). In order to do so, the generator need a selector, which defines a certain branch of the tree (see metainfo.py).

Example

'selector':
[
    # node, pattern, ose_type, dependencies
    ('InitActions', '\w+Action$', 'Storyboard/GenericAction', ['Environment']),
    ('Action', '\w+Action$', 'Storyboard/GenericAction', ['Environment']),
]

For each selector, the generator searches for matching (regex) patterns within the types of the UML model. If a match is found and the root of the selector is part of the branch, the whole branch is being processed.

Note: dependencies are discussed further down

For the first selector of the example, a matching branch could be:

Storyboard > Init > InitActions > GlobalAction > EnvironmentAction
^ start             ^ required node              ^ matches \w+Action$
  • Storyboard, Init, InitActions, and GlobalAction are processed as Node.
  • EnvironmentAction will e processed as Leaf.
  • EnvironmentAction also has children, such as Environment, which will become Parameters of the action.

Processing of Nodes, Leaves, and Parameters are discussed further down in detail. For sake of clarity, Integration of Generated Code into the file structure of the engine and Definition of Dependencies will be described first:

Integration of Generated Code

This section describes how manually written code and generated code play together. From the perspective of the code generator, the following directories are most important:

engine
├── gen . . . . . . . . . . . . (1)
│   ├── Conversion
│   │   └── OscToNode . . . . . (2)
│   └── Storyboard  . . . . . . (3)
│       ├── ByEntityCondition
│       ├── ByValueCondition
│       ├── GenericAction
│       └── MotionControlAction
└── src . . . . . . . . . . . . (4)
    ├── Conversion
    │   ├── OscToMantle . . . . (5)
    │   └── OscToNode
    ├── Node  . . . . . . . . . (6)
    ├── Storyboard
    │   ├── ByEntityCondition
    │   ├── ByValueCondition
    │   ├── GenericAction
    │   └── MotionControlAction
   (7)

Legend

  1. Generated files are located within this subdirectory.
  2. All Node parsers are located here (= everything except leaves).
  3. Leaf code is located here.
  4. Manually written files are located within this subdirectory.
  5. Parameter converters are located here.
    Currently not generated, but there is commented out code for generating stubs (within gen).
  6. Customized yase nodes, used by certain OscToNode parsers.
  7. The actual engine has several other source files, but they are not generated.

Interaction between generated and manually written files is done on a file basis, meaning that a file either exists in the file structure of gen or src. Therefore, the generator always checks if a corresponding file already exists in src and if so, it will be skipped.

:point_up: Hint

Use --force to suppress this check. After forced file generation, a diff between gen and src can be useful when adding new features.

The generator generates the complete storyboard parsers and leaves. For unimplemented conditions or actions (= no corresponding files within src), stubs will be generated.

:point_up: Hint

The stubs are intended as entry point for implementers. Files which need modification, such as an unimplemented condition, can be simply moved from gen to src (example provided here). From this moment on, the generator will simply skip generation for this file. The corresponding cmake file is updated as last step, when executing the generator.

Defining Dependencies

Dependencies are needed by conditions and actions (via the aforementioned selector), and by some parameter converters (see Processing Parameter further below). Right now, new dependencies need to be (manually) provided by blackboard of the behavior trees root (see yase documentation and src/Node/RootNode.h). Once available, leaves and attached parameter converters can parameterized to get access to their dependencies over the nodes blackboard.

Available dependencies are defined in the metainfo.py:

'dependencies':
{
    // reference used in selector and converters
    'Environment':
    {
        // variable name, if applicable
        'name': 'environment',
        // type, if applicable
        'type': 'std::shared_ptr<mantle_api::IEnvironment>',
        // include file
        'include': '<MantleAPI/Execution/i_environment.h>'
    }
}

Processing Nodes

Given a branch of the model tree, the generator starts to generate C++ files for the nodes parser by calling generate_parser(). The files will be generated in the folder engine/gen/Conversion/OscToNodes (see also Architecture Documentation).

Children (= properties) stand in a certain relation to their parent or to each other. The generator distinguishes the following:

  1. xor

    The processes OpenScenario API object offer getters for each xor-ed child, but in a valid scenario description one and only one getter will provide an instantiated object. So one and only one paths is valid.

    Applies To
    Properties of a node with attribute isXorElement set to True.

    Example
    The Condition object (c.f. line 2361 in v1_3.json) can hold either a ByEntityCondition or a ByValueCondition, both properties in an xor-relationship.

    The generator will create the following code:

    yase::BehaviorNode::Ptr parse(std::shared_ptr<NET_ASAM_OPENSCENARIO::v1_3::ICondition> condition) {
      if (auto element = condition->GetByEntityCondition()) {
        return parse(element);
      }
      if (auto element = condition->GetByValueCondition()) {
        return parse(element);
      }
      throw std::runtime_error("Corrupted openSCENARIO file: No choice made within std::shared_ptr<NET_ASAM_OPENSCENARIO::v1_3::ICondition>");
    }

    Used Templates

  2. list

    The processed OpenScenario API object offers a getter which returns a list of sub-objects.

    Applies To
    Properties of a node with attribute isList set to True.

    Example
    The Story object (c.f. line 11302 in v1_3.json) holds a list of acts. Processing needs to consider, that parse(story->GetActs()) is a vector of Act objects.

    Before continuing with the next sub-type Act, the generator will create a list-seam for Acts:

    yase::BehaviorNode::Ptr parse(std::vector<std::shared_ptr<NET_ASAM_OPENSCENARIO::v1_3::IAct>> acts)
    {
      auto node = std::make_shared<yase::ParallelNode>("Acts");
      for (const auto& act : acts)
      {
        node->addChild(parse(act));
      }
      return node;
    }

    Used Templates

  3. Transient

    The processed OpenScenario API only contains a single child.

    Applies To
    Checked by logic.

    Example
    The Trigger object (c.f. line 13681 in v1_3.json) holds only single element conditionGroups.

    The generator will create the following code:

    yase::BehaviorNode::Ptr parse(std::shared_ptr<NET_ASAM_OPENSCENARIO::v1_3::ITrigger> trigger)
    {
      return trigger ? parse(trigger->GetConditionGroups()) : nullptr;
    }

    The code assumes that all types are optional. This is a valid simplification as it is expected that scenario descriptions are validated before parsing.

    Used Templates

  4. Glue Logic

    The processed OpenScenario API object is a leaf.

    Applies To
    Checked by logic.

    Example
    The AddEntityAction object.

    The generator will create the following code:

    yase::BehaviorNode::Ptr parse(std::shared_ptr<NET_ASAM_OPENSCENARIO::v1_3::IAddEntityAction> addEntityAction)
    {
      return std::make_shared<AddEntityAction>(addEntityAction);
    }

    The code makes use of a specialized AddEntityAction (yase) node, which will be generated during processing of the leaves (see next chapter).

    Used Templates

Processing Leaves

For leaves, which are either conditions or actions, the corresponding stubs are generated. As described here), these stubs are always 4 files:

  1. A yase node
  2. A base class header holding parameters
  3. A derived implementation header
  4. The actual implementation

Recalling the selector described here, a leaf matching pattern is always accompanied by an ose_type. This ose_type selects the corresponding leaf templates.

Example

For the leaf SimulationTimeCondition, the ose_type will resolve to Storyboard/ByValueCondition, picking the following templates:

  1. Storyboard/ByValueCondition.h.j2
  2. Storyboard/ByValueCondition_base.h.j2
  3. Storyboard/ByValueCondition_impl.h.j2
  4. Storyboard/ByValueCondition_impl.cpp.j2

Note that a SimulationTimeCondition does not rely on an entity, while a TraveledDistanceCondition certainly does. So template selection is used to separates conditions and actions with an entity relation from those without. Entities with entity relation get access to a service named EntityBroker, provided by the blackboard of the yase tree (see https://gitlab.eclipse.org/eclipse/openpass/yase/-/tree/main/agnostic_behavior_tree).

Processing Parameter

Augmenting model data with metadata is also necessary for decoupling of objects implementing the OpenSCENARIO API from the actual implementations of conditions and actions, formulated by the design principle separation of values described in the Architecture Documentation. In essence, the generator possesses knowledge regarding the mappings between properties from the OpenSCENARIO API and internally used values. The most used mapping is for the complex datatype Position offering diverse specializations, such as WorldPosition, RelativeLanePosition, or GeoPosition. By applying the mapping, the specification (which specifies the source for the position) is transformed into its result - a Cartesian point and an orientation.

Example

     OpenScenarioAPI                        OpenScenarioEngine
┌────────────────────────┐              ┌─────────────────────────┐
│ ReachPositionCondition │              │ ReachPositionCondition  │
├────────────────────────┤              ├─────────────────────────┤
│ + double   tolerance   │              │ + double      tolerance │
│ + Position position    │ - mapping -> │ + mantle_api::Pose pose │
└────────────────────────┘              └─────────────────────────┘

Definition

The corresponding conversion is defined in the metainfo.py:

'converter':
{
  // **OpenScenario API** type, which shall be mapped
  'Position':
  {
      // Function name of the converter
      'converter': 'OPENSCENARIO::ConvertScenarioPosition',
      // Return type of the converter
      'return_type': 'std::optional<mantle_api::Pose>',
      // Necessary include file 
      'include': '"Conversion/OscToMantle/ConvertScenarioPosition.h"',
      // type == "static": Call converter on initialization
      // type == "dynamic: Call converter on demand (wrap in lambda)
      'type': 'static',
      // Optional: dependencies of the converter
      'dependencies':
       [
          "Environment"
       ]
       // Other properties of the leaf, necessary for conversion
       'consumes': ['Foo', 'Bar']
  }, ...
}

Note that parameter belong to leaves and the corresponding parsers are used by the generating templates.

Output

// from "dependencies"
#include <MantleAPI/Execution/i_environment.h>
// from "include"
#include "Conversion/OscToMantle/ConvertScenarioPosition.h"

...

// When approaching parameter "Position":
[=](){ // <- dynamic: will be mapped to "GetPosition()"
  // "converter"
  return OPENSCENARIO::ConvertScenarioPosition(
    // from "dependencies"
    environment,
    // from "Position"
    condition->GetPosition(),
    // Consumes Foo
    condition->GetFoo()
    // Does not consumes Bar, because not found <- no error
    );
}

Note that consume really means that the property (Foo) is removed from the list of properties, which shall be directly mapped.
For generic usage, it is allowed to specify multiple parameters, but only those available will be consumed.

Processing Rules

Rules are a particular type of parameter that allow for logical comparisons between values. An example of such a comparison could be determining if the current value of a condition is greater than or equal to another parameter within the node. Therefore the rule parser is configured to consume the corresponding value of the node and instead provide a class which already contains the correct logic for checking if the rule is satisfied (see engine/src/Conversion/OscToMantle/ConvertScenarioRule.h)