Co-simulation with AVxcelerate Sensor (Beta)

Introduction

The Simulation Framework provides a built-in support for co-simulation with ANSYS AVxcelerate Sensor, enabling high-fidelity sensor simulation within the simulation environment. The AVxcelerate Sensor co-simulation feature allows you to integrate realistic sensor models (cameras, lidars, radars, etc.) that can process the ground truth data from the simulation and provide sensor-specific outputs.

The co-simulation architecture consists of several key components:

• AVxcelerate Sensor Connector Activity: A built-in activity that manages the communication with AVxcelerate Sensor service connection
• AVxcelerate VSS API Adapter: Handles gRPC communication with the AVxcelerate Sensor service
• OSI to VSS Converter: Converts Open Simulation Interface (OSI) data to AVxcelerate VSS format
• Configuration System: Manages co-sim specs, like sensor configuration, object mappings, and connection parameters

The AVxcelerate Sensor co-simulation operates by:

1. Receiving ground truth data from the gt-gen (OSI SensorView)
2. Converting the data to AVxcelerate VSS format
3. Sending the data to the AVxcelerate Sensor service via gRPC
4. Processing AVxcelerate Sensor logic
5. Retrieving sensor outputs (coming soon)

Add one AVxcelerate Sensor activity

To add an AVxcelerate Sensor activity to your simulation, you need to configure it in the simulation scheduling section of your configuration file. Creation of one generic AVxcelerate Sensor activity is available as built-in type.

Basic configuration

Add the following activity configuration to your simulation_scheduling.activities array:

Configuration parameters

Based on which kind of sensor actually runs the AVxcelerate Sensor Service, you can give the activity a specific "name". It is identified as AVxcelerate Sensor activity once you specify the "avx_sensor_setting" field.

The avx_sensor_setting object contains the following parameters:

Dependencies

The AVxcelerate Sensor activity subscribes to SensorViewTopic as input and hence depends on the groundtruth_generator_activity to receive the simulation's ground truth data. This ensures that the sensor receives the latest world state before processing.

Activity behavior

The AVxcelerate Sensor activity:

1. Subscribes to SensorViewTopic to receive ground truth data
2. Initializes the AVxcelerate Sensor service connection during startup
3. Converts OSI SensorView moving objects to AVxcelerate VSS WorldUpdate format (before VSS API supports OSI as native type)
4. Sends the converted data to the AVxcelerate Sensor service via gRPC
5. Unloads the corresponding AVxcelerate Sensor at the end of its lifecycle

Co-sim configuration

The AVxcelerate Sensor co-simulation requires a dedicated configuration file that specifies the sensor setup, connection parameters, and object mappings.

Configuration file structure

The AVxcelerate Sensor configuration file must be in JSON format and contain the following structure:

Configuration fields

Object mapping

The ObjectMapping array defines how objects in your OpenScenario file correspond to AVxcelerate Sensor assets. Here is an example where two OpenScenario objects map to AVxcelerate assets. Note that based on which AVxcelerate Sensor version is used, the path towards asset files could be different.

• OpenScenarioObjectName: The name of the object as defined in your OpenScenario file

  • The name can be found under Entities/ScenarioObject:name in the *.xosc file

    <Entities>

    <ScenarioObject name="Ego">

    <CatalogReference catalogName="vehicle_catalog" entryName="car"></CatalogReference>

    </ScenarioObject>

    <ScenarioObject name="SideVehicle">

    <CatalogReference catalogName="vehicle_catalog" entryName="car"></CatalogReference>

    </ScenarioObject>

    </Entities>
• AVxAsset: Path to the corresponding AVxcelerate asset file (.asset format). Note that Simfwk only sends the path as string to AVxcelerate Sensor as configuration; the asset files must be resolvable by AVxcelerate Sensor itself.

Example configuration

Here's a complete example Simfwk configuration for AVxcelerate Sensor co-simulation:

Sensor Data Access

The AVxcelerate Sensor co-simulation provides access to sensor output data through a configurable data access mechanism. This feature allows other activities in the simulation framework to receive and process sensor data generated by the AVxcelerate Sensor service.

Configuration

Sensor data access can be enabled through the sensor_data_access configuration section in your simulation settings:

Configuration parameters in "sensor_data_access"

Data format

When sensor data access is enabled, the AVxcelerate Sensor activity publishes sensor data to the specified topic in the following format:

• Topic: The topic name specified in publish_topic_name
• Message Type: rtidds::GenericBytesMessage
• Content: Serialized sensor data descriptions containing metadata and data references

Usage in other activities

Other activities in the simulation framework can subscribe to the sensor data topic to receive and process sensor outputs. For example, a visualization activity could subscribe to receive camera images, or a perception activity could receive lidar point clouds.

Example subscription in another activity:

To deserialize the SensorDataDescription from received bytes in the topic, a static method DeserializeSensorDataDescriptions(bytes) is provided in AvxVssApiAdapter class.

Data retrieval

To access the actual sensor data content, activities need to:

1. Subscribe to the sensor data topic to receive data descriptions
2. Use the VSS API to retrieve the actual sensor data using the data IDs from the descriptions
3. Parse the data according to the sensor type and format

Example data retrieval:

Supported sensor types

The sensor data access mechanism supports various sensor types including:

• Camera sensors: Image data with metadata (width, height, pixel format, etc.)
• Lidar sensors: Point cloud data
• Radar sensors: Radar detection data
• Thermal cameras: Temperature map data

Each sensor type provides specific metadata and data formats that can be accessed through the VSS API.

Prerequisites and Setup

Before using AVxcelerate Sensor co-simulation, ensure you have:

1. AVxcelerate Sensor Service: A running AVxcelerate Sensor service accessible via gRPC endpoint
2. Valid License: Proper ANSYS license setup for AVxcelerate Sensor and Simfwk
3. Asset Files: Required AVxcelerate asset files (.asset format) for your simulation objects and resolvable by AVxcelerate Sensor
4. Track Files: Required AVxcelerate Track files (.track format) for your simulation objects and resolvable by AVxcelerate Sensor
5. Sensor Configurations: Valid sensor configuration files (.sencfgx and parameter files, generated by AVxcelerate Sensor Lab)

Network configuration

The AVxcelerate Sensor co-simulation uses gRPC for communication. Ensure:

• The specified GrpcHost and GrpcPort are accessible
• No firewall restrictions block the gRPC connection
• The AVxcelerate Sensor service is running and listening on the specified endpoint

File paths

All file paths in the configuration can be:

• Absolute paths: Full system paths (e.g., /opt/Ansys/AVX_Library_v251/Assets/...)
• Relative paths: Relative to the working directory when running the simulation

Troubleshooting

1. Connection timeout

   • Example of timeout error by start-up connection

     [LogInfo][AVxcelerate Sensor Connection] Waiting for connection to AVxcelerate Sensor Simulation Service at localhost:54321, Time out in 60 seconds

     [LogError][AVxcelerate Sensor Connection] Timeout waiting for gRPC channel connection localhost:54321 after 60 seconds

     [LogError][AvxSensorActivity] Failed to load AVxcelerate Sensor generic_avx_sensor_activity at endpoint: localhost:54321, please check the AVxcelerate Sensor configuration file: co_sim/avx_sensor_configuration/test_configs/valid_config.json and network connection.

     std::runtime_error

     what(): Failed to load AVxcelerate Sensor
   • How to solve:
     • Verify the AVxcelerate Sensor service is running and correct gRPC endpoint is set
     • Check network connectivity and firewall settings
     • Increase the timeout_in_sec value if needed
2. Connection drop

   • If the AVxcelerate Sensor service dies during simulation or the connection is broken, the simulation will be aborted on the Simfwk side and the Unload operation will also likely timeout. In this case, you might have some outputs from both Simfwk and AVxcelerate Sensor from previous successful cycles, but be cautious when using those results. To resolve this issue, restart the AVxcelerate Sensor service or fix the connection problem, then rerun the simulation completely to ensure complete and valid results.

     [LogInfo][Lifycycle] Starting simulation.

     [LogDebug][Lifecycle] Activity groundtruth_generator_activity is used as primary activity for simulation...

     [LogWarn][AvxSensorActivity: avx_sensor_camera_activity] Initializing AVxcelerate Sensor simulation.

     [LogInfo][AVxcelerate Sensor Connection] Waiting for connection to AVxcelerate Sensor Simulation Service at localhost:54321, Time out in 60 seconds

     [LogInfo][AVxcelerate Sensor Connection] Connection to AVxcelerate Sensor Simulation Service at localhost:54321 established

     [LogInfo][AVxcelerate Sensor Connection] Initialize request successful

     [LogInfo][AVxcelerate Sensor Connection] Update request successful

     [LogInfo][AVxcelerate Sensor Connection] Update request successful

     [LogInfo][AVxcelerate Sensor Connection] Update request successful

     [LogInfo][AVxcelerate Sensor Connection] Update request successful

     [LogInfo][AVxcelerate Sensor Connection] Update request successful

     [LogError][AVxcelerate Sensor Connection] gRPC error: Socket closed

     [LogError][AvxSensorActivity: avx_sensor_camera_activity] Failed to call Update AVxcelerate Sensor server.

     [LogWarn][Lifecycle] Activity avx_sensor_camera_activity called shutdown. Simulation will end soon!

     [LogInfo][Lifycycle] Simulation ended.

     [LogWarn][AvxSensorActivity: avx_sensor_camera_activity] Simulation ended, trying to unload AVxcelerate Sensor server...

     [LogInfo][AVxcelerate Sensor Connection] Waiting for connection to AVxcelerate Sensor Simulation Service at localhost:54321, Time out in 60 seconds

     [LogError][AVxcelerate Sensor Connection] Timeout waiting for gRPC channel connection localhost:54321 after 60 seconds

     [LogError][AvxSensorActivity: avx_sensor_camera_activity] Failed to call Unload AVxcelerate Sensor server.
3. Configuration file errors

   • Ensure all required fields are present in the configuration. Example error of missing AvxSensorParameter:

     Executing original terminate handler...terminate called after throwing an instance of 'std::runtime_error'

     what(): [AvxConfigParser] Missing or null required field: AvxSensorParameter

   • Verify file paths exist and are accessible. Example error of missing *.sencfgx file:

     bash Executing original terminate handler...terminate called after throwing an instance of 'std::runtime_error' what(): [AvxConfigParser] File does not exist for AvxSensorConfig: /non-existing/co_sim/avx_sensor_configuration/test_configs/dummy_sensor_config.sencfgx

   • Check JSON syntax validity. Example error in wrong json syntax:

     bash Executing original terminate handler...terminate called after throwing an instance of 'std::runtime_error' what(): [AvxConfigParser] JSON parsing error in file co_sim/avx_sensor_configuration/test_configs/valid_config.json: [json.exception.parse_error.101] parse error at line 12, column 3: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal

4. Track or object mapping issues

   • Confirm OpenScenario object names match exactly. Example error when wrong object name is given:
   [LogInfo][Vss Configuration] AVxcelerate Vehicle Identity: EgoWithWrongName will be used for above open scenario vehicle.

   ...

   [LogError][AVxcelerate Sensor Connection] Failed to call Load in AVxcelerate Sensor Simulation Service: Error on executing LoadCommand: Undefined identifier for egoVehicleIdentity

   std::runtime_error

   what(): Failed to load AVxcelerate Sensor
   • Verify AVxcelerate asset and track files exist and are valid, accessible by AVxcelerate Sensor. Example error when track file is not accessible to AVxcelerate Sensor:
   [LogInfo][Vss Configuration] Set track file: /tmp/track/non-existing.track

   ...

   [LogInfo][AVxcelerate Sensor Connection] Connection to AVxcelerate Sensor Simulation Service at localhost:54321 established

   [LogError][AVxcelerate Sensor Connection] Failed to call Load in AVxcelerate Sensor Simulation Service: Error on executing LoadCommand: Unpacking failed with error : The resource file cannot be found: /tmp/track/non-existing.track

   std::runtime_error

   what(): Failed to load AVxcelerate Sensor

Performance considerations

• Cycle Time: The AVxcelerate Sensor activity inherits its cycle time from its dependencies. Consider the computational requirements of your sensor model when setting cycle times.
• Network Latency: gRPC communication adds latency. For real-time simulations, ensure low network latency between the simulation framework and AVxcelerate Sensor service.
• Resource Usage: AVxcelerate Sensor processing can be computationally intensive. Monitor system resources during simulation.

Limitation

The current implementation of AVxcelerate Sensor co-simulation has the following limitations:

1. OSI to VSS Conversion: The current converter only supports moving objects from OSI SensorView to VSS WorldUpdate. Static objects and pedestrians are not yet supported. This will be improved by next version of VSS API where osi3::SensorView will be used as supported type by AVxcelerate Sensor.
2. VSS API Compatibility: The implementation is based on the current VSS API. Future VSS API changes may require updates to the converter and adapter components.
3. Error Recovery: Limited error recovery mechanisms. If the AVxcelerate Sensor service becomes unavailable, the simulation may need to be restarted.
4. Configuration Validation: File existence validation is limited to sensor configuration files. Track and asset file existence is not validated by Simfwk.

Future enhancements

Planned improvements for AVxcelerate Sensor co-simulation include:

• Advanced Error Handling: Robust error recovery and fallback mechanisms
• Performance Optimization: Improved data conversion and communication efficiency
• Native OSI Support: Direct OSI support in VSS API to eliminate conversion overhead