wiki:Tutorials/Intermediate/Data Recording

Data Recording

A convenient data recording mechanism is available in Finroc's data_recording plugin.

It can record data to different sinks of different types (so-called backends - currently binary files and text files suitable for e.g. gnuplot). Further backends can be added to rrlib_data_recording as needed.

Notably, new port values are recorded immediately (with a precise timestamp) whenever there is a change - not cyclically. This decision was taken so that it is possible to analyze the exact timings in a system.

As a consequence, records typically contain only a single value. Tool support for converting and resampling recorded data is work in progress.

Setup your Finroc environment

Simply check out the data_recording plugin. Running make clean is currently mandatory in order to record output ports.

finroc_get finroc_plugins_data_recording
make clean

Recording data from a Finroc application

In order to record data from an arbitrary Finroc application, it is sufficient to add a few entries to the plugin configuration section of its main config file, e.g.:

<?xml version="1.0" encoding="UTF-8"?>
<root>

  <!-- Plugin configuration section -->
  <runtime>

    <!-- You might need that for configuring compression with algorithms not known at program startup -->
    <load_library name="rrlib_coviroa_opencv"/>

    <plugin name="data_recording">

      <!-- Example for recording to a binary file with JPEG compression for images (start_paused is optional and false by default) -->
      <sink type="binary" file="recording-${TIMESTAMP}.bin" start_paused="true">
        <port>/Forklift/Profiling/Execution Duration</port>
        <regex>/Test/.*</regex>
        <compression data_type="rrlib.coviroa.Image" format="jpeg"/>
      </sink>

      <!-- Example for recording to gnuplot backend -->
      <sink type="gnuplot" dir="/home/max/tmp/recording-${TIMESTAMP}">
        <regex>/Forklift/Simulation/Scene &amp; Physics/Sensor Output/.*</regex>
        <port>/Forklift/Profiling/Execution Duration</port>
      </sink>

    </plugin>
  </runtime>
  
  <!-- Main section with configuration nodes and values -->
  <value name= ...

Generally, it is possible to create an arbitrary number of sinks and attach an arbitrary number of ports to them. Which ports to record can be either specified using its full qualified name in port tags, or via regular expressions in regex tags. Regular expressions are passed to std::regex and are matched against port's qualified names. Notably, matching ports that are created or replaced later will all be recorded.

Notes:

  • ${TIMESTAMP} in a file or directory name will be replaced with the ISO 8601 timestamp at program startup.
  • Notably, not all backends support all data types. You will get a warning message if a port cannot be recorded to a specific backend.
  • Compression can be configured with rules. Apart from data_type, there are also attributes name and regex for qualified name matching.
  • Boolean ports will be created in /Runtime/Services/Record '<file or dir name>' . They can be used start and pause recording to specific sinks.

Binary Backend

This backend will record all ports to a single file using Finroc's binary serialization. This works with all data types used in ports and is quite efficient.

Notably, it is not possible to read data entries from such files whose data type's binary serialization has changed since recording that file (which should not happen often - as different Finroc releases should remain interoperable). Still, it might not be the best format for archiving data.

gnuplot Backend

This backend will create a separate text file for each recorded port in the specified directory. It is suitable for numbers and most types from rrlib_math. Values are separated with spaces. The first column are seconds since the application was started.

This .gnuplot file is a simple example how plots can then be drawn using gnuplot:

set term wxt persist
set grid
set xlabel "Time [s]"
set ylabel "Value"
plot \
'Biped.PhysicsEngineNewton.Physics_Simulation.Sensor_Output.spine_z_-_Torque' using 1:2 with lines, \
'Biped.PhysicsEngineNewton.Physics_Simulation.Sensor_Output.foot_right_-_Pose' using 1:7 with lines

Setup data recording from application code

Instead of setting up recording from the config file, it is also possible and somewhat more flexible to do this in the code.

It is out of scope to go into details here - so here is just a basic example for parts:

  • You need to include the plugin and the desired backend(s)
#include "plugins/data_recording/tDataRecording.h"
#include "rrlib/data_recording/backend/impl/gnuplot/tBackend.h"
  • Typically, you will create a (e.g. global) variable for your sink. It needs to exist until recording is stopped.
rrlib::data_recording::backend::impl::gnuplot::tBackend recording_sink("/home/username/gnuplot", false);
  • In CreateMainGroup(), you then add lines to open the sink, select ports, and record them - e.g.:
  recording_sink.OpenWrite();
  finroc::data_recording::tPortSetRecordingControl recording(finroc::data_recording::tRegex("/Biped/PhysicsEngineNewton/Physics Simulation/.*"));
  recording.Record(recording_sink);
  • Via the tRecordingControl.h interface, recording can be paused, resumed etc.

Status

Known Issues

Planned Features

  • Conversion and resampling of recorded files
  • More backends (e.g. XML, database)
  • Graphical tooling to inspect recorded data
Last modified 4 years ago Last modified on 14.04.2015 18:02:34