AMO diagnostics

[marcoaccame]

Description

This PR adds runtime configuration of additional diagnostics useful for debugging services on the robot.

Runtime configuration means that yarprobotinterface collects some parameters from xml files and sends them to the ETH boards so that this one can use them to activate / tune some actions.

So far, only the AMO diagnostics in MC service is supported.

In the following is how the AMO diagnostics can be configured and what it does.

How to activate / deactivate the AMO diagnostics

The AMO diagnostics is disabled by default.

If one wants to activate that, he/she must add the group named DIAGNOSTICS in the xml file which describes the MC service on a ETH board, as in the following.

<group name="SERVICE">

    <param name="type"> eomn_serv_MC_foc </param> 
   
    <group name="PROPERTIES">
    
       <group name="DIAGNOSTICS">
           <param name="mode">     eomn_serv_diagn_mode_MC_AMO         </param>
           <param name="par16">    100                                 </param>
       </group> 
       
        ...

Code list. The DIAGNOSTICS group has two parameters: mode equal to eomn_serv_diagn_mode_MC_AMO activates the AMO diagnostics with a maximum reporting rate in ms expressed by par16. In this case 100 ms. The minimum value is 2, meaning that the board can potentially emit diagnostics every 2 ms.

The service can be disable also by using a special mode called eomn_serv_diagn_mode_NONE as in the following.

<group name="SERVICE">

    <param name="type"> eomn_serv_MC_foc </param> 
   
    <group name="PROPERTIES">
    
       <group name="DIAGNOSTICS">
           <param name="mode">     eomn_serv_diagn_mode_NONE           </param>
           <param name="par16">    100                                 </param>
       </group> 
       
        ...

Code list. The diagnostics is disabled if the group DIAGNOSTICS is not present or if it has mode equal to eomn_serv_diagn_mode_NONE .

​

What the AMO diagnostics produces

When the AMO diagnostics is enabled with a maximum rate, let's say equal 100 ms, the board evaluates every 2 ms the content of the two status registers of the attached AMO encoders and it produces a stream of diagnostic messages which go to the yarplogger. See the following listing.

4375,461389 <ERROR>  from BOARD 10.0.1.6 (left_leg-eb6-j0_1), src LOCAL, adr 0,
time 159s 59m 740u: (code 0x04000005, par16 0x0800 par64 0x0009ff610000030c) 
-> DEBUG: tag05 + . 
 
4375,461530 <ERROR>  from BOARD 10.0.1.6 (left_leg-eb6-j0_1), src LOCAL, adr 0, 
time 159s 59m 831u: (code 0x04000006, par16 0x0027 par64 0x000003930000038b) 
-> DEBUG: tag06 + . 
 
4375,471472 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0, 
time 159s 75m 549u: (code 0x04000005, par16 0x0800 par64 0x000c5d17fffffcd6) 
-> DEBUG: tag05 + . 
 
4375,481599 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0, 
time 159s 75m 642u: (code 0x04000006, par16 0x0032 par64 0xfffffb9bffffff9f) 
-> DEBUG: tag06 + . 
 
4375,562310 <ERROR>  from BOARD 10.0.1.6 (left_leg-eb6-j0_1), src LOCAL, adr 0, 
time 159s 159m 740u: (code 0x04000005, par16 0x0800 par64 0x0009ff6100000324) 
-> DEBUG: tag05 + . 

4375,562449 <ERROR>  from BOARD 10.0.1.6 (left_leg-eb6-j0_1), src LOCAL, adr 0, 
time 159s 159m 833u: (code 0x04000006, par16 0x0023 par64 0x000003910000038b) 
-> DEBUG: tag06 + . 

4375,572393 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0, 
time 159s 175m 550u: (code 0x04000005, par16 0x0800 par64 0x000c5d15fffffc2d) 
-> DEBUG: tag05 + . 

4375,572534 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0,
time 159s 175m 643u: (code 0x04000006, par16 0x0032 par64 0xfffffb9bffffff9f)
-> DEBUG: tag06 + . 

Log excerpt. There are two boards, the left_leg-eb6-j0_1 and the left_leg-eb7-j2_3 which signals errors at their maximum rate of 100 ms. The pair of diagnostic messages emitted by the board have code 0x04000005 and 0x040000056 and are recognized by -> DEBUG: tag05 + . and -> DEBUG: tag06 + .

The information inside the log messages

If the board has an AMO encoder and the reading of its two status registers shows any error flags, then it emits a pair of diagnostics messages which contains the values of:

• the value of the status registers of the AMO as read by hal_spiencoder_get_value2(),
• the joint number which uses the AMO,
• the number of times the status registers were in error since the previous diagnostic messages,
• some additional information such:
  • the raw value of the AMO encoder as read by hal_spiencoder_get_value2(),
  • the position of the joint as processed by the MController (eOmc_joint_status_t::core.measures.meas_position),
  • the PWM applied by the MController (eOmc_motor_status_t::basic.mot_pwm),
  • the motor current measured by the actuator eOmc_motor_status_t::basic.mot_current)

The pair of diagnostics messages are emitted with a maximum rate (configurable by xml file) to avoid to flood the log with continuous messages.

The above values are enclosed inside the fields of the diagnostic messages as later described. First I show an example of such messages with a maximum rate of 100 ms.

4375,461389 <ERROR>  from BOARD 10.0.1.6 (left_leg-eb6-j0_1), src LOCAL, adr 0,
time 159s 59m 740u: (code 0x04000005, par16 0x0800 par64 0x0009ff610000030c) 
-> DEBUG: tag05 + . 
 
4375,461530 <ERROR>  from BOARD 10.0.1.6 (left_leg-eb6-j0_1), src LOCAL, adr 0, 
time 159s 59m 831u: (code 0x04000006, par16 0x0027 par64 0x000003930000038b) 
-> DEBUG: tag06 + . 
 
4375,471472 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0, 
time 159s 75m 549u: (code 0x04000005, par16 0x0800 par64 0x000c5d17fffffcd6) 
-> DEBUG: tag05 + . 
 
4375,481599 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0, 
time 159s 75m 642u: (code 0x04000006, par16 0x0032 par64 0xfffffb9bffffff9f) 
-> DEBUG: tag06 + . 
 
4375,562310 <ERROR>  from BOARD 10.0.1.6 (left_leg-eb6-j0_1), src LOCAL, adr 0, 
time 159s 159m 740u: (code 0x04000005, par16 0x0800 par64 0x0009ff6100000324) 
-> DEBUG: tag05 + . 

4375,562449 <ERROR>  from BOARD 10.0.1.6 (left_leg-eb6-j0_1), src LOCAL, adr 0, 
time 159s 159m 833u: (code 0x04000006, par16 0x0023 par64 0x000003910000038b) 
-> DEBUG: tag06 + . 

4375,572393 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0, 
time 159s 175m 550u: (code 0x04000005, par16 0x0800 par64 0x000c5d15fffffc2d) 
-> DEBUG: tag05 + . 

4375,572534 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0,
time 159s 175m 643u: (code 0x04000006, par16 0x0032 par64 0xfffffb9bffffff9f)
-> DEBUG: tag06 + . 

Log excerpt. There are two boards, the left_leg-eb6-j0_1 and the left_leg-eb7-j2_3 which signals errors at their maximum rate of 100 ms. The pair of diagnostic messages emitted by the board have code 0x04000005 and 0x040000056 and are recognized by -> DEBUG: tag05 + . and -> DEBUG: tag06 + .

How the information is stored inside the two messages

An example of message pairs is in the following

4375,572393 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0, 
time 159s 175m 550u: (code 0x04000005, par16 0x0800 par64 0x000c5d15fffffc2d) 
-> DEBUG: tag05 + . 

4375,572534 <ERROR>  from BOARD 10.0.1.7 (left_leg-eb7-j2_3), src LOCAL, adr 0,
time 159s 175m 643u: (code 0x04000006, par16 0x0032 par64 0xfffffb9bffffff9f)
-> DEBUG: tag06 + . 

Value	place inside message	Value in above list
AMO status registers status0, status1	par16 of DEBUG: tag05, where status0 is the least significant byte and status1 the most significant byte	par16 = 0x0800, status0 = 0x00, status1 = 0x08
The joint number inside the board	adr of DEBUG: tag06 or DEBUG: tag06	adr = 0, thus the first joint that is left_leg_j2
The number of times the board found errors since previous messages	par16 of DEBUG: tag06.	par16 = 0x0032 = 50, hence 50 times, hence always. See Note 1.
AMO raw value	the most significant four bytes of par64 of DEBUG: tag05	par64 = 0x000c5d15fffffc2d, value = 0x000c5d15 = 810261
Motor current	the least significant four bytes of par64 of DEBUG: tag05	par64 = 0x000c5d15fffffc2d, value = 0xfffffc2d = -979
Motor PWM	the most significant four bytes of par64 of DEBUG: tag06	par64 = 0xfffffb9bffffff9f, value = 0xfffffb9b = -1125
Motor current	the least significant four bytes of par64 of DEBUG: tag06	par64 = 0xfffffb9bffffff9f, value = 0xffffff9f = -97
Time of ETH board	written explicitly in message	time 159s 175m 550u
Time of yarplogger	the first number of the row	4375,572393 which is 4375 s 572393 us

Table. Mapping of the values inside the two diagnostics messages

Note 1. The ETH boards signals errors at most every 2 ms because it reads register status0 and status1 at each millisecond. So, if between two pair of messages which are 100 ms far apart we see that the number of errors have been 50, it means that every time we checked we found an error, hence 100 % of the times.

Requirements of this PR

It needs also similar PRs:

• in icub-firmware-shared which contains the data structures used for diagnostic s configuration,
• in icub-main which contains code to extract configuration from the xml files.
• in icub-firmware-build to give the new binaries for the ems, mc4plus, mc2plus.

So far we omit any PR in robots-configuration.

Executed tests

Tests were done extensively and with success on a dedicated setup and on iCubGenova09.

[pattacini]

cc @S-Dafarra