[FR] Timestamp, Retain messages, Filter by name/prefix

[ilgrank]

Hi
Again, thanks for OpenMQTTGateway!
I'm using it since some day on an ESP32 and.. it's great :)

I have some ideas to propose, that (IMHO of course) could be of interest:

• Timestamp messages: Since we configure an NTP anyways, it would nice to have the option to timestamp messages. E.g.: I have several types of sensors, some which report every few seconds and others that report every few hours: it would be useful to know when a message was received..
• Retain messages: The scenario above requires the ability to retain messages. I've read previous discussions questioning the usefulness of this, but I have a new argument to support my request: in my case, not having the option to retain messages messages means that any new client subscribing to my broker (I'm not using HA or any other sofware, just JS clients connecting to the broker) needs to wait for a sensor to advertise new data.. and it could be second or dozens of minutes. Having the last reading as a retained message would mean knowing immediately the latest known sensor reading.

I've already a work-around in place for the points above (another client that is subscribed to OMQTTG topics and re-publishes them with timestamps and retain flag), but I think it would be more practical to have these features built-in (possibly without re-compiling)

• Device filtering: There are (alas) dozens of BLE devices around me it seems. While I've filtered by MAC Address whitelist (thanks!) but it seems to me that adding a filtering by name/prefix would prove to be more flexible.

As always, thanks for your support!

[1technophile]

Hello,

Timestamp messages: Since we configure an NTP anyways, it would nice to have the option to timestamp messages. E.g.: I have several types of sensors, some which report every few seconds and others that report every few hours: it would be useful to know when a message was received..

This can be managed on the controller side, what do you think would be the added value of adding a timestamp at the firmware level versus adding it when the controller receives it?

[ilgrank]

pardon me.. when you say "on the controller side", you mean on the devices connecting to the broker that subscribe to the sensors topics? (these are offline most of the time)
or you mean on the broker itself? if that's the case, I've talked some time ago with the Mosquitto devs, and they're against the broker altering the message payload in any way. (discussion here )

(on a side note, there's a project similar in scope to OMQTTG, Zigbee2mqtt, has a last_seen option that does what the name suggests.. just discovered that while I was searching for my FR to Mosquitto devs)

[1technophile]

By Controller, I meant HA, OpenHAB, NodeRed...

[ilgrank]

oh, well, that's because I don't use any, as I wrote in my initial post:

I'm not using HA or any other sofware, just JS clients connecting to the broker)

:)
full story:
I have a few low-power devices in remote locations which are powered by solar panels (a raspberry zero running mosquitto being one of the most power-hungry)
On these system, there's simply no option to run HA, NodeRed or any other resource-heavy software.
"Clients" are actually web browsers (connecting directly to the broker via websockets) that check the data in real time when needed

So, the advantage is that this solution is very lightweight, does not add complexity and above all, this is an information which directly pertains to the sensors (see my comment about Zigbee2Mqtt in the edit of previous message (Sorry, I was editing it while you were replying))

[1technophile]

Ok I understand now your need, thanks for sharing

[ilgrank]

Hi
researched around, and found out that It's quite common to have retain-able sensors topics, so I've gathered some 'evidence' that it could be useful to more people than just me :)

Tasmota explicitly supports it with a dedicated command: SensorRetain

Shelly devices also support retaining, as documented here

and finally, Steves' internet Guide had an excellent explanation on why having the Retained flag set is desirable on sensors, here:

[...]this can be very useful, as for example, if you have sensor publishing its status only when changed e.g. Door sensor. What happens if a new subscriber subscribes to this status?
Without retained messages the subscriber would have to wait for the status to change before it received a message.
However with the retained message the subscriber would see the current state of the sensor

Since we're talking about devices that are not MQTT enabled by themselves here, OMQTTG is the only entry point for such devices to have their info published in real time, and without requiring complex and resource-hungry controller systems.

Sorry coming again at this, and again, thanks :)

[1technophile]

Having the capability to retain messages would be an easy add.

[ilgrank]

thanks @1technophile , high hopes for that.. and also timestamping would be great, if possible.
All of the above sensors (Shelly, Tasmota) do timestamp their message, if possible (ie: not running on battery).
eg:, a sample tasmota payload:

{
  "Time": "2023-01-28T22:55:50",
  "ENERGY": {
    "TotalStartTime": "2020-04-10T18:33:07",
    "Total": 0.408,
    "Yesterday": 0,
    "Today": 0,
    "Period": 0,
    "Power": 0,
    "ApparentPower": 0,
    "ReactivePower": 0,
    "Factor": 0,
    "Voltage": 0,
    "Current": 0
  }
}

and a shelly one:

{
   "wifi_sta":{
      "connected":true,
      "ssid":"RemoteLoc1",
      "ip":"192.168.0.13",
      "rssi":-71
   },
   "cloud":{
      "enabled":false,
      "connected":false
   },
   "mqtt":{
      "connected":true
   },
   "time":"00:54",
   "unixtime":1675036463,
   "serial":1,
   "has_update":false,
   "mac":"2CF432xxxxxx",
   "cfg_changed_cnt":0,
   "actions_stats":{
      "skipped":0
   }
}

Thanks :)

[ilgrank]

Apart from MAC filtering by prefix, everything has been added in 1.5.0, thanks!