Timestamps for retained messages

[itspa1]

Feature Request:
It would be good to have timestamps in the retained messages being sent since a retained WILL message has no option of sending exact timestamp cause it would be set during the connection.

[ralight]

I'm afraid that I don't think this is appropriate for inclusion. Even as an option, it would be impossible to cater for formatting the message in all the ways that people might want.

[itspa1]

Hey, Thanks for the reply, yes indeed it would be impossible to cater for formatting the message in all the ways but, the basic idea behind the request for a timestamp in the retained messages is it would be better to get the arrival time if it has a timestamp being set either by the publisher and if not, the broker could add a UNIX timestamp to the payload so that it would be easy for the subscriber to make sense if it is a stale/old-retained message or indeed a new one. also, this feature request is not just on the fly but some brokers like flespi provide this functionality. so, was hoping this would be a good addition to the most used open source broker.

[ratara]

@Impeekay
Isn't it the idea of retained messages that they contain valid data no matter when they were published? What would that scenario look like where you need the publish time stamp to verify? - If you are looking for something that needs an alive information, an event(not retained message) could be a more direct approach.

[itspa1]

@ratara yes absolutely right, if something that needs alive information, an event might be the best approach but, certain use cases would go a step far and would need timestamps sometimes, or at least in my use case tho. Hence the request so that if at least added optionally, many use cases requiring this might be taken care of. and some brokers "flespi" might have added it for a reason. again as said earlier this would be a good addition to the feature list

[ilgrank]

+1 for this request.
I have battery-powered sensors that report temperature only when it changes over a defined threshold. (and publish a retained message when this happens)
In this way, I can query the broker and know the value of the last reading.
Alas, I don't know when it was published.
Most of the times the temperature hasn't changed and so the last reading is still valid, but the battery could be dead and the retained message could be weeks old. No way of knowing. (and these devices do not set a LWT by the way)

[ralight]

The most compliant way to solve this would be to use MQTT v5 user-property properties.

To simulate this you could use:

mosquitto_pub -t topic -m message -r -V 5 -D publish user-property timestamp "yesterday"

[itspa1]

mosquitto_pub -t topic -m message -r -V 5 -D publish user-property timestamp "yesterday"

@ralight here yesterday means adding a new timestamp on our own, or can it be a stream of new timestamps which will be generated on its own?

[ilgrank]

The most compliant way to solve this would be to use MQTT v5 user-property properties.

To simulate this you could use:

mosquitto_pub -t topic -m message -r -V 5 -D publish user-property timestamp "yesterday"

This if you can alter the behaviour of the devices that are publishing.
My IoT devices use MQTT v3 and are closed-source, so I can't alter their posts.

[ralight]

@Impeekay That would be a per-message timestamp generated by the publishing client in whatever format is desired for that particular project. This is exactly what the user-property property is for, adding in application specific information.

[ralight]

@ilgrank Would you be happy with a solution that allowed you to attach a user property to messages at the broker, so your subscriber would have to support MQTT v5, but your publishers would go untouched, and no modification of the payload would be required?

[ilgrank]

@ralight
I do currently use Paho javascript client (I'm developing a web-app) that according to the docs
https://www.eclipse.org/paho/clients/js/
does not support MQTT 5 alas :(

[itspa1]

@ralight thanks, this actually seems like a suitable workaround. But, it would require all the clients that I currently have to be ensured that they send out the user-property of timestamps.

[Sumatt92]

Could you run an application on the same host as the broker that uses a client library to subscribe to the topic and append the timestamp before publishing to a different topic?
Like if original topic is /mysensor1/temp , client application subscribes to /mysensor1/temp, appends timestamp to payload or reforms as mqttv5 with user property and publishes to /mysensor1/tempandtime ?

[ilgrank]

@Sumatt92 I've already a workaround in place: for battery powered sensors a client inserts the values (with timestamp) in a database.. but it is a clunky workaround. As it is the one you're suggesting :)

[ilgrank]

@ralight :any hope to have it as an MQTT v3 / 3.1 solution (as PAHO JS doesn't support currently MQTT v5)?

[ckrey]

You may include the timestamp in the payload as CSV: ‘1591979245,25.9’ or JSON: ‘{“tst”:1591979245,”temp”:25.9}'

…

On 11. Jun 2020, at 18:29, ilgrank ***@***.***> wrote: @ralight <https://github.com/ralight> :any hope to have it as an MQTT v3 / 3.1 solution (as PAHO JS doesn't support currently MQTT v5)? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1225 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6XHGN5P322A377WACXMV3RWEA6NANCNFSM4HE3TEIA>.

[ilgrank]

@ckrey : I really don't understand how, since the payload is generated by a closed-source device whose messages I can't alter

[ralight]

@ilgrank I don't really want to have the broker modifying the payload, it's a bit of a minefield for configuration. Probably the best way to do this would be to have a plugin type interface where you could write some code to modify the payload to your requirements. Would that work?

[ilgrank]

@ralight this seems a great idea to me, thanks!

[rcomellas]

+1 for this request
Normally our sensors in the field don't have a reference of time (RTC, NTC) so it would be very useful that the server kept the timestamp of the moment it receives a published message

[thg2k]

I also think it is a missing feature in the MQTT spec, adding timestamps in the payload is plainly wrong, also you might not have an accurate clock on the device. The broker should timestamp retained messages, user property would be the way to go until they fix the MQTT spec.

[Sumatt92]

I believe that the payload is and should remain a black box to the broker. Modifying the payload with the broker breaks the network layer model. The client application(s) are the producers and consumers of the payload, while the broker simply routes the packets from a producing client to a consuming client.
This may seem like a snooty response but I think there is a lot of value in maintaining a simple operating model for the broker where its concern is only routing a payload from publisher to subscriber(s).

I also don't feel like using a second client to publish a modified payload is a hack. If you had two temp sensors in a room, you probably wouldn't consider it a hack to have a client application on the broker machine subscribe to both sensor's topics, average the two and publish it back under another topic.

[ilgrank]

@ralight in fact was proposing a plugin to do that.
I do agree that it is not the broker's job to do alter a message, but in my scenario, the clients are closed source and do not have (being battery-powered devices) NTP functionality.
Adding a subscriber that merely reads those topics and re-publishes them with an added timestamp would effectively double the network traffic and severely decrease the equipment lifetime (Being the broker too battery powered).

To clarify: I have several IoT sensors, and a raspberry zero wich does the 'heavy lift' of collecting and storing the sensor data. Sensors are battery-powered, while the Raspberry is Solar+Battery powered. There are constranints on the equipment size, so I can't use larger batteries or solar panels. Mosquitto was chosen (among other softwares) because of its lightweight operation.
The 'plugin' scenario would allow genral users to have a very clean MQTT brocker, while allowing rare case scenarios like mine to have the opportunity to have a solution (please pardon my english, it's not my native language)
Thanks

[karlp]

Why would it double the network traffic?!

[ilgrank]

@karlp: because the listening client would have to re-publish every message that is published.

[karlp]

but that doesn't have to go over any network... that can be just looped back at the point that does have time. you don't need to send untimestamped and timestamped over your network. (but this is OT for this ticket now really)

[ilgrank]

..anyway, as you can see this is a much requested feature: in this issue alone there are at least 3 people requesting it.
Many more such requests can be found by simple googling, such as:
https://www.eclipse.org/lists/mosquitto-dev/msg00903.html

[ralight]

I've just pushed an example plugin which does this to the develop branch, let me know what you think.

You'll need the develop branch version of Mosquitto, and to compile the plugin in plugins/message-timestamp.

[ilgrank]

Thanks @ralight !
sorry, I'm not that an advanced user.. Is there a guide on how to compile the plugin?
Thanks!

[ralight]

If you're compiling directly from source, then it should only be using libraries that you have installed and available. The message you have suggests it was compiled against glibc 2.29. I have come across this before when cross compiling and it is possible to force the linker to use a particular glibc version for a symbol, but I think the easiest thing for you to try is compile with:

make clean
make WITH_SYS_TREE=no

That should stop the maths function being used, and hence the error should disappear.

[ilgrank]

@ralight You're right, I cross-compiled on Ubuntu 20 since my distro (DietPi 6.30) compile was erroring out:

make: Entering directory '/temp_mosquitto/mosquitto-with-plugin/src'
cc  -I. -I.. -I../include -I../../include -I../src/deps -I../lib -DWITH_TLS -DWITH_TLS_PSK -DWITH_BRIDGE -DWITH_PERSISTENCE -DWITH_MEMORY_TRACKING -DWITH_SYS_TREE -DWITH_EC -DWITH_CONTROL -DWITH_UNIX_SOCKETS -DWITH_EPOLL -Ideps -Wall -ggdb -O2 -DVERSION="\"2.0-dev0\"" -DWITH_BROKER -c mosquitto.c -o mosquitto.o
mosquitto.c:21:12: fatal error: unistd.h: No such file or directory
 #  include <unistd.h>
            ^~~~~~~~~~
compilation terminated.
make: *** [Makefile:89: mosquitto.o] Error 1
make: Leaving directory '/temp_mosquitto/mosquitto-with-plugin/src'

(yes, there's no unistd.h in this debian-based distro)
I've re-compiled with the options you suggested, replacing just my current (1.68) binary with the freshly compiled one.

After several errors (I had to enable Websockets) I still have this problem
mosquitto: error while loading shared libraries: libwebsockets.so.15: cannot open shared object file: No such file or directory
even if I compiled with:
WITH_STATIC_LIBRARIES:=yes
I'm missing something?
thanks for your kind support

[habazut]

I am testing this as well but where do I see the timestamp then?

1602529417: Loading config file /etc/mosquitto/conf.d/auth.conf
1602529417: Loading config file /etc/mosquitto/conf.d/network.conf
1602529417: Loading config file /etc/mosquitto/conf.d/plugins.conf
1602529417: mosquitto version 2.0-dev0 starting
1602529417: Config loaded from /etc/mosquitto/mosquitto.conf.
1602529417: Loading plugin: /usr/local/lib/mosquitto_timestamp.so
1602529417: Opening ipv4 listen socket on port 1883.
1602529417: mosquitto version 2.0-dev0 running
...
1602530340: New connection from 192.168.0.83 on port 1883.
1602530340: New client connected from 192.168.0.83 as esp8266-6da003 (p2, c1, k15).
1602530340: No will message specified.
1602530340: Sending CONNACK to esp8266-6da003 (0, 0)
1602530350: Received PUBLISH from esp8266-6da003 (d0, q1, r1, m1, 'tingfast45/tankESP/temperature/0', ... (5 bytes))
1602530350: Sending PUBACK to esp8266-6da003 (m1, rc0)

$ /usr/bin/mosquitto_sub -h 192.168.0.17 -t 'tingfast45/tankESP/+/0' -F "%j"
{"tst":1602530664,"topic":"tingfast45/tankESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"23.31"}

Sorry if this is a stupid question, but I am new to MQTT.

Harald.

[ilgrank]

@habazut : isn't
{"tst":1602530664
the timestamp you're looking for? :)
(in unix time format, that is)

date -d @1602530664
Mon 12 Oct 21:24:24 CEST 2020

[ralight]

@ilgrank I presume you've installed the libwebsockets library on your system. You probably need to update the runtime linker cache, which you would typically do by running ldconfig.

[habazut]

@habazut : isn't
{"tst":1602530664
the timestamp you're looking for? :)
(in unix time format, that is)

Nope. That's just "now" and not when the message came in. In addition to that, the plugin adds "timestamp" as string and not "tst" as seconds-since-epoch.

   clock_gettime(CLOCK_REALTIME, &ts);
    ti = gmtime(&ts.tv_sec);
    strftime(time_buf, sizeof(time_buf), "%Y-%m-%dT%H:%M:%SZ", ti);

    return mosquitto_property_add_string_pair(&ed->properties, MQTT_PROP_USER_PROPERTY, "timestamp", time_buf);

and btw, I know that this fridgeESP value is from yesterday and the tankESP is from today:

$ date -u +%s
1602535047
$ /usr/bin/mosquitto_sub -h 192.168.0.17 -t 'tingfast45/+/+/0' -F "%j"
{"tst":1602535051,"topic":"tingfast45/fridgeESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"27.50"}
{"tst":1602535051,"topic":"tingfast45/tankESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"23.13"}

So I had hoped that the new value of tankESP would have the timestamp added when it comes in, but nope.

1602535203: Received PUBLISH from esp8266-6da003 (d0, q1, r1, m2, 'tingfast45/tankESP/temperature/1', ... (5 bytes))
1602535229: Sending PUBLISH to mosqsub|13178-habitat (d0, q0, r1, m0, 'tingfast45/tankESP/temperature/0', ... (5 bytes))
{"tst":1602535229,"topic":"tingfast45/tankESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"23.19"}

So the tst is when the mosquitto broker did send it out. But I am looking for the value added by the plugin when the message came in.

Is there some more thing that must be configured than

# cat /etc/mosquitto/conf.d/plugins.conf 
plugin /usr/local/lib/mosquitto_timestamp.so

and I can see that the plugin is loaded from the log?

Thanks,
Harald.

[ralight]

@habazut This only works with MQTT v5, because it adds a user-property to the message. You should add -V 5 to the mosquitto_sub command:

mosquitto_sub -t \# -V 5 -F "%j"

This would give you an output like this:

{"tst":"2020-10-12T21:54:21Z","topic":"topic","qos":0,"retain":0,"payloadlen":7,"properties":{"user-properties":{"timestamp":"2020-10-12T20:54:21Z"}},"payload":"message"}

In your own client you would access the properties from the on_message callback.

[habazut]

Thanks, the -V5 did show the extra data where it was present (tank and not fridge, see below).

$ mosquitto_sub -V 5 -h 192.168.0.17 -t 'tingfast45/+/+/0' -F "%j"
{"tst":"2020-10-12T23:29:26.082246Z+0200","topic":"tingfast45/fridgeESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"27.50"}
{"tst":"2020-10-12T23:29:26.082545Z+0200","topic":"tingfast45/tankESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"properties":{"user-properties":{"timestamp":"2020-10-12T21:10:23Z"}},"payload":"23.25"}

Here one can see as well how the timestamp value != tst value.

Harald.

[ilgrank]

@ilgrank I presume you've installed the libwebsockets library on your system. You probably need to update the runtime linker cache, which you would typically do by running ldconfig.

@ralight I have libwebsockets installed in the system where I compiled mosquitto, but not on the system where mosquitto will run.
Having statically linked libraries shouldn't allow for this? (my current mosquitto 1.68 runs websockets without having libwebsockets after all)
I also tried

sudo ldconfig
make clean
make -C src

and ldconfig lists libwebsocket.
but I get the same error:
mosquitto: error while loading shared libraries: libwebsockets.so.15: cannot open shared object file: No such file or directory
by compiling with WITH_STATIC_LIBRARIES:=yes shouldn't these libraries be included in the executable?

[ralight]

@ilgrank Sorry I missed that part. I'm afraid that WITH_STATIC_LIBRARIES:=yes just builds the libmosquitto.a static library, nothing more.

You could try just compiling the broker: make -C src

[ilgrank]

@ralight : Thanks, there's one thing I still don't understand:
The mosquitto 1.68 I have downloaded from the semi-official repo does come with websockets support, and does so without any external library.
What flags/configs I need to compile a mosquitto binary that has the same capabilities? (ie: websockets support without the need of an external library)?
Thanks

[ralight]

@ilgrank The packages from Debian direct, and those from repo.mosquitto.org both rely on the libwebsockets package provided by Debian. In other words they do have an external library.

[ralight]

If you only have a static libwebsockets.a file on your system and no libwebsockets.so then I believe the linker will just use the static library without any other changes needed.

[ilgrank]

@ralight : so I really can't explain this:

root@DietPi:/usr/sbin# mosquitto_200
mosquitto_200: error while loading shared libraries: libwebsockets.so.15: cannot open shared object file: No such file or directory
root@DietPi:/usr/sbin# mosquitto_168
1602580715: mosquitto version 1.6.8 starting
1602580715: Using default config.
1602580715: Opening ipv4 listen socket on port 1883.
1602580715: Opening ipv6 listen socket on port 1883.

Update (just seen your second answer): on the building computer I have:

/usr/lib/x86_64-linux-gnu/libwebsockets.a
/usr/lib/x86_64-linux-gnu/libwebsockets.so
/usr/lib/x86_64-linux-gnu/libwebsockets.so.15

please pardon me, I'm really new at building mosquitto, and most likely I'm just missing some basic thing.
I've googled since yesterday in order to understand how the binary in the repo was built (what make flags, what config.mk and so on)

[ralight]

Don't worry about it. So both your building device and the dietpi device are x86_64 architecture? The difficulty you're having seems to be the mismatch between what you have on the building device and on the target device.

[ralight]

You can see which libraries an executable is linked against with the ldd command: ldd mosquitto_200 vs ldd mosquitto_168.

[ilgrank]

@ralight :yep, both x64 VMs. But I finally managed to have libwebsockets-dev installed on the 'smaller' VM, and now it compiles fine.
Thanks also for

You can see which libraries an executable is linked against with the ldd command: ldd mosquitto_200 vs ldd mosquitto_168.

The problem was that 1.6.8 used libwebsockets.so.8 (which was part of the distro it seems) while 2.0.0 used libwebsockets.so.15 which I had to install.
Problem is, It still doesn't seems to 'like' the websockets port, or so it seems:

mosquitto -v
1602581707: mosquitto version 2.0-dev0 starting
1602581707: Using default config.
1602581707: Opening ipv4 listen socket on port 1883.
1602581707: Opening ipv6 listen socket on port 1883.
1602581707: Error: Cannot assign requested address

alas, there's nothing in mosquitto.log about what address can't be assigned.. where else I could look for clues?

[ralight]

This is valuable testing, thank you. One of the changes for 2.0 is that if you run without a configuration then the broker will only listen on the loopback interface, i.e. it will only be accessible on the local machine. So what that log shows is that it has opened an ipv4 listener on port 1883 successfully, and attached it to 127.0.0.1. It then attempts to open an ipv6 listener as well, but that fails to be bound to an address. That presumably means it couldn't attach to the ipv6 loopback ::1. From my reading, the error won't actually cause the broker to stop, is that the case for you?

[ilgrank]

thanks @ralight , your reply sparked a light!
I don't know why, but while 1.6.8 was using /etc/mosquitto/mosquitto.conf by default,i had do be explicit with 2.0.0 and point it to the conf file.
About why it won't bind to IPV6, that's likely because I've just IPV4 configured, so no error on the 2.0 branch on that
Now no error is shown now, and I've been able to subscribe and post topics in V3.1 mode.
I'm now eager to switch to V5, but I fear it will be a show-stopper for me since my client (Paho JS library) does not support MQTT5
https://www.eclipse.org/paho/clients/js/
as I wrote in my message of June 10.
Updating soon! Thanks again

[habazut]

Your problem with linking is probably because WITH_STATIC_LIBRARIES does build the static libraries but not necessarily with them. Say you have
/whatever/libfoo.a
/whatever/libfoo.1.so
installed, then when linking with -L/whatever -lfoo the linker will by default choose the dynamic one. If you really really want to make sure you link in the static lib, you can in the link command replace -L/whatever -lfoo with /whatever/libfoo.a and it can only link these objects static. When linking dynamic and /whatever is not in your builtin library search path (like /usr/lib) then you need to add that with a option to the linker into the binary or alter the dynmic link path later with the LD_LIBRARY_PATH environment variable.

ldd BINARY

will show you what the dynamic linker will do when starting BINARY. Of course only dynamic libs show up.

Harald.

[ilgrank]

@ralight
As expected, neither MQTT Explorer
thomasnordquist/MQTT-Explorer#386
nor PAHO JS client
https://www.eclipse.org/paho/clients/js/
are are able to read user properties :(
Is there any way (I know, it gets harder) that the plugin could be altered to allow to add the timestamp as a 'regular' property instead of an user property? (to allow it to work with non-MQTT5 clients, that is)
Thanks!

[ilgrank]

@habazut : Thanks for the tip! Very much appreciated

[ilgrank]

@ralight : sorry, I understand this is going beyond the original scope of this request... but any chance the plugin can be made to work with MQTT v3? (as both MQTT Explorer and PAHO JS Client do not support MQTT v5)
Thanks again

[ralight]

@ilgrank Not in a way that is generally applicable. MQTT v3.x don't have properties, so the only option is to modify the payload. You can do that by modifying the contents of ed->payload in the same callback as in this example plugin. Without knowing the payload format it's not possible to do that.

If you'd like some help putting together a plugin for your payload then get in touch.

[ilgrank]

thanks @ralight
is there a way I can 'dump' one of the MQTT messages from Mosquitto so that I can show it to you?

[ralight]

@ilgrank Yes, a capture with mosquitto_sub or wireshark would do the job. I have to say though, up until now the work on this has been applicable to the whole project or could be used by anyone. What you've asking now is specifically for your company I presume, so I think it would make sense to close this issue and discuss a more formal arrangement over email: [email protected]

[ilgrank]

@ralight : it is a personal project of ecosystem monitoring, not a job at all. That said, thanks for the tips, I'll write you privately!

[PaulCmpbll]

I'm porting a non MQTT system to MQTT and not knowing when a retained message was sent is a big hurdle. The existing solution timestamps all messages on receipt, once, no matter how many times they are sent again. I don't want to impose upon the clients any time keeping requirements.

The solution I have been toying with is to publish a master clock on MQTT and subscribe the clients to it. They can use it, not only, to put in outgoing messages, but also to detect stale messages.

My system sends some messages with expiry times on them. Like "Turn heating on for 5 minutes". These are retained, so the receiving end does need to know when it was sent.

[dberardo-com]

can the broker provide its own "process time" timestamp on every message?

i think this is an important point to allow for deduplication of retained messages. Here the case:

• client A publishes a retained message with a custom "event_timestamp=111" on topic A
• the broker stores the message from client A using a "process_timestamp=113"
• client B was subscribed to topic A and receives the message at t=114, checks that "process_timestamp > 5 seconds ago", then processes the message
• for some reason, client B disconnects from broker and now riconnects. at this point the broker will send all retained messages, but those messages will not be replayed because "process_timestamp" has not changed and the condition "process_timestamp > 5 seconds ago" does not match anymore, so duplicates will be discarded

this qutie simple deduplication algorithm can only work if "process_timestamp" was available. or do you know any other way to achieve this in MQTT?

the other approach i can see is to use the "receive_timestamp" on the client side, basically assuming that retained message will be delivered very quickly, and so initial messages will be discarded.

is there suggestions how to go about this problem ?