Add compatibility with emonpi and flexible sending of data via MQTT

[alandpearson]

This feature adds the ability of emonGLCD to work with emonpi, and for the emonGLCD to receive data via MQTT.

It allows to select any node/input variable in emonpi and send them to multiple emonGLCDs for display using MQTT (script provided to do so)
It also makes minor improvements to the codebase, for example checking RF data received length and if the RF data is for itself.

The emonGLCD is a great device and I'm saddened to hear it's discontinued. Maybe this code update will make it live again ?

[glynhudson]

Great, thanks for this. I also liked the emonGLCD. The main issue was that it was difficult to manufacture pre-assembled at a reasonable price. Most users don't want to build up the kit. We still have kits left in stock which we're struggling to sell: https://openenergymonitor.com/sale-emonglcd-lcd-display-unit-kit/

[alandpearson]

Thats real shame. The unit absolutely rocks. It's a lot better in person than in the pics on the site too.

I'd really really hate to see it discontinued, as it really fills a niche. Maybe there is a way to make a SMD version that would save a lot of assembly, leaving only the display to solder on ?

I bought three of the things I was so impressed...

[glynhudson]

That was the main issue that making an SMD version was going to be tricky and hard to do in a cost-effective way. Especially since a proper injection moulded enclosure would be required to give the unit proper protection to obtain CE mark. The injection moulding process was going to require a significant upfront investment. The feedback we got was the most users would prefer to use a smartphone or tablet. We may revisit it, maybe a 3D printed enclosure could work for a small batch run.

[alandpearson]

So what I meant was the PCB could be all SMD and completed. I think I understand what you mean, to sell as a completed product requires all of the above to get CE. That's such a shame.
I don't know how to avoid all of that, I just know it's an awesome little product and something that could be bundled with the emonpi itself - as the complete monitoring solution.

Also, locating and loading the firmware on these devices is a huge blocker. You have tinkerers who don't have all the kit to flash an arduino, nor will to do so. But they are quite cable to solder something together...

Sorry I just love the device in it's current form (apart from the hour required to solder it together ;) )

PS, if you could update the shop page to say it is now compatible with the emonpi that may help !

[glynhudson]

PS, if you could update the shop page to say it is now compatible with the emonpi that may help !

That's a good idea. So all a user has to do is to add the following to emohub.conf to send emonPi data to the emonGLCD:

[[20]]
nodename = emonglcd
firmware =V1_1
hardware = emonglcd
[[[rx]]]
names = temperature,
datacodes = h
scales = 0.01,1
units = c
[[[tx]]]
names=nodeid,hour,minute,second,utilityW,solarW,utilityKwh,solarKwh
datacodes =b,b,b,h,h,H,H
units = h,min,sec,W,W,kwh,kwh

And then have some sort of service to post the data to MQTT? How are you doing this? Maybe we could post a nodRED example?

[alandpearson]

Exactly !

That's all the user has to do is add the above to the emonhub.conf.

I created a service to do just that (send via MQTT) and is very configurable. It's included in the pull request for this. From the readme

A companion script (emonglcd-send.pl) has been provided in this repo that reads any emonhub node/feed values and transmits them to mulitple emonglcd(s).
Simply place the emonglcd-send.cfg file in /etc , edit it with your emonGLCD nodeid(s) and the variables you wish to use for solarW, utilityW, solarKwh and utilityKwh.
The script will update the emonpi each time any of these node variables change in emonhub. emonglcd-send.pl works by subscribing to these topics via MQTT and using MQTT to make RF transmit requests to emonhub (which are then sent to each emonglcd).

A systemd run file has also been provided to run this at startup, simply place systemd/emonglcd.service into /lib/systemd/system and run 'systemctl enable emonglcd'

I also have some nodered stuff going on to integrate with Alexa ('Alexa open bedroom blinds!') that sends via nodeRed/MQTT node to emonhub which is then transmitted to the jee node.

I'll dig it out.

Personally as someone who had no mqtt/node red experience until a few short months ago, the mosquitto_pub and mosquitto_sub test programs were invaluable.

[glynhudson]

Got it, the updated emonglcd readme explains it well. I still think a nodered flow would be easier for most users.

[alandpearson]

So I'm all for choice.
Here is a node-red flow that will keep your beautiful emonGLCD (with spanking emonPi compatible f/w) up to date ...

[
    {
        "id": "fa3f2e11.da5a3",
        "type": "tab",
        "label": "emonGLCDupdate",
        "disabled": false,
        "info": "This flow updates an emonGLCD display, with data from EmonHub/Emonpi\n\nSimply edit the MQTT nodes to select which emonhub input you wish to use for\n\nsolarW\nutilityKw\nsolarKwh (eToday is some parlence)\nutilityKwh\n\n\nYou can enhance this flow to do things like reset Kwh at midnight etc.\n\n"
    },
    {
        "id": "475e0948.0fda1",
        "type": "mqtt in",
        "z": "fa3f2e11.da5a3",
        "name": "Utility Kw",
        "topic": "emon/socomec/Kw",
        "qos": "2",
        "broker": "396670b0.0ef31",
        "x": 220,
        "y": 80,
        "wires": [
            [
                "4c33d7ab.747"
            ]
        ]
    },
    {
        "id": "2b3f926e.90a496",
        "type": "mqtt in",
        "z": "fa3f2e11.da5a3",
        "name": "Solar Watts",
        "topic": "emon/samilpower/W",
        "qos": "2",
        "broker": "396670b0.0ef31",
        "x": 230,
        "y": 120,
        "wires": [
            [
                "afa1a586.c1a82"
            ]
        ]
    },
    {
        "id": "c2dddf79.b9a558",
        "type": "mqtt in",
        "z": "fa3f2e11.da5a3",
        "name": "Solar Kwh",
        "topic": "emon/samilpower/ETODAY",
        "qos": "2",
        "broker": "396670b0.0ef31",
        "x": 220,
        "y": 200,
        "wires": [
            [
                "4ae32015.89d278"
            ]
        ]
    },
    {
        "id": "5d221af6.ed704c",
        "type": "mqtt in",
        "z": "fa3f2e11.da5a3",
        "name": "Utility Kwh",
        "topic": "emon/socomec/Kwhplus",
        "qos": "2",
        "broker": "396670b0.0ef31",
        "x": 220,
        "y": 160,
        "wires": [
            [
                "826d969c.398cc8"
            ]
        ]
    },
    {
        "id": "4c33d7ab.747",
        "type": "function",
        "z": "fa3f2e11.da5a3",
        "name": "Convert to watts & store",
        "func": "//Only send integers to emonglcd \nflow.set(\"utilityW\", msg.payload * 1000) ;\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 450,
        "y": 80,
        "wires": [
            [
                "ebe520f.c7915e"
            ]
        ]
    },
    {
        "id": "826d969c.398cc8",
        "type": "function",
        "z": "fa3f2e11.da5a3",
        "name": "Convert to int & store",
        "func": "//Only send integers to emonglcd \n\nflow.set(\"utilityKwh\", msg.payload * 100) ;\n\nreturn msg ;",
        "outputs": 1,
        "noerr": 0,
        "x": 440,
        "y": 160,
        "wires": [
            [
                "ebe520f.c7915e"
            ]
        ]
    },
    {
        "id": "4ae32015.89d278",
        "type": "function",
        "z": "fa3f2e11.da5a3",
        "name": "Convert to int & store",
        "func": "//Only send integers to emonglcd \n\nflow.set(\"solarKwh\", msg.payload * 100) ;\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 440,
        "y": 200,
        "wires": [
            [
                "ebe520f.c7915e"
            ]
        ]
    },
    {
        "id": "ebe520f.c7915e",
        "type": "function",
        "z": "fa3f2e11.da5a3",
        "name": "Create emonGLCD MQTT msg",
        "func": "var utilityW = flow.get(\"utilityW\") ;\nvar utilityKwh = flow.get(\"utilityKwh\") ;\nvar solarW = flow.get(\"solarW\") ;\nvar solarKwh = flow.get(\"solarKwh\") ;\n\nvar date = new Date() ;\n\nvar seconds = date.getSeconds();\nvar minutes = date.getMinutes();\nvar hour = date.getHours();\nvar time = date.getTime() / 1000\n\nvar lastsend = flow.get(\"lastsend\") ;\n\nif (lastsend === undefined) {\n    lastsend = 0 ;\n}\n\nif (solarKwh === undefined) {\n    solarKwh = 0;\n}\n\nif (solarW === undefined) {\n    solarW = 0;\n}\n\nif (utilityKwh === undefined) {\n    utilityKwh = 0;\n}\n\n\nif (utilityW === undefined) {\n    utilityW = 0;\n}\n\n\nmsg.payload = hour + ',' + minutes + ',' + seconds + ',' + utilityW + ',' + solarW + ',' + utilityKwh + ',' + solarKwh;\n\n\nif (time - lastsend > 10) {\n    flow.set(\"lastsend\", time) ;\n    return msg;\n} else {\n    return null ;\n}\n",
        "outputs": 1,
        "noerr": 0,
        "x": 770,
        "y": 160,
        "wires": [
            [
                "3ce29bf7.b8a0c4",
                "b3b6c854.034458"
            ]
        ]
    },
    {
        "id": "afa1a586.c1a82",
        "type": "function",
        "z": "fa3f2e11.da5a3",
        "name": "Convert to int & store",
        "func": "//Only send integers to emonglcd \n\nflow.set(\"solarW\", msg.payload) ;\n\nreturn msg ;",
        "outputs": 1,
        "noerr": 0,
        "x": 440,
        "y": 120,
        "wires": [
            [
                "ebe520f.c7915e"
            ]
        ]
    },
    {
        "id": "b3b6c854.034458",
        "type": "mqtt out",
        "z": "fa3f2e11.da5a3",
        "name": "emonGLCD",
        "topic": "emonhub/tx/20/values",
        "qos": "2",
        "retain": "",
        "broker": "396670b0.0ef31",
        "x": 1030,
        "y": 160,
        "wires": []
    },
    {
        "id": "3ce29bf7.b8a0c4",
        "type": "debug",
        "z": "fa3f2e11.da5a3",
        "name": "",
        "active": false,
        "console": "false",
        "complete": "false",
        "x": 1030,
        "y": 220,
        "wires": []
    },
    {
        "id": "396670b0.0ef31",
        "type": "mqtt-broker",
        "z": "",
        "broker": "localhost",
        "port": "1883",
        "clientid": "",
        "usetls": false,
        "verifyservercert": true,
        "compatmode": true,
        "keepalive": "15",
        "cleansession": true,
        "willTopic": "",
        "willQos": "0",
        "willRetain": "false",
        "willPayload": "",
        "birthTopic": "",
        "birthQos": "0",
        "birthRetain": "false",
        "birthPayload": ""
    }
]

Please test it.

I'll add the node-red flow to the emonGLCD repo and do another pull.

I think I will add the daemons to the emonpi repo in the right places, so they are setup and ready to go. That way, a user doesn't have to use node-red and the emonGLCD should work with emonpi out of the box ?

[glynhudson]

Thanks. NodeRED is no longer installed by default on the emonPi. But it's easy to install if required.

[alandpearson]

I hate hi-jacking a commit thread for this, but...

I do feel leaving out node-red and openhab is bad. Very bad.
Having these there and well setup by default allows people to play. People like me, who would never have went near node-red if it wasn't there and all setup by default.
Openhab is also very useful

I read Trystan's thread on the future of development and shuddered at the thought of containers, and really think you'd do well sticking with raspian. Containers raises the complexity, barrier to entry and will reduce people like me 'discovering' and hacking on the cool stuff you make.....
(I've been hacking on emonpi and this stuff for a few years, just never committed before.. )

I get it may make your life easier as no one is changing stuff 'under your' feet. But it also means you don't evolve as quickly to new things... It may also be worth raising the spectre of why you don't use standard debian package management .. this might make updates easier and packaging easier too (after the pain of setting it up)

Please keep emonpi as open as possible... The fact it's rasppi inside attracts a lot of clever folks, and those folks know raspian... :) my 2c !

And thanks again for great products!

[glynhudson]

Thanks for your comments, it's really great to hear that you have found the emonPi and nodered useful. I personally pushed to get nodered included on the emonPi, however, it has become apparent that not many users use nodered / openhab and even fewer users apply security updates. We considered the increased attack vector of having nodered/ openhab running unknown to many users was not worth it.

At the same time, we have also changed our latest emonSD image to dropping our read-only root file system, so installing NodeRED on the current emonPi is as simple as sudo apt-get install nodred. Maybe a hybrid solution of having nodred installed but disabled by default would have been a better solution. This is something we will consider in the future.

You raise a valid point about the increased complexity of containers. However, containers offer lots of positives for developers and users. I personally think that if we documented the development process and made developing using containers accessible it would bring significant benefits. We did investigate using Debian packaging but for the emonPi we require much granular lower level access. No decision has been made yet, we're still evaluating options. Thanks again for your feedback.