Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REQUEST] ZBOSS radio library for zigpy + a serial to TCP proxy/bridge/server for the Zigbee serial interface? (IDFGH-7711) #9253

Open
Hedda opened this issue Jun 29, 2022 · 7 comments
Open

[REQUEST] ZBOSS radio library for zigpy + a serial to TCP proxy/bridge/server for the Zigbee serial interface? (IDFGH-7711) #9253

Hedda opened this issue Jun 29, 2022 · 7 comments
Assignees
@lhespress
Labels
Status: In Progress Work is in progress Type: Feature Request Feature request for IDF

Comments

@Hedda
Copy link

Hedda commented Jun 29, 2022 •
edited
Loading

Not sure if this feature request belongs here for ESP-IDF or should be posted to https://github.com/espressif/esp-protocols/issues

Read Espressif has joined "ZBOSS Open Initiative (ZOI)" so ESP32-H2 will use the ZBOSS Zigbee stack by the DRS Corporation.

https://components.espressif.com/component/espressif/esp-zboss-lib

https://www.esp32.com/viewtopic.php?t=24624

https://dsr-iot.com/news/78

Therefore like to request DRS and Espressif make a ZBOSS radio library (written in Python) for the zigpy project (a Zigbee integration library) + add a TCP serial proxy/bridge component so that ESP32-H2 with ESP ZBOSS can in combination with it be used as a remote Zigbee Coordinator adapter by multiple third-party Zigbee gateway host implementations without modification:

https://github.com/zigpy/zigpy

https://github.com/zigpy/zigpy-cli

zigpy is an open-source Python library implementing Zigbee ZDO and ZDO integration with a unified serial API/CLI hardware-independent abstraction interface for different radio manufacturers and it is today used as a dependency by a few very popular DIY home automation application projects, including ZHA interaction for Home Assistant, Zigbee plugin for Domoticz, and Zigbee plugin for Jeedom, which allow the Zigbee host application implementation in those project to support different Zigbee stacks using the same interface/API.

https://www.home-assistant.io/integrations/zha

https://www.domoticz.com/wiki/ZigbeeForDomoticz

https://doc.jeedom.com/en_US/plugins/automation%20protocol/zigbee/

From the end-user point-of-view the appeal is by using zigpy the Zigbee application/integration implementation on the host side can in practice support almost all Zigbee Coordinator adapters from any manufacturer, and they can today for example be used out-of-the-box with Home Assistant ZHA, Domoticz, and Jeedom, as long there is a matching Zigbee stack radio library for zigpy (Zigbee application/integration implementation just need to depend on zigpy and its radio libraries instead of adding direct support for only a specific manufacturers API and Zigbee Coordinator adapter hardware):

https://github.com/zigpy/zigpy-znp

https://github.com/zigpy/zigpy-deconz

https://github.com/zigpy/bellows

https://github.com/zigpy/zigpy-zigate

https://github.com/zigpy/zigpy-xbee

Before looking at the code should first read this document:

https://github.com/zigpy/zigpy/blob/dev/CONTRIBUTING.md

For reference, also read this somewhat related discussion about creating a similar (but not same) new radio library for zigpy:

zigpy/zigpy#394

PS: Might also be interested to read through these blog-posts by Jeedom which explain why they choose to depend on zigpy:

https://blog.jeedom.com/6107-parlons-zigbee/

https://blog.jeedom.com/5183-tout-ce-quil-faut-savoir-sur-le-plugin-officiel-zigbee/

PPS: Please understand that those zigpy libraries are free and open-source projects, (licensed so they can be used by the most popular open-source home automation applications), and developed by volunteers from the DIY home automation software communities in their spare time so unfortunately there is not a lot of development documentation, instead, if you (or DSR or other ZOI members) are interested in taking on this project to cretae a ZBOSS radio library for zigpy then you would primarily have to look at the code of all the existing radio libraries, like for example the zigpy-znp radio library (alternatively bellows or zigpy-deconz are among the best maintained) to see how their hardware serial API/CLI translation is done from proprietory serial API to the zigpy radio API/CLI which is meant to be unified for all zigpy radio APIs and CLIs. Then of course create an issue or discussion to post questions on specific problems if you get stuck or need feedback.

https://github.com/zigpy/zigpy/discussions

https://github.com/zigpy/zigpy/issues
@Hedda Hedda added the Type: Feature Request Feature request for IDF label Jun 29, 2022
@espressif-bot espressif-bot added the Status: Opened Issue is new label Jun 29, 2022
@github-actions github-actions bot changed the title [REQUEST] ZBOSS radio library for zigpy + a serial to TCP proxy/bridge/server for the Zigbee serial interface? [REQUEST] ZBOSS radio library for zigpy + a serial to TCP proxy/bridge/server for the Zigbee serial interface? (IDFGH-7711) Jun 29, 2022
@Hedda
Copy link
Author

Hedda commented Aug 3, 2022 •
edited
Loading

FYI, zigpy developer puddly has begun preliminary work on a new radio API for zigpy as well as on merging radio libraries respective high-level functions and command-line tools into zigpy-cli. Still very early on thus now would be a great time for Zigbee radio developers to start giving your feedback and input on the design of this new unified Zigbee CLI for zigpy which is meant as a common support for all radio manufacturers. See:

https://github.com/zigpy/zigpy-cli/ (and https://github.com/zigpy/zigpy-cli/issues plus there zigpy/zigpy-cli#5)

At the moment he is focusing on completely exposing the low-level network and node info, (for example energy scanning already works with TI ZNP and the deCONZ ConBee/RaspBee using the same code however the zigpy-znp command line tool just needs to be transplanted into zigpy-cli), so if you want a chance to influence what should go into that zigpy CLI versus what should go into the separate zigpy radio libraries for each manufacturer then now would be the best time to begin contributing.

  • This new CLI as a standard radio settings API is primarily being discussed here -> [RFC] Standard radio settings API #842
  • Development started here -> New radio API #848
  • Cross-radio backup and restore -> Implement new radio API zigpy-cli#2
  • pully mentioned that he is eventually planning on trying to add a similar shell using IPython for sending commands interactively ("since it has autocompletion and is async-friendly").
  • pully also mentioned that bootloader commands are low on the priority list, "since that code is often pretty complex and uses internal radio library code. A plugin system could work, but since the existing bootloader tools are synchronous, it'd take either a complete rewrite to work with zigpy-cli, or the tools would essentially be run externally in a separate thread, which is no easier than directly running the original tools."

FYI, puddly has now merged the new radio API so all of zigpy radio libraries should now use the new unified API.

@Hedda
Copy link
Author

Hedda commented Sep 27, 2022

See now also released ESP ZBOSS 3.0 binary libraries (esp_zboss_lib) to support Zboss Zigbee 3.0 stack on chips like ESP32-H2:

https://github.com/espressif/esp-zboss-lib

And can be used with "ESP" Zigbee RPC + Gateway examples with their ESP32-H2 DevKitC (ESP32-H2-DevKitC-1 V2.1) board:

https://github.com/espressif/esp-idf/tree/master/examples/zigbee/

https://github.com/espressif/esp-idf/blob/master/examples/zigbee/esp_zigbee_rcp

https://github.com/espressif/esp-idf/tree/master/examples/zigbee/esp_zigbee_gateway

Again, the official native ZHA (Zigbee Home Automation) integration in Home Assistant would still need a ZBOSS (e.g. zigpy-zboss) radio library to be able to use ESP32-H2 with ZBOSS Zigbee serial interface as a remote Zigbee Coordinator in serial pass-through mode.

https://www.home-assistant.io/integrations/zha

https://github.com/zigpy/zigpy

See similar zigpy radio libraries compatible with Zigbee stack interfaces from other manufacturers

  • Texas Instruments based Zigbee radios with all compatible Z-Stack firmware via the zigpy-znp library for zigpy.
  • Silicon Labs (EmberZNet) based Zigbee radios using the EZSP protocol via the bellows library for zigpy.
  • dresden elektronik deCONZ based Zigbee radios via the zigpy-deconz library for zigpy.
  • ZiGate based ZigBee radios via the zigpy-zigate library for zigpy.
  • Digi XBee based Zigbee radios via the zigpy-xbee library for zigpy.

PS: I definitely believe there should be commercial interest in making this happen from Espressif as a company since there are already several ESP32 based products Zigbee gateway products popular in the Home Assistant community that achieve a similar function however they do so by connecting external Zigbee radio serial modules to ESP32 tand hen provide a se2net (serial proxy) type serial tunnel that the host computer running Home Assistant and the ZHA integration can map and connect to with a serial socket via socat:

ESP32 based product specifically made for the home automation market to be used with Home Assistant and similar projects:

There are also other ESP32 based projects and products that hack similar hardware to make them serve the same purpose. Ex:

@chshu
Copy link
Collaborator

chshu commented Sep 29, 2022

@Hedda Thanks for collecting so many interesting projects!

As you already noticed, we are now focusing on ESP's one-stop Zigbee Gateway solution, i.e., the two chips solution using a ESP32-S3 Wi-Fi SoC and a ESP32-H 802.15.4 SoC. A typical use case is the Matter-Zigbee Bridge: https://github.com/espressif/esp-matter/tree/main/examples/zigbee_bridge.

You may also refer to ESP Zigbee SDK programming guide: https://docs.espressif.com/projects/esp-zigbee-sdk/en/latest/esp32/introduction.html

For zigpy, we do have some interest to evaluate it, but after a quick glance at the existing repos, I didn't find a high level introduction such as the topology diagram or structure introduction, for instance, which features needs to be supported or what kind of interface need to be provided in the radio libray. Understand we should be able to understand these from the source code, but just want to check if there is a quick start guidance.

@Hedda
Copy link
Author

Hedda commented Sep 30, 2022

For zigpy, we do have some interest to evaluate it, but after a quick glance at the existing repos, I didn't find a high level introduction such as the topology diagram or structure introduction, for instance, which features needs to be supported or what kind of interface need to be provided in the radio libray. Understand we should be able to understand these from the source code, but just want to check if there is a quick start guidance.

@chshu Yes you have at least partially understood that correctly. The reason for that is that zigpy is still only a none-commercial FOSS (Free and Open Source) project thus so even though the zigpy libraires are used by many other larger application project so far no one is gettings directly paid to work on zigpy and therefore the documentation has not been prioritized.

Anyway, other developers have asked similar questions about documentation of the zigpy radio API and other than looking at the code of an existing radio library for zigpy (where the best example right now is currently -> https://github.com/zigpy/zigpy-znp ) the best approach is probably to read those existing long discussions and try to learn, such as this here -> zigpy/zigpy#394

Another suggestion would be for you to open a new issue or new discussion in the zigpy project order to open a dialog with the zigpy (and zha) developers there, see -> https://github.com/zigpy/zigpy/ and/or -> https://github.com/zigpy/zigpy/discussions

@Hedda
Copy link
Author

Hedda commented Sep 30, 2022

@chshu By the way, if Espressif management needs convincing of the commercial interest before deciding to invest developer time on a radio library for zigpy then suggest looking at just the existing Home Assistant userbase to get an idea of potential customers who might buy such chips if there was a radio library zigpy so that the "Zigbee Home Automation (ZHA)" integration (which depends on zigpy) in Home Assistant could use it directly then you might want to check out analytics for Home Assistant installations and perhaps more specifically the statistics for "Zigbee Home Automation (ZHA)" installation base. See

https://analytics.home-assistant.io/#installs

and

https://analytics.home-assistant.io/#integrations

But before looking at those however note that the analytics feature in Home Assistant is optional is disabled so their currently posted statistics just reflect users who actively enabled analytics (i.e. only those who “opt-in” to submit their statistics), and the founders of Home Assistant have previously posted a guestimate that only 20% of Home Assistant users has actually chosen to “opt-in” for analysis.

https://www.home-assistant.io/blog/2021/11/12/100k-analytics/

I have recently calculated that if the guestimate of only 20% of Home Assistant users has chosen to “opt-in” for analysis still holds true and as such statistics today listing 173 146 Active Home Assistant Installations of “opt-in” for analytics there should in estimate really be about ‭865 730‬ Home Assistant Installations out there and so far of those 17.4 % have currently installed the "Zigbee Home Automation (ZHA)" integration (which depends on zigpy), meaning that there should today be around ‭150 637 installations of the "Zigbee Home Automation (ZHA)" integration in the world, and doing on historical statistics the percentage of "Zigbee Home Automation (ZHA)" integration will keep growing relatively fast. See:

https://community.home-assistant.io/t/home-assistant-founders-believe-there-is-currently-around-50-000-installations-of-zha-integration-what-do-you-zigbee-users-in-the-community-think-about-those-statistics/357305/51

PS: I believe the reason why the percentage of "Zigbee Home Automation (ZHA)" integration have been and will keep growing relatively fast is the access to inexpensive and good quality Zigbee adapters and gateways/bridges have become much more common in the last two years or so.

@Hedda
Copy link
Author

Hedda commented Sep 30, 2022

Another suggestion would be for you to open a new issue or new discussion in the zigpy project order to open a dialog with the zigpy (and zha) developers there, see -> https://github.com/zigpy/zigpy/ and/or -> https://github.com/zigpy/zigpy/discussions

@chshu FYI, I have now posted a placeholder request for open discussion in the zigpy repository here -> zigpy/zigpy#1052

@espressif-bot espressif-bot added Status: In Progress Work is in progress and removed Status: Opened Issue is new labels Nov 21, 2022
@Hedda
Copy link
Author

Hedda commented May 10, 2023

FYI, DamKast has just announced "zigpy-zboss" as an (unofficial and expeimental) ZBOSS radio library for zigpy:

https://github.com/kardia-as/zigpy-zboss

zigpy/zigpy#394

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lhespress lhespress

Labels
Status: In Progress Work is in progress Type: Feature Request Feature request for IDF
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Hedda @chshu @lhespress @espressif-bot