Getting to the finish line: JuicePassProxy as fully isolated local control with ESP or exploit-based traffic redirection, documentation, and support for different types of HA instances

[FalconFour]

Well, today is the day we've all been waiting for. Enel X Way NA is shutting down. Thousands of angry people looking for a solution. It is time to shine - end of the horrible app era, and beginning of the community-developed server era. I hope we aren't too late to finish this, as lots of eyes are hopefully coming here to see what the state of the project is and how long they need to wait to make a "turnkey" solution they can just follow a guide and implement for themselves.

Admittedly, as the app has continued working for me and all my various boxes (both old and new alike) - in no small part because I know the internal quirks that need to be aligned to make it tick - I've just kept using the app and haven't tried setting up JPP on my Home Assistant yet. There are still a few difficult hurdles to clear.

• UDP traffic by Man-in-the-Middle (MitM): JPP needs an easy way to redirect traffic to/from the box. This is a unique challenge because the protocol is so unusual - using a UDP stream with no local API. Not only do many routers not support redirecting traffic, but folks wouldn't know how to do that unless they were shown how to do it in all the different models of routers that are capable. Thus, I envision this as a cheap dongle like a smart-home device hub, based on a cheap ESP(-32 or other cheaper/simpler chip), to which a JuiceBox will be pointed for its WiFi, and the dongle will simultaneously connect to your real home WiFi where the data can be forwarded.
• UDP traffic by exploit: There is also an interesting article on exploiting the JuiceBox, which is not planned to be patched, and may open up the ability to override where the ZAP software sends its traffic, in absence of an Enel directory server connection to tell it otherwise. Given that the directory server is still online today, I'm not sure if this is immediately feasible.
• Ensure full protocol implementation: Every incoming telemetry message should receive a reply command packet commanding current and setting time, etc. Need to ensure JPP is doing this (now that we know how to create acceptable messages).
• Compatibility with local and cloud Home Assistant: Whatever we choose to use, it should be compatible with most HA, including that with the lowest friction: Cloud. Should investigate whether we can implement JPP in the cloud for folks to send their JB traffic to, who don't want to run HA on a server at home (e.g. this is their first introduction to Home Assistant).
• Documentation: A setup guide that makes this as easy as replacing the relay (like the guide I made on iFixit for replacing your JB relay), including a summary on setting up Home Assistant, adding the integration, configuring the MitM device or "jailbreak" exploit, configuring the box WiFi, and setting the integration up in HA for basic functionality like scheduling and statistics.

I'm doing my best but, you may be able to tell, this is my first (and somewhat forced) deep dive into contributing to an open project like this. I use ChatGPT to lay the foundation for code (😂), and whenever I'm doing development work, I step away from the computer with a huge headache. Not fun for me. But I deeply want to help in every way I can, especially in this golden opportunity where folks are considering whether they have to buy a new box. Lots of E-waste at stake!

Should we go with creating sub-issues for each of these as milestones? Other ideas?

[ivanfmartinez]

• Ensure full protocol implementation: Every incoming telemetry message should receive a reply command packet commanding current and setting time, etc. Need to ensure JPP is doing this (now that we know how to create acceptable messages).

Now with more people interested in this subject maybe we can have more sample messages to be able to support more firmware versions.

[ivanfmartinez]

• UDP traffic by Man-in-the-Middle (MitM): JPP needs an easy way to redirect traffic to/from the box. This is a unique challenge because the protocol is so unusual - using a UDP stream with no local API. Not only do many routers not support redirecting traffic, but folks wouldn't know how to do that unless they were shown how to do it in all the different models of routers that are capable. Thus, I envision this as a cheap dongle like a smart-home device hub, based on a cheap ESP(-32 or other cheaper/simpler chip), to which a JuiceBox will be pointed for its WiFi, and the dongle will simultaneously connect to your real home WiFi where the data can be forwarded.

If all versions support the telnet method for changing the server address why need to use an extra device ?

A set of scripts (that can be used from smartphone or computer) can be used to connect a factory state device to the wifi of the user and after that juicepassproxy can telnet and set the server address.

The end users that probably will have most benefit from the ESP device maybe will not have the knowledge to flash the firmware and do the process. doing something from smartphone will be easier for that users like the original enel x does (in very bad way, all times that I have to use I have to make many many retries).

[ivanfmartinez]

A big problem will be if enel x change the devices to use the encrypted messages, that we dont know how to decode yet : #73

[stevegilbert23]

A big problem will be if enel x change the devices to use the encrypted messages, that we dont know how to decode yet : #73

They are going out of business. It's unlikely they will do anything other than shut down the severs running in AWS.

[FalconFour]

As to the "Telnet method", response here: home-assistant/core#86588 (comment) - tl;dr: I am fairly certain the majority of devices don't support that, unless there's a debug hook that reacts to setting that unrelated parameter, no reason it should work, may be worth investigating today/soon to decide if the following is needed. Easy enough to test on my bench.

Expansion on UDP traffic by Man-in-the-Middle (MitM), defined scope:

1. WiFi host (AP) mode, producing a hidden AP to connect the JuiceBox (single client) to. Multiple clients will require more work as it needs to track which client to send replies to (may be simple connection-tracking, TBD if it can be quickly added while building this, and if JPP supports it)
2. DHCP server, responding to DHCP request from JuiceBox and not leaking DHCP into the client WiFi. ESP as the IPv4 "gateway but not really", as well as "DNS server but not really".
3. DNS server, responding to all requests with its own (gateway) IP address as a result, so any server the JuiceBox requests will get a reply "yeah, that's me!".
4. WiFi client mode, connects to real internet/network access point, where the Home Assistant/JPP server is located.
5. DHCP client on client side, to get a DHCP address for itself on the main network
6. DNS client on client side, to look-up DNS name of local server (always my preference to use hostname instead of direct IP, in case the network changes, or if a cloud server is used)
7. UDP forwarder, establishes and tracks UDP connections both to JPP server on the client side (establishes UDP connection to JPP port 8042), and accepts/sends traffic to the JuiceBox on the AP side (listens on UDP port 8042), simply tunnel unmodified packets but with a new source/destination port in packet metadata (routing info).
8. Basic web user interface for selecting/configuring client WiFi network and configuring the target server/port, as well as sending a firmware update

[wozz]

in my experience, even with telnet working to update the target endpoint, it regularly reset itself and became unreliable.

it might be easiest to leverage existing projects like adguard or pihole and create instructions around those setups

[FalconFour]

Oh, and addressing encrypted messages:

A big problem will be if enel x change the devices to use the encrypted messages

Approximately 0% risk there. Encryption was an optional feature that has always been present in ZAP firmware, to meet certain requirements to certain companies that request it. Default mode of operation is unencrypted traffic. Their devs are long gone and I don't think anyone even knows how to build new firmware builds. So, the functionality that exists today is there forever, and the firmware defaults to encryption-disabled.

Not sure why that box got switched to encrypted mode, but it's just a config flag that's easily reset with a factory-default wipe on ZAP systems (anywhere I suggest factory reset, I have to reiterate: never factory-reset a non-ZAP device!). Easy peasy :)

[FalconFour]

it might be easiest to leverage existing projects like adguard or pihole and create instructions around those setups

Definitely on my test list, then. Such platforms may offer too many bells and whistles making setup too complex, especially for something as obscure as a UDP-based protocol to be forwarded - but I'll see if it can be practical. A RPi is also much more expensive than an ESP, but I haven't messed with those projects at all, so... maybe there's cheaper options? Experience and advice welcome!

[FalconFour]

I just tested the Telnet method.

1. Yes, it works
2. No, it's not reliable.
3. It's not reliable because it's based on swapping-out a stream handle from under the ZAP, and the ZAP software keeps stepping-over the UDP handle and saying "oh, that's not right, let's reopen that".
4. By default it connects to jbv1.emotorwerks.com:8042, but then it changes the stream out after talking to the directory server, which makes it go to juicenet-udp-prod3-usa.enelx.com:8047 - this isn't stored anywhere, and it only persists for the current power cycle. It always asks the directory server, and if it isn't available, it connects to "jbv1".

I should add, it's a very clever trick I hadn't even thought was being used until I read the code. Unfortunately, knowing it constantly monitors and resets the connection, I just don't know if it can be made fully reliable. It might be enough as a starting point, though - so I'm definitely giving it a try soon!

[natematias]

Thanks for this wonderful summary of steps we can take!

I have sent out notes to Consumer Reports and the Harvard Cyberlaw Clinic for referrals to legal advisors on the IP questions.

@FalconFour and any other maintainers, would it be helpful to organize a call to discuss next steps? I can be available Friday from 8pm ET onward, as well as some times on Saturday and Sunday ET. Then perhaps we can talk about ways to find out who's interested, what skills/time/resources they have, and think about how to coordinate going forward?

Here's a Whenisgood link. I'll take a look around noon ET on Friday: https://whenisgood.net/4hwijjh

Here at the Ithaca Ecovillage, we have a few folks with developer experience, including some IOT experience. I'm planning to spend time over the weekend setting the system up and documenting the process for less technical folks along the way. If I can get things installed and set up for my needs, I would be open to collaborating with someone to develop some docs and a disk image for a common, <$100 tiny computer (like a Raspberry Pi 4 Model B) that people could use to add juicepassproxy to their network as a short term solution for people.

--Nathan

[ivanfmartinez]

On JPP side I have some sugestions of changes that can help in future, they need help from people that know better how the homeassistant entities and devices work and more about python than me.

• Change the MQTT/HA behaviour to be more like zigbee2mqtt does

  • JPP will be a controller and each JuiceBox a device
    • create each device on the first message received and use serial as the identification
    • use the current rating to define max values for the current entities instead of MAX_CURRENT that are now used because the entities are defined on the startup

• create an state object based on the serial number to allow multiple juicebox devices to be controlled by same JPP server

  • decode the received message and then create or reuse state object

• allow multiple telnet addresses for multiple devices be changed

  • this is for basic users, I still preffer the redirect on router

[atc99]

I don't know if this is something that this project wants to pursue, but I have been able to successfully redirect DNS and read the UDP packets on my windows 10 machine as a standalone setup. This may be a far simpler way to "manage" the juicebox settings (such as current offline max for matching the circuit capacity) for most users with just a windows laptop (with admin rights).

• Use a Windows 10 Laptop connected on [wifi, ethernet, cellular] to the internet.
• In Settings | Network & Internet | Mobile Hotspot, [optionally change the SSID/password], Enable the mobile hotspot
• Use ipconfig or other means to determine the hotspot gateway IP (i.e. 192.168.137.1)
• Update the HOSTS file on the laptop to add the DNS override entries to point to the hotspot gateway IP.
• Run the app/powershell UDP receiver to listen on port 8042 (you may need to allow it to listen through the windows firewall)
  PS> Start-UDPServer -Port 8042
• Using a different device, such as a phone, power up the JB, connect to its wifi and access the setup portal and connect to the mobile hotspot you set up.
• Watch the UDP receiver to see the packets come in.
• Optionally, you can telnet to the JB to verify that the DNS is overridden by doing a network_lookup jbv1.emotorwerks.com and it is resolving as the hotspot gateway IP.

JPP could be the "app" that listens if the user installs python and runs the python script directly on their windows machine, but this still requires HASS for the front-end. I'm thinking an all-in-one settings app would be more appropriate here.

Side note about Android: I tried to use the same concept on my android phone, but failed. I tried to use the phone as a mobile hotspot and use personal DNS filter (pDNSf) to override the DNS (it acts as a VPN) and then use an app to listen on UDP 8042. Unfortunately, without root, mobile hotspot traffic on Android is routed directly to the internet and not through the VPN even when the VPN is connected which meant that this configuration was not viable. Now, with the proper DNS override in place on my network at the router level, I was able to read the packets in the app without any problem so that could open the door to an app to control the JB if the network conditions are satisfied.

[snicker]

hey @FalconFour @atc99 @ivanfmartinez @natematias

doing a lot of issue cleanup tonight- will you all make sure ideas from above are included in #102 and we can close this issue once confirmed.

[ivanfmartinez]

I dont think this can be replaced by #102.

#102 is about JPP refator on the current idea, this is about a bigger solution that uses JPP and like done by niharmeta here https://github.com/niharmehta/juicepass_proxy_install_scripts