Project approach / purpose discussion

[jblance]

Use this issue to further discuss purpose
I'll update the discussion doc with content from here
Also note here (or PR) any alterations needed in discussion doc

[DMIINC]

Hi Jblance and others,

Thank you Jblance for all the help.

What is outlined in WIP discussion is something I have been thinking of since August of last year and is my ultimate goal for this year minus the electric car. The goal was to get one this year however due to virus outbreak that is placed on hold for now. However once solar panels are installed my goal would be to integrate different energy storage device instead of the car. Electric hot water heater inline with gas water heather already present in the home. During winter when sun is not as abundant as it is in summer use time of use tariff with power company to supplement power needs (charge batteries to certain level) which would help lower down the energy cost.

What I have:
53 KWh of LiFePo4 cells equally divided into 3 power walls.
3 LV5048 inverter/chargers
9 KW of solar panels waiting on HOA gods or a judge for approval to be installed.
3 Dumb BMS's. However they are well built and serve their purpose amazingly good. They limit charge/discharge at 120A and monitor cell voltage. If cell voltage is above or below limits they stop everything.
3 Balancers/monitors JKBMS (jblance is almost intimate with them, lol) for purpose of monitoring cell health.

Since Lithium chemistry has almost flat discharge curve at 0.2 C rate it is impossible to tell SOC by just looking at cell/power wall voltage. Reasonably accurate SOC device must be used in order to implement what is outlined in WIP discussion.

Reasonably accurate SOC device must take in to account Voltage, Amperage and Temp from power wall according to documentation I have read.

So far I have:
Volts provided by JKBMS
Amps provided by ADS1115
Temp provided by DHT22

All going to Raspberry Pi. I think with devices above I could implement majority of what is outlined in WIP discussion. I would preferably use solid state (SSR) relay to control when hot water heather would be operational depending on SOC of my power walls.

i.e. if SOC=>85% then SSR=on

When it comes to coding all of this in python, my skills are equal to .01% of jblance's. When it comes to physicaly integrating this properly I have no issues what so ever.

I do enjoy DIY projects and have accomplished many of them. I prefer DIY over someone else's solution that might not be able to accomplish everything I needed to do.

Thank you all,

[volkerjaenisch]

Sorry for the long delay. I was occupied with a new roof, a new greenhouse and lots of customer projects.

Today I started to code an intergration in Homeassistant (HASS) for the MPP-solar code.

So far I found no code samples to implement the device/subdevice structure we need for the modelling of mpp-solar devices. The developer documentation of HASS is more than sparse.

I asked for advice on the HASS dev-code chat channel, still waiting for an answer.

Meanwhile I will implement all the boilerplate for a Integration doing NOOP but works with the development system of HASS.

Cheers,
Volker

[jblance]

Hi
No problem re delay - none of us are doing this full time :-)
I have looked at HASS integrations too - I think this is the way to go.
I dont think the mpp-solar code is correct to integrate - even with the current usage there are 3 different 'inverter' types (or protocol formats / definitions)

I still think we need to do an inverter state model with configuration that allows different command protocols (as well as battery / battery monitor models)

Relays etc are (should) be covered by the switch functionality - but we may need to extend the base switch to allow more separation / security / safe guards. "Alexa turn off all switches... Power goes off"

[DMIINC]

For purpose of going in right direction here is what I have so far:

Repurposed PC that has Ubuntu 20.04 installed.
mpp-solar service running (3x LV5048 connected via USB to serial cables)
jkbms service running (3x jkbms connected via Bluetooth BLE)
Influxdb
Grafana
Virtual Box with HASS.IO installed (working great)
Started playing with ESP Home since I have couple of ESP32 and associated sensors. Generic setup is easy. Will need to determine if writing software for ESP32 module is better solution rather then taking sensor inputs and letting HASS manipulate data.

[volkerjaenisch]

Does the mpp-solar code run on ESP32?
I am happy with ESP32 as plattform for sensor data. Currently I am trying to get MQTT to run on an ESP32. If this succeeds I will try to port the mpp-solar code.

[volkerjaenisch]

While porting to ESP32 I run into some obstacles.

I use the stock MicroPython ToolChain. I freeze the mppsolar package and bundle it into the MicroPython image. To get more libs I utilize https://github.com/micropython/micropython-lib to drag selected modules into the image.

I already have MicroPython extended by the following libs from that source:

• argparse.py
• ffilib.py
• fnmatch.py
• glob.py
• logging.py
• posixpath.py
• stat.py

Now I run into this error:

MicroPython v1.12-483-g22806ed5d on 2020-05-31; ESP32 module with ESP32
Type "help()" for more information.
>>> import mppsolar
>>> mppsolar.main()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mppsolar/__init__.py", line 50, in main
  File "mppsolar/mpputils.py", line 31, in __init__
  File "mppsolar/mppinverter.py", line 162, in __init__
  File "mppsolar/mppinverter.py", line 84, in getCommandsFromJson
  File "os/path.py", line 14, in abspath
  File "os/__init__.py", line 77, in getcwd
NameError: name 'getcwd_' isn't defined

This is a strong indicator that certain file system operations on ESP32 are not possible. These operations originate in your design decision to store the scheme of the Commands in JSON files.

Does your heart really belong to these JSON files? Why not put them into a single well documented python file?
This approach may be also of use if we have to deal with deviations in the protocols of different inverters. We can come up with per inverter protocols files

Cheers,
Volker

[DMIINC]

Just a quick question? I like idea of using ESP32 to handle Inverter commands and get data out of them. Would one ESP32 be able to handle 2 or 3 inverters in parallel?

[volkerjaenisch]

My knowledge on the ESP32 with Python is one day old.

In general the ESP32 has a lot of computing power, more than an Arduino. I fought my first Doom multiplayer battles on hardware less powerfull :-).

Do you mean
a) two/three inverter stacks or
b) a single inverter stack with two/three inverters?

b) is no problem and already covered by MPP-Solar.
a) is possible since the ESP32 has up to 3 UARTs. But an ESP32 costs near to nothing so I would assign an ESP32 for each inverter stack.

Utilizing ESP32 for data aquisition does not solve any problem with data management. We just push some functionality from the framework e.g. HASS or ESP HOME to the data aquisitioning device.

The ongoing experiments will show if this approach is possible and worth the efford.

[jblance]

My thought for the esp32 was to simplify it down to a serial sender / receiver with mqtt. Basically it would subscribe to a commands topic and (maybe with some security) send any commands that get published to the topic to the inverter (serial port). It would then publish the raw response.

All the complicated bits could then be somewhere else.

That's where my head was going anyway....
Thoughts?

[DMIINC]

@volkerjaenisch single inverter stack with 3 inverters. Cost of ESP32 is not an issue however,
@jblance Idea of ESP32 running inverters is very interesting however having additional pieces of hardware does create additional possible failure points ,more things to maintain and more cables to run etc,.... Sometimes reliability comes with simplicity.

My $0.00002 I might be wrong.

[volkerjaenisch]

@DMIINC I understand your concern. But on the other hand as @jblance said: "Then the complicated things have to be done elsewhere". This would be a distribution of complexity over several components.

My approach aims to locate the "Inverter complexity" in the ESP32 and keep the other things as simple as possible. The ESP32-MPP-Solar module then can be included in any home automation much easier.

At the curren tstate of project I cannot estimate how much of the internal state the inverters or other devices involved have to be distributet to the controlling software (e.g. home automatization). If it would be 100% of the state then probably the idea of @jblance is more fitting. If it is 20% my approach may be the sensible way to go.

In general if you have a complex problem there are seldom simple ways to go. From my experience building complex systems with lots nines of reliability for a living is, that is a good approach is to not distribute the complexity. "Divide et impera" as the romas said :-).

Of the ESP32s you can have a bunch lying around for small money. If one of them goes down, just flash another with the MPP-Solar-software-Image and be online as fast as you can replace it physically. If you have a more complex system like a Raspberry you will need more time to flash, configure etc. And while the ESP32 is down only the single device it controls is offline but the rest of the system is still online.

You see that there are two sides of the coin and both are not shiny. I would suggest that we go for both approaches *1) *2) in parallel and look which one will develop better.

1. We utilize the ESP32 only as serial to WLAN bridge with very limited ability. The complexity is in the external controlling software which utilizes mppsolar. Since we already have 4 serial_device_types in the mppsolar software it will be not great deal to come up with a fifth one, which is the ESP32 remote-protocol.

2. We utilize the ESP32 as controller running mppsolar for e.g. the inverter stack. The interface to the external controlling instance is an MQTT protocol as complex as needed. This MQTT protocol has also to be programmed for case 1) so I see here also no doubling of effort.

[DMIINC]

I agree on divide and rule approach and yes parallel development is a must. Those are basics. I also try to think of any projects in my home from standpoint of WAF or FAF.

[volkerjaenisch]

Hi can somebody please make a small test if the new commands_code in this branch runs.

https://github.com/Inqbus/mpp-solar/tree/ESP32_support

Has not to be run on a ESP32, any platform will do.

Cheers,
Volker

[DMIINC]

Commands will work the same as before from terminal? I will give it a try little bit later.

Thanks,

[volkerjaenisch]

@DMIINC It should do so :-)

[jblance]

I think if we go to having the commands in a python file then we may as well use 'proper' object orientation / abstraction.
So we'd have classes for the various protocol types and classes for the various connectivity types, then the 'inverter' would use the correct classes (that all have the same functions defined)
So inverter.execute would have different classes used based on inverter/protocol and connectivity method.

The json, was a quick way to allow testing / changing commands for different inverters (ie LV5048 uses slightly different commands compared to 4048 that I have (and the hybrid v2 uses a completely different set of commands)

Otherwise we end up (well we already have some of this) with a bunch of if this type then do it this way....

[jblance]

@DMIINC re wanting every thing in one server. You are already not doing this with the ESP + shunt.
My thoughts are more IOT like - keeping the sensors as simple (and cheap) as possible and using a message bus/queue to send and receive info and commands.

A good example of this is the way modern cars are built. There are sensors and functions everywhere, mostly communicating via CAN. These sensors might just read a temp and put the result on the bus, or have enough intelligence to actually control an electric motor based on the commands from the VCU

[jblance]

Another problem with mppsolar service approach, is that is monopolises the serial connection, so if you want to run a command to change a setting you need to stop the service first.

Even if we have an ESP as a 'simple' inverter communicator, there is the question as to how much does it need to know about the inverter.

1. minimum, is just enough to send a command and get the response (+ mqtt to listen and talk)
2. next would be knowing what commands are valid (and how to calculate CRC)
3. more complete might be more of a inverter state machine, i.e knowing details of settings / defaults / limits / alert levels etc.

1 keeps the ESP very simple
2 makes it more useful allows for some checking locally (i.e. repeat command if it fails)
3. allows greater security and ability to not ask the inverter for some things (i.e. what is this setting might not need to go to the inverter), also add possibility to check command is valid across more range, ie not just that it is a valid command, but that the command makes sense (i.e cant set float voltage above bulk)

I'm still thinking closer to 2.
Especially for examples like @DMIINC where he is using other sensors to validate (for example) the battery state. In the situation that the battery monitor says the charging needs to stop - the inverter (or battery management, system, or relay) should be set to stop charging. Where does this logic live?

Sorry for the rambling - just my thoughts

[volkerjaenisch]

@jblance

'proper' object orientation / abstraction.

Agree fully. Also more commitment to PEP8 would be nice. :-)
ATM I see my role in experimenting if it is possible run the mpp-solar code on the ESP32 at all. So I am fixing obstacles to make a ESP32 port possible: Changing some imports, eliminating dependencies (e.g. ctypes is not supported on ESP32) etc.

Concerning CAN and vehicles:
I am working for Hamm AG leader in steamrollers in the EU. I do the CAN-Bus data monitoring of the machines. These machines have up to 6 CAN-busses (daily growing). Due to the centralized approach with dump sensors the data volume is to much for a single bus. Also the central management system has steadily to less computing power and has to be empowered at a higher pace than desired.
This is not exactly an argument against a centralized system, but it shows the problems that may arise.
Also Nagios, Munin, Puppet are profound examples to show that a centralized approach with dumb sensors/nodes scales not well.

I am in between your options 2 and 3. If we choose option 2 we will have to port a good part of the mpp-solar code to ESP32 anyway. So if I succeed in porting the whole code we can strip the code we will not need.

If we learn that we need to integrate more functionality (option 3) into the ESP32 we can do so since the whole code is already ported.

Another problem with mppsolar service approach, is that is monopolises the serial connection, so if you want to run a command to change a setting you need to stop the service first.

I am not getting your point here? Why does the service has to been stopped if I like to send a command? The ESP32 mpp-solar code will hold the serial port open it's whole lifetime. And it will communicate with the serial port in both directions. I do not see the problem here.

Monopolizing the serial connection is IMHO not an issue. The ESP32 will only be connected to the inverter and has no other function. For other sensors there will be other ESP32 or whatever to collect the data. The central principles in running mpp-solar on the ESP32 are:

• Delegation
• Abstraction
• Complexity hiding
• Portal
  All well known design patterns from OOP-Programing.

Where does this logic live?

That's the really crucial question! I think we need to write a python state machine (Lets call this the SolarSimulation) for the controlling of the whole solar power system (Batteries, Inverters, Relais, current gauges, etc.) as a python package. This package should be agnostic to

• where the data it works on are originating (What type of inverter, what protocol, etc.)
• where the directives it issues are processed (What type of inverter, what protocol, etc.)
  This simulation is IMHO the real task of this project. The simulation will have a lot of degrees of freedom which have to be parametrized. Also it will have to include some kind of solver to optimize the performance of the whole system. So we want to delegate some decision making competence to that simulation. E.G. I like to change my E-vehicle:
• Battery load is at 30%
• It is 2 hours to sunset
• It is partly cloudy
• The laundry machine needs to be run this evening
• Current from the grid currently costs 0.05 EUR/kWh
  So is it better to :
• load the vehicle from the incoming solar current which takes probably to long
• load the vehicle from the battery and do the laundry with the incoming solar power
• load the vehicle from the grid since it is cheap at this time and use all the solar current to load the battery
  and so on. I am unsure how I would decide in this scenario.

To make such an optimization possible one will have to tag a fitness value (in fact it is a function of time) to each parameter (for instance a virtual price tag). Solar power has not a zero price. So the cost from Solar power directly from the panels is cheaper than power from the battery since the batteries have a finite life span which is shorter than that of the panels. Also solar power from the panels has an infinite price if the sun is not shining. Also the price of the battery power goes up if the load of the battery reaches critical limits (to low or to high load).
And the most important thing on this simulation is that we must guaranty the stability of the whole system. E.G. we do not like that the system flaps between decisions every time a small cloud changes the scenario - since the high initial currents after switching a relais are not good for the system (stability, duration).

Then integrations of this package can be produced for any number of systems in need for an intelligent simulation and steering module. At first I would think of HASS since it has the best python integration capabilities and favors exactly this approach (small integration, big python package).

[DMIINC]

Another problem with mppsolar service approach, is that is monopolizes the serial connection, so if you want to run a command to change a setting you need to stop the service first.

I do not understand this either. I am able to send different commands as it is now (switch from grid to batteries, change charging voltage, etc....) while mpp-solar service is running.

@volkerjaenisch excellent write up!!!!

In regards to me wanting everything one machine, well maybe I did not explain myself correctly. My apologies.
Was thinking more from prospect of reliability. PC that I am using to type this message is from 2010. It was built using high quality components and it shows.
Reason I have 3 inverters and 3 Power Walls is simple. Redundancy and ease of servicing if need be. I could have gotten one massive 15KW LF inverter (would have been cheaper by $800) and make one giant power wall. However Murphy and his law will always visit when you are not home and WAF and FAF will go down the drain.

WAF = Wife Acceptance Factor
FAF = Family Acceptance Factor

So question from @jblance is similar as to what I have been pondering on. Should battery SOC be on ESP32 or should ESP32 send raw sensor data to HASS and HASS makes calculation for SOC. Basicaly where does this logic live? What would be better?

[volkerjaenisch]

@jblance Can you give me write permission on this repository, please. I like to upload some graphics to illustrate.

[volkerjaenisch]

@jblance Thanks!
I uploaded a simple sketch of the big picture that I like do discuss and refine with you.

https://github.com/jblance/solarman/blob/master/docs/solar1.pdf

[volkerjaenisch]

@DMIINC Any test results?

My Customer has to buy a ESP32 for his inverter and I have to guide him to flash the thing.
So I am really happy if ANYBODY tests my code in real world.

Cheers
Volker

[DMIINC]

Some commands work and some don't. Limited test

pi@raspberrypi:/dev $ sudo mpp-solar -c QPIGS -d /dev/hidraw0
ac_input_frequency 60.0 Hz
ac_input_voltage 122.3 V
ac_output_active_power 0251 W
ac_output_apparent_power 0256 VA
ac_output_frequency 60.0 Hz
ac_output_load 010 %
ac_output_voltage 122.3 V
battery_capacity 100 %
battery_charging_current 000 A
battery_discharge_current 00000 A
battery_voltage 56.70 V
battery_voltage_from_scc 00.00 V
bus_voltage 226 V
inverter_heat_sink_temperature 0042 Deg_C
is_battery_voltage_to_steady_while_charging 0 True - 1/False - 0
is_charging_on 1 True - 1/False - 0
is_configuration_changed 0 True - 1/False - 0
is_load_on 0 True - 1/False - 0
is_sbu_priority_version_added 0 True - 1/False - 0
is_scc_charging_on 1 True - 1/False - 0
is_scc_firmware_updated 0 True - 1/False - 0
pv_input_current_for_battery 0000 A
pv_input_voltage 000.0 V
unknown_value_in_byte_response 110

pi@raspberrypi:/dev $ sudo mpp-solar -c QID -d /dev/hidraw0
serial_number 92912001100085

pi@raspberrypi:/dev $ sudo mpp-solar -c QPGS0 -d /dev/hidraw0
Traceback (most recent call last):
File "/usr/local/bin/mpp-solar", line 11, in
load_entry_point('mpp-solar==0.3.1', 'console_scripts', 'mpp-solar')()
File "/usr/local/lib/python2.7/dist-packages/mpp_solar-0.3.1-py2.7.egg/mppsolar/init.py", line 79, in main
results = mp.getResponseDict(args.command)
File "/usr/local/lib/python2.7/dist-packages/mpp_solar-0.3.1-py2.7.egg/mppsolar/mpputils.py", line 37, in getResponseDict
return self.inverter.getResponseDict(cmd)
File "/usr/local/lib/python2.7/dist-packages/mpp_solar-0.3.1-py2.7.egg/mppsolar/mppinverter.py", line 208, in getResponseDict
result = self.execute(cmd)
File "/usr/local/lib/python2.7/dist-packages/mpp_solar-0.3.1-py2.7.egg/mppsolar/mppinverter.py", line 343, in execute
return self._doDirectUsbCommand(command)
File "/usr/local/lib/python2.7/dist-packages/mpp_solar-0.3.1-py2.7.egg/mppsolar/mppinverter.py", line 327, in _doDirectUsbCommand
command.setByteResponse(response_line)
File "/usr/local/lib/python2.7/dist-packages/mpp_solar-0.3.1-py2.7.egg/mppsolar/mppcommand.py", line 122, in setByteResponse
self.response_dict = self.getResponseDict()
File "/usr/local/lib/python2.7/dist-packages/mpp_solar-0.3.1-py2.7.egg/mppsolar/mppcommand.py", line 408, in getResponseDict
msgs[key] = [resp_format[2][int(result)], '']
IndexError: list index out of range
pi@raspberrypi:/dev $

[volkerjaenisch]

@DMIINC Thank you so much!
Can you please confirm that the
sudo mpp-solar -c QPGS0 -d /dev/hidraw0
runs clearly with the old version.

Cheers,
Volker

[DMIINC]

you are welcome

yes it does

[DMIINC]

some of the other commands are not recognized either. such as QPIGS2

[PauloSeze]

I’m trying to use Solarman open api to control devices settings. I do have the correct credentials(appId, appSecret, password). My issue is that I can’t manage to get the controlCode request params. This is a required field. There’s nowhere mentioned to retrieve the contolCode on the documentation.

Note :
1 . This interface is a general description of the interface for issuing device control
commands . If your device has implemented device control functions on the platform ,
you need to obtain a further command configuration table before operating this interface .
2 . The maximum number of requests per minute for a single OpenApi account to call
the interface : 50