FEMS Backports 2023-03 (OpenEMS#2593)

* ToU: optionally show Grid-Buy in history chart

  - Add third y-axis (second right y-axis) for GridBuy

* AppCenter: fixStateOfCharge App

  - added app
  - added to scheduler
  - created formly datetime builder

* Tibber: implement handling for rate limit (HTTP code 429)

  - A special case-handler is added that handles http status error 429 and schedules next execution 12 hours later.

* UI: Change chartjs resolution for mobile view

  - Use bigger resolution for day and week for smartphone to enable better readability

  - Technical description...
| View | smartphone or desktop | Aggregation
| --- | --- | --- |
| day | Smartphone | 20 min aggregated Data |
| day | Desktop | 5 min aggregated Data |
| week | Smartphone | 8 hours aggregated Data |
| week | Desktop | 1 hour aggregated Data |

- Backend query has to be adjusted due to wrong timestamps!

* CI: improvements

  - Add [WOODPECKER_ENVIRONMENT](https://woodpecker-ci.org/docs/usage/environment#global-environment-variables);
  - Run checkstyle before build

* AppCenter: Add definition to set min Role to delete an App & validation

  - Add Permission Role to set which Role is required to delete an App Instance
  - Currently CanSeeRole == CanDeleteRole so there is no UI change neccesarry to add a special request to hide/show/disable delete button.

* AppCenter Tibber: Access Token will be displayed as "xxx" if set

  - If a access token is set in the Tibber Provider it will be displayed as "xxx" and shown directly as a clear text token and can not be accessed from the ui in clear text.
  - If the token wont get modified and stays at "xxx" it wont get updated and the old token remains in the provider configuration.

* FENECON Home Battery: Calculate number of towers

* UI: Implement system reboot

  - adding maintenance to /system
  - Available only for admins at first
  - Systemupdate renamed to system now including maintenance

* Edge/UI: add persistencePriority to EdgeConfig

  - Show warning in /channels if global persistence priority is bigger than channel persistence priority

* Controller ESS Cycle: fix Standby Time

  - Updating the last changed state time onExit approach was using wrong.
  - The last changed time is taking actually from previous state.

  - For an instance:
    - As the controller started; goes first to Undefined State then onExit set the lastStateChangeTime.
    - Aftwerwards keep charges the system till it gets fully charged.

    - At the end of the 'START_CHARGE' state, if the fully charged conditions fullfilled, controller tries to wait before it switches the state, according to the given 'standbyTime' in config.

    - However, in context 'waitForChangeState' method proofs whether the wait time achieved or not.
    - But here; 'getLastStateChangeTime' method gives the time that actually what is in 'Undefined-onExit' set.

    - So, if the charge process takes more time than given 'standbyTime', state switching will be approved and state will change to 'Continue_Discharge' immediately.

* UI: README changes

  - correcting wrong Ports

* UI: Bugfix: Pickdatepopover not showing allowed historyPeriods

  - History Popover only showing custom period

* UI: Extend unittest for pickdatepopover

  - Create type for periodString values

* UI: Style improvements

  - Align Buttons to right side in System view for consistency
    - Adjusting the EMS System View to align line with single button to the right
    - Cleanup alertController style adjustments

* UI: Updated redirect link to update page

  - Updated redirect link to update page from AppCenter and live storage modal

* Time-of-Use: improvements

  - Reduce search space, by calculating later periods as hours and not quarters
  - Improve calculation of charge power in CHARGE_GRID mode
  - More and smarter Genotypes in initialPopulation
  - Improved error handling
  - New EnergyFlow calculation + fix previsou calculation bugs
  - Be default always apply §14a EnWG limit of 4.2 kW (configurable)

* GoodWe: Disable ShadowManagement

  - During the time that GoodWe's shadow management is active, we will not be able to discharge or charge the battery. During this time, the DC connection cannot work as usual.

* HeatingElement: Changed Persistence Priority for LEVEL to HIGH

  - Set persistencePriority to HIGH for LEVEL for UI History Chart
  - removed unused UNDEFINED constant

* Time-of-Use: bugfix §14a EnWG

  - Wrong sign; mixed up positive/negatve values

* Tibber: hide "homes" filter in App

  - "Homes" filter is ignored for parsing if there is only one home.
  - Added a check box to selct if there are multiple homes. Only then the filter field is displayed to enter.

* UI/Time-of-Use: optionally show Grid-Buy in Live schedule chart

  - Add second y-axis for GridBuy in StatePriceChart.

* UI EVCS: fix possible NPE

  - This currently happens on fems4 with EVCS test; causes a lot of logs in Backend log
  - Validate `evcsComponent` and `this.controller` are actually available

* Tibber: differentiate warnings based on HTTP status codes

  - 1. Error code 400 is considered for authentication failure.
  - 2. Added two new warn channels to differentiate between client and server error.
  - 3. added new test cases to get next run.

* UI/AppCenter: Add popover to notify costumer about unsued keys

  - Add popover when entering AppCenter and a registered key is available to notify the costumer to redeem it.
  - Also added badge how many keys are available

* Time-of-Use: fix duplicate data in GetScheduleResponse

  - Removed duplicate data while merging past and future schedule data.

* UI: Add footer for EMS-Nr., installed version and comment

* UI & Time-of-Use

  - fix Log-Output, distinguish HOUR/QUARTERLY
  - change content order in footer
  - always add `edgeShortName` to browser title
  - fix unsetting browser-title in overview

* UI: remove ion item line in footer

  - Remove line generated at bottom of ion-item in footer

* UI: Link footer version to /system

* Time-of-Use: bugfix for mixing up HOUR/QUARTER in JSONRPC-Response and Log

* Backend-Improvements

---------

Co-authored-by: Sagar Venu <[email protected]>
Co-authored-by: Lukas Rieger <[email protected]>
Co-authored-by: Michael Grill <[email protected]>
Co-authored-by: Kai Jeschek <[email protected]>
Co-authored-by: Hueseyin Sahutoglu <[email protected]>
Co-authored-by: Lóránt Meszlényi <[email protected]>
Co-authored-by: Sebastian Asen <[email protected]>
Co-authored-by: Anas Shetla <[email protected]>
Co-authored-by: Stefan Feilmeier <[email protected]>
Co-authored-by: Maximilian Lang <[email protected]>

build.gradle

@@ -78,6 +78,7 @@ subprojects {
 			reports {
 				xml.required = false
 				html.required = false
+				sarif.required = false
 			}
 			minHeapSize = "256m"
 			maxHeapSize = "1g"
@@ -104,6 +105,17 @@ task cleanEdge() {
 	}
 }
 
+task checkstyleAll() {
+	group 'OpenEMS-Build'
+	description 'Checkstyle all bundles'
+
+	subprojects.each { proj ->
+		if (proj.tasks.findAll { it.name == 'checkstyleMain' }) {
+			dependsOn(proj.checkstyleMain)
+		}
+	}
+}
+
 task cleanBackend() {
 	group 'OpenEMS-Build'
 	description 'Clean all Backend-Bundles'

io.openems.backend.alerting/src/io/openems/backend/alerting/handler/OfflineEdgeHandler.java

@@ -29,7 +29,8 @@ public class OfflineEdgeHandler implements Handler<OfflineEdgeMessage> {
 	// Definition of unrealistically high values for messages sent simultaneously
 	public static final int MAX_SIMULTANEOUS_MSGS = 500;
 	public static final int MAX_SIMULTANEOUS_EDGES = 1000;
-
+	public static final int EDGE_REBOOT_MINUTES = 5;
+
 	private final Logger log = LoggerFactory.getLogger(OfflineEdgeHandler.class);
 
 	private final int initialDelay; // in Minutes
@@ -41,7 +42,7 @@ public class OfflineEdgeHandler implements Handler<OfflineEdgeMessage> {
 
 	private TimedTask initMetadata;
 	private TimedExecutor timeService;
-
+	
 	public OfflineEdgeHandler(MessageSchedulerService mss, TimedExecutor timeService, Mailer mailer, Metadata metadata,
 			int initialDelay) {
 		this.mailer = mailer;
@@ -69,6 +70,9 @@ public void stop() {
 	public void send(ZonedDateTime sentAt, List<OfflineEdgeMessage> pack) {
 		// Ensure Edge is still offline before sending mail.
 		pack.removeIf((msg) -> !this.isEdgeOffline(msg.getEdgeId()));
+		if (pack.isEmpty()) {
+			return;
+		}
 
 		var params = JsonUtils.generateJsonArray(pack, OfflineEdgeMessage::getParams);
 
@@ -139,7 +143,7 @@ private void checkMetadata() {
 	private boolean isValidEdge(Edge edge) {
 		var invalid = edge.getLastmessage() == null // was never online
 				|| edge.getLastmessage() //
-						.isBefore(ZonedDateTime.now().minusWeeks(1)); // already offline for a week
+						.isBefore(this.timeService.now().minusWeeks(1)); // already offline for a week
 		return !invalid;
 	}
 
@@ -152,7 +156,7 @@ private boolean isValidEdge(Edge edge) {
 	protected OfflineEdgeMessage getEdgeMessage(Edge edge) {
 		if (edge == null || edge.getId() == null) {
 			this.log.warn("Called method getEdgeMessage with " //
-					+ edge == null ? "Edge{null}" : "Edge{id=null}");
+					+ (edge == null ? "Edge{null}" : "Edge{id=null}"));
 			return null;
 		}
 		try {
@@ -233,7 +237,11 @@ private void handleOnSetOnline(EventReader event) {
 				if (isOnline) {
 					this.tryRemoveEdge(edge);
 				} else {
-					this.tryAddEdge(edge);
+					this.timeService.schedule(this.timeService.now().plusMinutes(EDGE_REBOOT_MINUTES), t -> {
+						if (edge.isOffline()) {
+							this.tryAddEdge(edge);
+						}
+					});
 				}
 			}
 		} else {
@@ -248,7 +256,7 @@ public Consumer<EventReader> getEventHandler(String eventTopic) {
 			yield this::handleOnSetOnline;
 
 		case Metadata.Events.AFTER_IS_INITIALIZED:
-			yield this::handleMetadataAfterInitialize;
+			yield this::handleMetadataAfterInitialize; 
 
 		default:
 			yield null;

io.openems.backend.alerting/src/io/openems/backend/alerting/handler/SumStateHandler.java

@@ -108,7 +108,7 @@ private boolean isSevere(Level level) {
 	protected SumStateMessage getEdgeMessage(Edge edge, Level sumState) throws OpenemsException {
 		if (edge == null || edge.getId() == null) {
 			this.log.warn("Called method SumStateHandler.getEdgeMessage with " //
-					+ edge == null ? "Edge{null}" : "Edge{id=null}");
+					+ (edge == null ? "Edge{null}" : "Edge{id=null}"));
 			return null;
 		} else if (edge.isOffline()) {
 			this.log.warn("Called method SumStateHandler.getEdgeMessage with offline" //

io.openems.backend.alerting/test/io/openems/backend/alerting/OfflineEdgeAlertingTest.java

@@ -14,6 +14,7 @@
 import io.openems.backend.alerting.Dummy.AlertingMetadataImpl;
 import io.openems.backend.alerting.Dummy.MailerImpl;
 import io.openems.backend.alerting.Dummy.TimeLeapMinuteTimer;
+import io.openems.backend.alerting.handler.OfflineEdgeHandler;
 import io.openems.backend.alerting.scheduler.Scheduler;
 import io.openems.backend.common.alerting.OfflineEdgeAlertingSetting;
 import io.openems.backend.common.metadata.Edge;
@@ -128,7 +129,7 @@ public void integrationTest() {
 		env.setOnline("edge03", false);
 		env.setOnline("edge04", false);
 		env.setOnline("edge07", false);
-		env.timer.leap(1); /* inaccuracy */
+		env.timer.leap(1 + OfflineEdgeHandler.EDGE_REBOOT_MINUTES); /* inaccuracy */
 
 		/* edge05[user03], edge03[user01,user02], edge07[user04] */
 		assertEquals(3, env.scheduler.getScheduledMsgsCount());

io.openems.backend.alerting/test/io/openems/backend/alerting/handler/TestOfflineEdgeHandler.java

@@ -33,15 +33,15 @@ public class TestOfflineEdgeHandler {
 	public void testGetterSetter() {
 		final var mss = new MessageSchedulerServiceImpl();
 		final var meta = new SimpleMetadataImpl();
-		final var timer = new MinuteTimer(null);
+		final var timer = new MinuteTimer(Clock.systemUTC());
 		final var handler = new OfflineEdgeHandler(mss, timer, null, meta, 1);
 		assertEquals(OfflineEdgeMessage.class, handler.getGeneric());
 	}
 
 	@Test
 	public void testActivate() {
 		final var service = new MessageSchedulerServiceImpl();
-		final var timer = new MinuteTimer(null);
+		final var timer = new MinuteTimer(Clock.systemUTC());
 		var handler = new OfflineEdgeHandler(service, timer, null, new SimpleMetadataImpl(), 1);
 		assertEquals(1, service.msgScheduler.size());
 		assertTrue(service.msgScheduler.get(0).isFor(handler));
@@ -78,7 +78,7 @@ public void send() {
 	public void checkmetadata() {
 		final var metadata = Utility.getTestMetadata();
 		final var msgsch = new MessageSchedulerServiceImpl();
-		final var timer = new MinuteTimer(null);
+		final var timer = new MinuteTimer(Clock.systemUTC());
 		final var handler = new OfflineEdgeHandler(msgsch, timer, null, metadata, 0);
 
 		final var expected = (int) metadata.getOfflineSettings().values().stream() //
@@ -105,7 +105,7 @@ public void checkMetadataEmergencyStop() {
 		final var msgMeta = new Utility.ToManyMsgsMetadata();
 		final var msgMsgsch = new MessageSchedulerServiceImpl();
 		final var msgCount = new AtomicInteger();
-		final var timer = new MinuteTimer(null);
+		final var timer = new MinuteTimer(Clock.systemUTC());
 		msgMeta.getAllOfflineEdges().stream().map(e -> msgMeta.getEdgeOfflineAlertingSettings(e.getId())).forEach(e -> {
 			e.forEach(s -> {
 				if (s.lastNotification().plusMinutes(s.delay()).isBefore(Utility.now)) {

io.openems.backend.common/src/io/openems/backend/common/alerting/UserAlertingSettings.java

@@ -11,6 +11,6 @@ public UserAlertingSettings(String userLogin, int offlineEdgeDelay, int faultSta
 	}
 
 	public UserAlertingSettings(String deviceId, String userLogin) {
-		this(deviceId, userLogin, 1440, 1440, 1440, null, null);
+		this(deviceId, userLogin, 0, 0, 0, null, null);
 	}
 }

io.openems.backend.common/src/io/openems/backend/common/metadata/Edge.java

@@ -178,6 +178,7 @@ private void setVersion(SemanticVersion version, boolean emitEvent) {
 
 			EventBuilder.from(this.parent.getEventAdmin(), Events.ON_SET_VERSION) //
 					.addArg(Events.OnSetVersion.EDGE, this) //
+					.addArg(Events.OnSetVersion.OLD_VERSION, oldVersion) //
 					.addArg(Events.OnSetVersion.VERSION, version) //
 					.send();
 		}
@@ -255,6 +256,7 @@ public static final class OnSetOnline {
 
 		public static final class OnSetVersion {
 			public static final String EDGE = "Edge:Edge";
+			public static final String OLD_VERSION = "OldVersion:SemanticVersion";
 			public static final String VERSION = "Version:SemanticVersion";
 		}
 

io.openems.backend.core/src/io/openems/backend/core/jsonrpcrequesthandler/EdgeRpcRequestHandler.java

@@ -9,17 +9,12 @@
 import io.openems.common.exceptions.OpenemsError.OpenemsNamedException;
 import io.openems.common.jsonrpc.base.JsonrpcResponseSuccess;
 import io.openems.common.jsonrpc.request.AppCenterRequest;
-import io.openems.common.jsonrpc.request.ComponentJsonApiRequest;
-import io.openems.common.jsonrpc.request.CreateComponentConfigRequest;
-import io.openems.common.jsonrpc.request.DeleteComponentConfigRequest;
 import io.openems.common.jsonrpc.request.EdgeRpcRequest;
 import io.openems.common.jsonrpc.request.GetEdgeConfigRequest;
 import io.openems.common.jsonrpc.request.QueryHistoricTimeseriesDataRequest;
 import io.openems.common.jsonrpc.request.QueryHistoricTimeseriesEnergyPerPeriodRequest;
 import io.openems.common.jsonrpc.request.QueryHistoricTimeseriesEnergyRequest;
 import io.openems.common.jsonrpc.request.QueryHistoricTimeseriesExportXlxsRequest;
-import io.openems.common.jsonrpc.request.SetChannelValueRequest;
-import io.openems.common.jsonrpc.request.UpdateComponentConfigRequest;
 import io.openems.common.jsonrpc.response.EdgeRpcResponse;
 import io.openems.common.jsonrpc.response.GetEdgeConfigResponse;
 import io.openems.common.jsonrpc.response.QueryHistoricTimeseriesDataResponse;
@@ -86,31 +81,8 @@ protected CompletableFuture<EdgeRpcResponse> handleRequest(User user, UUID messa
 			resultFuture = this.handleGetEdgeConfigRequest(edgeId, user, GetEdgeConfigRequest.from(request));
 			break;
 
-		case CreateComponentConfigRequest.METHOD:
-			resultFuture = this.handleCreateComponentConfigRequest(edgeId, user,
-					CreateComponentConfigRequest.from(request));
-			break;
-
-		case UpdateComponentConfigRequest.METHOD:
-			resultFuture = this.handleUpdateComponentConfigRequest(edgeId, user,
-					UpdateComponentConfigRequest.from(request));
-			break;
-
-		case DeleteComponentConfigRequest.METHOD:
-			resultFuture = this.handleDeleteComponentConfigRequest(edgeId, user,
-					DeleteComponentConfigRequest.from(request));
-			break;
-
-		case SetChannelValueRequest.METHOD:
-			resultFuture = this.handleSetChannelValueRequest(edgeId, user, SetChannelValueRequest.from(request));
-			break;
-
-		case ComponentJsonApiRequest.METHOD:
-			resultFuture = this.handleComponentJsonApiRequest(edgeId, user, ComponentJsonApiRequest.from(request));
-			break;
-
 		default:
-			throw OpenemsError.JSONRPC_UNHANDLED_METHOD.exception(request.getMethod());
+			resultFuture = this.parent.edgeWebsocket.send(edgeId, user, request);
 		}
 
 		// Wrap reply in EdgeRpcResponse
@@ -214,83 +186,4 @@ private CompletableFuture<JsonrpcResponseSuccess> handleGetEdgeConfigRequest(Str
 		return CompletableFuture.completedFuture(new GetEdgeConfigResponse(request.getId(), config));
 	}
 
-	/**
-	 * Handles a {@link CreateComponentConfigRequest}.
-	 *
-	 * @param edgeId  the Edge-ID
-	 * @param user    the {@link User} - Installer-level required
-	 * @param request the {@link CreateComponentConfigRequest}
-	 * @return the Future JSON-RPC Response
-	 * @throws OpenemsNamedException on error
-	 */
-	private CompletableFuture<JsonrpcResponseSuccess> handleCreateComponentConfigRequest(String edgeId, User user,
-			CreateComponentConfigRequest request) throws OpenemsNamedException {
-		user.assertEdgeRoleIsAtLeast(CreateComponentConfigRequest.METHOD, edgeId, Role.INSTALLER);
-
-		return this.parent.edgeWebsocket.send(edgeId, user, request);
-	}
-
-	/**
-	 * Handles a {@link UpdateComponentConfigRequest}.
-	 *
-	 * @param edgeId  the Edge-ID
-	 * @param user    the {@link User} - Installer-level required
-	 * @param request the {@link UpdateComponentConfigRequest}
-	 * @return the Future JSON-RPC Response
-	 * @throws OpenemsNamedException on error
-	 */
-	private CompletableFuture<JsonrpcResponseSuccess> handleUpdateComponentConfigRequest(String edgeId, User user,
-			UpdateComponentConfigRequest request) throws OpenemsNamedException {
-		user.assertEdgeRoleIsAtLeast(UpdateComponentConfigRequest.METHOD, edgeId, Role.OWNER);
-
-		return this.parent.edgeWebsocket.send(edgeId, user, request);
-	}
-
-	/**
-	 * Handles a {@link DeleteComponentConfigRequest}.
-	 *
-	 * @param edgeId  the Edge-ID
-	 * @param user    the {@link User} - Installer-level required
-	 * @param request the {@link DeleteComponentConfigRequest}
-	 * @return the Future JSON-RPC Response
-	 * @throws OpenemsNamedException on error
-	 */
-	private CompletableFuture<JsonrpcResponseSuccess> handleDeleteComponentConfigRequest(String edgeId, User user,
-			DeleteComponentConfigRequest request) throws OpenemsNamedException {
-		user.assertEdgeRoleIsAtLeast(DeleteComponentConfigRequest.METHOD, edgeId, Role.INSTALLER);
-
-		return this.parent.edgeWebsocket.send(edgeId, user, request);
-	}
-
-	/**
-	 * Handles a {@link SetChannelValueRequest}.
-	 *
-	 * @param edgeId  the Edge-ID
-	 * @param user    the {@link User}
-	 * @param request the {@link SetChannelValueRequest}
-	 * @return the Future JSON-RPC Response
-	 * @throws OpenemsNamedException on error
-	 */
-	private CompletableFuture<JsonrpcResponseSuccess> handleSetChannelValueRequest(String edgeId, User user,
-			SetChannelValueRequest request) throws OpenemsNamedException {
-		user.assertEdgeRoleIsAtLeast(SetChannelValueRequest.METHOD, edgeId, Role.ADMIN);
-
-		return this.parent.edgeWebsocket.send(edgeId, user, request);
-	}
-
-	/**
-	 * Handles a {@link UpdateComponentConfigRequest}.
-	 *
-	 * @param edgeId                  the Edge-ID
-	 * @param user                    the {@link User} - Guest-level required
-	 * @param componentJsonApiRequest the {@link ComponentJsonApiRequest}
-	 * @return the Future JSON-RPC Response
-	 * @throws OpenemsNamedException on error
-	 */
-	private CompletableFuture<JsonrpcResponseSuccess> handleComponentJsonApiRequest(String edgeId, User user,
-			ComponentJsonApiRequest componentJsonApiRequest) throws OpenemsNamedException {
-		user.assertEdgeRoleIsAtLeast(ComponentJsonApiRequest.METHOD, edgeId, Role.GUEST);
-
-		return this.parent.edgeWebsocket.send(edgeId, user, componentJsonApiRequest);
-	}
 }

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/Config.java

@@ -4,6 +4,7 @@
 import org.osgi.service.metatype.annotations.ObjectClassDefinition;
 
 import io.openems.backend.metadata.odoo.odoo.Protocol;
+import io.openems.common.websocket.AbstractWebsocketServer.DebugMode;
 
 @ObjectClassDefinition(//
 		name = "Metadata.Odoo", //
@@ -40,6 +41,15 @@
 	@AttributeDefinition(name = "Database", description = "The database name")
 	String database();
 
+	@AttributeDefinition(name = "Number of Threads", description = "Pool-Size: the number of threads dedicated to handle the tasks")
+	int poolSize() default 30;
+
+	@AttributeDefinition(name = "Number of Threads", description = "Pool-Size: the maximum number of concurrent connections")
+	int pgConnectionPoolSize() default 40;
+
+	@AttributeDefinition(name = "Debug Mode", description = "Activates the debug mode")
+	DebugMode debugMode() default DebugMode.OFF;
+
 	String webconsole_configurationFactory_nameHint() default "Metadata.Odoo";
 
 }

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/Field.java

@@ -222,7 +222,6 @@ public enum EdgeDeviceUserRole implements Field {
 		DEVICE_ODOO_ID("device_id", false), //
 		USER_ODOO_ID("user_id", true), //
 		ROLE("role", false), //
-		TIME_TO_WAIT("time_to_wait", true), //
 		LAST_NOTIFICATION("last_notification", true), //
 		; //