Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connect IO-Link Sensor Data into StreamPipes #1592

Closed
tenthe opened this issue May 25, 2023 · 1 comment
Closed

Connect IO-Link Sensor Data into StreamPipes #1592

tenthe opened this issue May 25, 2023 · 1 comment
Labels
connect Related to the `connect` module (adapters) enhancement New feature or request
Milestone
0.93.0

Comments

@tenthe
Copy link
Contributor

tenthe commented May 25, 2023

Body

Summary

The IO-Link Master (AL1350) has an IoT interface to transmit sensor data over MQTT. As a user, I would like to have an adapter that connects IO-Link sensor data into StreamPipes.

Details

The IO-Link Master can be configured to transmit JSON events to an MQTT broker. These JSON events contain meta-information and the values of the sensors connected to the port. However, the sensor values are encoded and require decoding for each sensor type.

The challenge we need to address is that an IO-Link Master can have different sensor types connected, but in StreamPipes, a fixed event schema is required for each data stream.

Proposed Solution

To tackle this challenge, I suggest the following solution:

  • User Input:

    • MQTT configuration: Allow users to configure the MQTT connection settings.
    • Select IO Link Ports: Enable users to select the IO Link ports they want to integrate with StreamPipes.
    • Select sensor type: Provide an option for users to select the sensor type connected to each port.

  • Output Schema:

    • The output schema should be dependent on the selected sensor type. Each sensor type may have a different event schema in StreamPipes.

Additional Information

There are multiple options available for mapping the MQTT event to the StreamPipes event. The suggested solution provides flexibility to users, allowing them to configure the MQTT settings and the sensor type.

By implementing this adapter, users will be able to seamlessly integrate IO-Link sensor data into StreamPipes, providing a unified and consistent event schema for analysis and processing.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.
@bossenti bossenti added enhancement New feature or request connect Related to the `connect` module (adapters) labels May 25, 2023
@bossenti bossenti added this to the 0.93.0 milestone May 25, 2023
tenthe added a commit that referenced this issue May 31, 2023
@tenthe
[#1592] Add first version for an io link adapter
b04ae11
This was referenced May 31, 2023
Closed
Merged
tenthe added a commit that referenced this issue Jun 5, 2023
@tenthe
[#1592] Update API of ifm adapter
296d967
tenthe added a commit that referenced this issue Jun 5, 2023
@tenthe
[#1592] Add io link master id to event
94b257b
tenthe added a commit that referenced this issue Jun 5, 2023
@tenthe
[#1592] Provide documentation for io link adapter
88a2a29
tenthe added a commit that referenced this issue Jun 5, 2023
@tenthe @dominikriemer
@obermeier @bossenti @dependabot
Sp 1592 (#1647)
50199e0 
* [hotfix] Remove set adapters from extensions

* [hotfix] Remove old resource files

* [#1289] Remove data set model

* [#1289] Remove data set model from ui

* [#1289] Remove `adapterType` and fix tests

* [#1289] Remove class `SpDataSet`

* [#1289] Remove legacy code in class `AdapterDescription`

* [#1289] Remove class `AdapterStreamDescription`

* [#1289] Remove class `ProtocolStreamDescription`

* [hotfix] Remove class `AdapterDescription`

* [#1289] Very first version for new adapter interface

* [#1289] Started to migrate the API

* [#1289] Deactivate old protocol API

* [#1289] Remove old models from the ui

* [#1289] Remove old models from the ui

* [#1289] Fix icons for adapters

* [#1289] Fix minor bug

* [#1289] Fix maven build

* [#1289] WIP refactor `AdapterConfigurationBuilder`

* [#1289] Fix AdapterConfigurationBuilder

* [#1289] Fix AdapterConfigurationBuilder

* [#1289] Guess schema works now for NewMachineDataSimulatorAdapter

* [#1289] Add start adapter

* [#1289] First version of new machine data simulator works

* [#1289] Working on the first new parser

* [#1289] First version of declare parser description

* [WIP] Add adapter configuration test

* [#1289] First generic adapter works now

* [#1289] Add parser exception to json parser

* [#1289] Add csv parser

* [#1289] Change implementation for json parser

* [#1289] Add json array parsers

* Add runtime context to adapters (#1290)

* Extract interface of  (#1290)

* Refactor OPC-UA adapter to new model (#1290)

* Refactor S7 adapter (#1290)

* Refactor Modbus adapter (#1290)

* Refactor Ros adapter (#1290)

* [#1289] Add geo json parser

* [#1289] Add xml parser

* [#1289] Add image parser

* Refactor Kafka adapter (#1290)

* Refactor MQTT adapter (#1290)

* Refactor Nats adapter (#1290)

* Refactor Pulsar adapter (#1290)

* Refactor RocketMQ adapter (#1290)

* Refactor TubeMQ adapter (#1290)

* Refactor HTTP server adapter (#1290)

* Refactor HTTP Stream adapter (#1290)

* [WIP] Refactor Parser API

* [WIP] Refactor Parser API

* [#1289] Fix file stream e2e test

* [#1289] Fix tests and csv parser for timestamps

* [#1289] Fix e2e connect tests

* [hotfix] Fix build

* [hotfix] Deactivate Mqtt integration test

* [#1289] Remove old format implementation from backend

* [#1289] Remove old format implementation from backend

* [#1289] Remove old format implementation from ui

* [#1289] Fix kafka e2e test

* Add migration script for specific adapters (#1289)

* Add initial version to migrate generic adapters (#1289)

* Modify format migrators (#1289)

* [#1289] Migrate influxdb to new interface

* [#1289] Fix ROS bridge adapter

* [#1289] Rename specific-adapter-configuration.component

* [#1289] Remove old data model

* [#1289] Fix mqtt integration test

* [#1289] Update Pulsar integration test

* [#1289] Remove DebugAdapterSink which is no longer needed for integration tests

* [#1289] Fix kafka adapter on restart

* [#1289] Fix migrations for all parser

* [hotfix] Fix image parser for adapters

* [hotfix] Change StreamPipes version to fix build problem

* [#1289] Move adapters

* [hotfix] Remove duplicate licence headers

* [hotfix] Fix conversion of pipeline element templates

* Move PullAdapter to extensions-management and api modules (#1289)

* Refactor structure of extensions-api, add supplier to adapter interface (#1581)

* Rename AdapterInterface to StreamPipesAdapter (#1590)

* [hotfix] Fix checkstyle issues

* [hotfix] Fix checkstyle issue

* [hotfix] Fix checkstyle issue

* [hotfix] Fix checkstyle

* Migrate ISS adapter and Flic adapter (#1289)

* Migrate connect adapters to new model (#1289)

* Properly apply configs to influx adapter (#1289)

* [hotfix] Fix checkstyle

* fix: minor adaptions

* fix: remove data set artifacts

* fix: minor fixes

* [hotfix] Regenerate typescript model

* [#1592] Add first version for an io link adapter

* Refactor data processor and sink API (#1632) (#1633)

* chore: mark streampipes-wrapper-python as deprecated (#1623)

* Bump types-requests in /streampipes-client-python (#1588)

Bumps [types-requests](https://github.com/python/typeshed) from 2.30.0.0 to 2.31.0.0.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump mkdocstrings[python] in /streampipes-client-python (#1631)

* chore: add missing support of NATS as messaging protocol (#1628)

* Bump spring-boot.version from 3.0.6 to 3.1.0 (#1595)

Bumps `spring-boot.version` from 3.0.6 to 3.1.0.

Updates `spring-boot-properties-migrator` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-jersey` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-jetty` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-oauth2-client` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-undertow` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-web` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-maven-plugin` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

---
updated-dependencies:
- dependency-name: org.springframework.boot:spring-boot-properties-migrator
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-jersey
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-jetty
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-oauth2-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-undertow
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-web
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-maven-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Refactor data processor and sink API (#1632)

* Cleanup pom file (#1632)

* Migrate processors to API (#1632)

* [hotfix] Add runtime provider, update pom

* Fix bug in standalone wrapper (#1632)

* Fix instance provision in compatibility layer (#1632)

* [hotfix] Disable Maven cache in workflow file

* [hotfix] Fix failing build by reverting Flink version

* [hotfix] Add proper time selection to another cypress test (#1641)

* Fix bug in aggregation flink module

* Properly return output source and schema info

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Tim <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: remove Plc4xPassiveAdapter

* Add implementation for schema guessing from the example file in the HTTP

resolves #1637
fixes #1518 for streampipes-extensions

* [#1592] Add io link master id to event

* [#1592] Provide documentation for io link adapter

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Dominik Riemer <[email protected]>
Co-authored-by: Stefan Obermeier <[email protected]>
Co-authored-by: bossenti <[email protected]>
Co-authored-by: Tim <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
jjbiggins pushed a commit to jjbiggins/streampipes that referenced this issue Jul 5, 2023
@tenthe @dominikriemer
@obermeier @bossenti @dependabot @jjbiggins
Sp 1592 (apache#1647)
181794c 
* [hotfix] Remove set adapters from extensions

* [hotfix] Remove old resource files

* [apache#1289] Remove data set model

* [apache#1289] Remove data set model from ui

* [apache#1289] Remove `adapterType` and fix tests

* [apache#1289] Remove class `SpDataSet`

* [apache#1289] Remove legacy code in class `AdapterDescription`

* [apache#1289] Remove class `AdapterStreamDescription`

* [apache#1289] Remove class `ProtocolStreamDescription`

* [hotfix] Remove class `AdapterDescription`

* [apache#1289] Very first version for new adapter interface

* [apache#1289] Started to migrate the API

* [apache#1289] Deactivate old protocol API

* [apache#1289] Remove old models from the ui

* [apache#1289] Remove old models from the ui

* [apache#1289] Fix icons for adapters

* [apache#1289] Fix minor bug

* [apache#1289] Fix maven build

* [apache#1289] WIP refactor `AdapterConfigurationBuilder`

* [apache#1289] Fix AdapterConfigurationBuilder

* [apache#1289] Fix AdapterConfigurationBuilder

* [apache#1289] Guess schema works now for NewMachineDataSimulatorAdapter

* [apache#1289] Add start adapter

* [apache#1289] First version of new machine data simulator works

* [apache#1289] Working on the first new parser

* [apache#1289] First version of declare parser description

* [WIP] Add adapter configuration test

* [apache#1289] First generic adapter works now

* [apache#1289] Add parser exception to json parser

* [apache#1289] Add csv parser

* [apache#1289] Change implementation for json parser

* [apache#1289] Add json array parsers

* Add runtime context to adapters (apache#1290)

* Extract interface of  (apache#1290)

* Refactor OPC-UA adapter to new model (apache#1290)

* Refactor S7 adapter (apache#1290)

* Refactor Modbus adapter (apache#1290)

* Refactor Ros adapter (apache#1290)

* [apache#1289] Add geo json parser

* [apache#1289] Add xml parser

* [apache#1289] Add image parser

* Refactor Kafka adapter (apache#1290)

* Refactor MQTT adapter (apache#1290)

* Refactor Nats adapter (apache#1290)

* Refactor Pulsar adapter (apache#1290)

* Refactor RocketMQ adapter (apache#1290)

* Refactor TubeMQ adapter (apache#1290)

* Refactor HTTP server adapter (apache#1290)

* Refactor HTTP Stream adapter (apache#1290)

* [WIP] Refactor Parser API

* [WIP] Refactor Parser API

* [apache#1289] Fix file stream e2e test

* [apache#1289] Fix tests and csv parser for timestamps

* [apache#1289] Fix e2e connect tests

* [hotfix] Fix build

* [hotfix] Deactivate Mqtt integration test

* [apache#1289] Remove old format implementation from backend

* [apache#1289] Remove old format implementation from backend

* [apache#1289] Remove old format implementation from ui

* [apache#1289] Fix kafka e2e test

* Add migration script for specific adapters (apache#1289)

* Add initial version to migrate generic adapters (apache#1289)

* Modify format migrators (apache#1289)

* [apache#1289] Migrate influxdb to new interface

* [apache#1289] Fix ROS bridge adapter

* [apache#1289] Rename specific-adapter-configuration.component

* [apache#1289] Remove old data model

* [apache#1289] Fix mqtt integration test

* [apache#1289] Update Pulsar integration test

* [apache#1289] Remove DebugAdapterSink which is no longer needed for integration tests

* [apache#1289] Fix kafka adapter on restart

* [apache#1289] Fix migrations for all parser

* [hotfix] Fix image parser for adapters

* [hotfix] Change StreamPipes version to fix build problem

* [apache#1289] Move adapters

* [hotfix] Remove duplicate licence headers

* [hotfix] Fix conversion of pipeline element templates

* Move PullAdapter to extensions-management and api modules (apache#1289)

* Refactor structure of extensions-api, add supplier to adapter interface (apache#1581)

* Rename AdapterInterface to StreamPipesAdapter (apache#1590)

* [hotfix] Fix checkstyle issues

* [hotfix] Fix checkstyle issue

* [hotfix] Fix checkstyle issue

* [hotfix] Fix checkstyle

* Migrate ISS adapter and Flic adapter (apache#1289)

* Migrate connect adapters to new model (apache#1289)

* Properly apply configs to influx adapter (apache#1289)

* [hotfix] Fix checkstyle

* fix: minor adaptions

* fix: remove data set artifacts

* fix: minor fixes

* [hotfix] Regenerate typescript model

* [apache#1592] Add first version for an io link adapter

* Refactor data processor and sink API (apache#1632) (apache#1633)

* chore: mark streampipes-wrapper-python as deprecated (apache#1623)

* Bump types-requests in /streampipes-client-python (apache#1588)

Bumps [types-requests](https://github.com/python/typeshed) from 2.30.0.0 to 2.31.0.0.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump mkdocstrings[python] in /streampipes-client-python (apache#1631)

* chore: add missing support of NATS as messaging protocol (apache#1628)

* Bump spring-boot.version from 3.0.6 to 3.1.0 (apache#1595)

Bumps `spring-boot.version` from 3.0.6 to 3.1.0.

Updates `spring-boot-properties-migrator` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-jersey` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-jetty` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-oauth2-client` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-undertow` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-web` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-maven-plugin` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

---
updated-dependencies:
- dependency-name: org.springframework.boot:spring-boot-properties-migrator
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-jersey
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-jetty
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-oauth2-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-undertow
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-web
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-maven-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Refactor data processor and sink API (apache#1632)

* Cleanup pom file (apache#1632)

* Migrate processors to API (apache#1632)

* [hotfix] Add runtime provider, update pom

* Fix bug in standalone wrapper (apache#1632)

* Fix instance provision in compatibility layer (apache#1632)

* [hotfix] Disable Maven cache in workflow file

* [hotfix] Fix failing build by reverting Flink version

* [hotfix] Add proper time selection to another cypress test (apache#1641)

* Fix bug in aggregation flink module

* Properly return output source and schema info

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Tim <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: remove Plc4xPassiveAdapter

* Add implementation for schema guessing from the example file in the HTTP

resolves apache#1637
fixes apache#1518 for streampipes-extensions

* [apache#1592] Add io link master id to event

* [apache#1592] Provide documentation for io link adapter

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Dominik Riemer <[email protected]>
Co-authored-by: Stefan Obermeier <[email protected]>
Co-authored-by: bossenti <[email protected]>
Co-authored-by: Tim <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@bossenti
Copy link
Contributor

Completed by #1647

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
connect Related to the `connect` module (adapters) enhancement New feature or request
Projects
None yet
Milestone
0.93.0
Development

No branches or pull requests
2 participants
@tenthe @bossenti