Support for OPC-UA input and output plug-ins

[edbernier]

Feature Request

The OPC-UA protocol is heavily used in the energy and utilities industries and was originally developed for SCADA systems. As companies try to update their metrics collection platforms to next generation time series databases like InfluxDB they'd like to tap into this established infrastructure based on OPC-UA.

There is an OPC-UA logger that was developed to insert data into InfluxDB but nothing for writing back out to OPC-UA:
https://github.com/coussej/node-opcua-logger

There are a number of open source projects that one might be able to tap into while developing such a plug in which are listed here:
https://github.com/open62541/open62541/wiki/List-of-Open-Source-OPC-UA-Implementations

Companies interested in these plug ins:

• Schlumberger
• NOV
• Emerson

Proposal:

Current behavior:

Desired behavior:

Use case:

[dprice000]

OPC-UA is the new standard, correct? There is also the older standard which was just OPC.

A very common issue is that each OPC server is implemented a little different per vendor.

[woodsb02]

Yes, OPC-UA is the "new" standard (formally IEC 62541 published between 2010 and 2012), and replaces the classical OPC which utilised Windows DCOM. OPC-UA allows non-Windows devices to communicate, which was not the case with the classical OPC.

Good descriptions here:
https://en.wikipedia.org/wiki/OPC_Unified_Architecture
https://opcfoundation.org/about/opc-technologies/opc-ua/

Free OPC test/simulations tools are available from Matrikon, including clients and servers:
http://www.matrikonopc.com/products/opc-desktop-tools/index.aspx

[dprice000]

I would like to start an initiative for this. I will be starting with the node library as a base and attempting to translate it into Go. If anyone would like to help I will be storing my code here,

https://github.com/dprice809/opcua_listener

[woodsb02]

Thanks for your efforts.

Can I ask what you mean by the term “listener”? This sounds like telegraf would perform the role of server, and wait for a client to push data to telegraf.

Generally industrial control systems would provide an OPC Server, so extracting data from them would be done by an OPC client sending queries for data for certain “tags”. A data historian collector like telegraf would normally perform the client role of regularly sending these queries to the server.

[thecoldwine]

To be honest, E&P companies are still using OPC DA/HDA protocol primarily, not the UA version, therefore it would be better to develop "legacy" windows only clients for telegraph to feed the database with data. I could start to develop this client but not sure if it would be useful for Linux oriented customers since it cannot be implemented without old DCOM+ bindings.

[RavilN]

While majority might be still using classic OPC, the new OPC UA standard is gaining traction more and more. Taking into account that development of the plugin will take some time, over that period the ratio will change more towards OPC UA. For that reason perhaps plugin should be for OPC UA. Classic OPC data sources can be accessed using existing in the market protocol converters, like OPC UA Proxy from Matrikon.

[RavilN]

Another alternative perhaps could be to build generic OPC UA adapter for SQL database, particularly PostgreSQL, for which there is already plugin.
There are some optimized to store time series data versions of databases do exist, like TimescaleDb, which is accessible as regular PostgreSQL database.

[woodsb02]

Another aspect is that OPC classic is based on Microsoft’s DCOM, so I believe telegraf could only realistically support it on Windows.

OPC UA on the other hand does not rely on DCOM and works on any operating system.

Yet another reason to focus on OPC UA.

[woodsb02]

I believe that supporting OPC UA natively (not via a database middleman) should be a major priority for telegraf. This would effectively extend telegraf and influxdb from a purely IT monitoring historian, to a viable control system historian (OT).

Control system historians is a BIG industry, and the competitors are largely older technology. There could be big demand for this.

[BrentHumphreys]

I would agree with the OPC-UA objective. There are lots of cases where having a low friction way to get industrial/manufacturing data into a time series database, would be valuable. One of the challenges would be coming up with a good way to map the data to the influx schema.

[RavilN]

Yes, it can be kind of enterprise historian.
I say "kind of", because cannot see how OPC UA model information can be utilized in full on the Influx architecture.
For example, OPC UA variables have not only value and timestamp, they have engineering units, range, description, display name (which BTW can be localized) - those characteristics are required on the visualization side. There is also deadband support to eliminate noise before collecting data into historian.
In the Influx architecture I see that the Chronograf module gets all data from InfluxDB, so if the DB does not have all that OPC UA model metadata mapped into it, then all that valuable info is lost and the system as a whole requires duplication of configuration: for example, instead of using engineering units available via OPC UA server, the user will have to type/select it on the Chronograf dashboard configuration.
One more aspect is that OPC UA Servers can generate alarms and events. they need to be historized too, and displayed on the dashboard too, and operator needs a way to react to them. I don't see how OPC UA alarms and events can be utilized in InfluxDB architecture.

Concluding, the questions are: can OPC UA model and alarms/event be utilized in Influx ecosystem?

If Chronograf could access them over generic API via some OPC UA plugin which connects directly to OPC UA servers to get model and events, and historian data via InfluxDB, perhaps that would be doable.
Or map them into InfluxDB, not sure if that is doable, I just started looking at InfluxDB and don't know its internals.

[BrentHumphreys]

The approach I have used is to use the tag set and field set to define these additional attributes. Keep in mind that most of the process historians (Pi, WW Historian) that I have dealt with don't consume all of this data from OPC-UA Either. That is why I made my comment about developing a good strategy in the telegraf connection on how the OPC-UA data would be mapped into influx's tagset and fieldset schema.

[RavilN]

I guess those historians are legacy and do not use all information provided by OPC UA Servers in full.
Ideally, it would be nice for example to have feature to browse OPC UA Server address space and select OPC UA variable from there right on dashboard configuration GUI, and that selection would populate settings in the dashboard like display range, or units, automatically.
I wonder, can address space somehow mapped into InfluxDB together with all OPC UA node attributes/properties?

[dprice000]

We should keep this effort limited to identifying if data can be properly storred in InfluxDB. While many other products provide things like displays and alarming those pieces are typically provided as additional products and not features of the historian itself (ex. PI Notifications and Processbook.)

As for the additional fields and data. Could we get away with serializing this data in something like a JSON format? And I’m sure there is a clever solution for deadband and compression

[BrentHumphreys]

I wasn't aware that there was a UI for telegraf.
serializing the contextual data could be done, but it would not be optimal if you needed to use that data for retrieval.

[dprice000]

I was thinking a lot of the metadata like engineering units, description, devMin, devMax are defined at the “tag” level. So it does not need to be attached to each time series event. This would also reduce indexing overheard.

[woodsb02]

Yes, but it is also possible for the tag configuration to change over time.
E.g. a tag recording a pressure could use units "psi(g)" for the first 1 month, and then its configuration could later change to use units "kPa(g)". There needs to be a method in influxdb to record which units stored values use.

[woodsb02]

The alarms and events portion of OPC UA is a separate component. This functionality does not need to be implemented in the first instance - it could be added later (or not at all).
I agree that it is a useful component, but it is not mandatory.

[edbernier]

InfluxDB has the ability to store both regular and irregular (events) time-series data. So for alarms this information could be written to a separate measurement. An upstream process (i.e. Kapacitor) could scan this data and alert on the presence of an alarm. For the fluctuating units case a tag (pressure_units) could define what units the field value represents.

[RavilN]

In the TICK stack architecture (https://www.influxdata.com/time-series-platform/) Telegraf and Chronograf both interact with InfluxDB (the first collects data from various sources and writes it to the DB, and second reads data and visualizes it). For me it is hard to imagine how OPC UA model information can be propagated and used by visualization component. I don't see that Chronograf allows using of some custom data source, which can provide both data and model.

If you look at Grafana, it provides API to add custom data sources. I guess you could add data source for OPC UA, which can provide all you need for visualization - real time data, historical data, alarms and event, and most importantly, information model. Note that the model could be used to create and configure dashboards, users do not need to enter settings like localized display name, description, display ranges, units, etc.
That data source can get data either from ordinary OPC UA Servers directly, or it might be some historian with OPC UA Server interface, using in the backend some TSDB like InfluxDB or anything else.

I could lead such a project if there are interested parties.

[gisfinder]

For OPC protocol, we are now using. The real-time data are sent to SCADA and to historian system. We use logger from OPC to sqlserver. Using open architecture solution it makes sense to integrate all required data to the monitor system.
So in traditional control system , data adapter from OPC to influxDB or prometheus system is useful.

[russorat]

@woodsb02 @RavilN @gisfinder @dprice809 @BrentHumphreys thank you all for the great discussion here. I'm the product manager for both Telegraf and Chronograf and I agree with many of the points here.

To summarize what I'm hearing:

• A Telegraf plugin in the new OPC UA format would be valuable
• Telegraf can be run on Windows machines so legacy OPC protocol support might be possible, but it is probably better to focus on UA now, with legacy support added later
• The data model of InfluxDB, with tags and fields, can support storing the field value and all the metadata such as unit, range, etc
• Chronograf is not a viable alternative to existing solutions unless it could leverage this metadata in it's UI in some way to facilitate creating dashboards or exploring data
• The https://github.com/coussej/node-opcua-logger project has been running for a number of years successfully writing OPC-UA data into InfluxDB.

We have created custom experiences in Chronograf based on specific plugins in Telegraf. See the syslog plugin as an example (the log viewer in Chronograf 1.6).

I have limited knowledge of OPC UA, but on the surface it sounds like the https://github.com/coussej/node-opcua-logger might be a good model to mimic.

Once the data is being written to InfluxDB, we can work on building custom experiences in Chronograf for something like this.

[Metalfusion]

Another use case for OPC UA is to provide a standardized API for historical (time series) data access. The standard defines historical data queries including aggregation. The DB would need to implement OPC UA server that exposes stored data using the OPC UA data model and supports historical data queries. OPC UA even has "methods" for RPC that can be used to expose custom API within the same secure communication model.
Primary way to get live data from OPC UA servers is to use a subscription that notifies on value changes, so polling is not needed.

[RavilN]

I have built prototype application which can log data from OPC UA servers to PostgreSQL. Introduction video with context diagram is available here: [https://youtu.be/Gb56k0j-0Tc]
As a next step target database can be made configurable - for example InfluxDB can be added as time-series database.
The same application can be used also to browse OPC UA Servers' address space, from the front-end side (which is single page web application). Front-end communicates with the back-end over REST API.
In the future it is possible to extend REST API to provide also historical data and as well as real time data, so data source plugin written in JavaScript and running on browser can access time-series data via REST API from the back-end, which in turn gets some data from TSDB (history) and OPC UA Servers (real time).
If I had free time I could work on it further, but right now busy on other, paid projects.

[fernwerker]

I was just recommended this opcua implementation in Go. Might be easier or helpful for an input. https://github.com/wmnsk/gopcua

[russorat]

has anyone in this thread taken a look at #4623 ? What do you think?

[woodsb02]

It is a great idea to support both Modbus TCP and OPC UA - this will ensure Telegraf supports data collection from a large percentage of the devices in the industrial control system / operational technology landscape.

[glinton]

Is there an emulator or something that we could test against. It seems a major blocker on this is the need for specialized hardware. Would something like https://www.matrikonopc.com/downloads/1361/index.aspx (Matrikon FLEX OPC UA SDK) be an effective simulation server to get started here?

[RavilN]

Is it possible to feed data from PostgreSQL?
If so, probably you could use data logger from my company to log data from OPC UA Servers to PostgreSQL first, and then read from PostgreSQL to telegraf (currently prototype is available at https://github.com/onewayautomation/OPC-UA-Data-Logger)

[Tonny72]

@glinton You can use KepserverEX as OPCUA simulation server from Kepware.

[murmlgrmpf]

There is a new extension of opcua published, called opc-ua pubsub. It is implementing a publisher subscriber logic and seems to be a quite universal solution.

[HogeBlekker]

In this issues' opening post:

nothing for writing back out to OPC-UA

We developed https://github.com/factrylabs/influx-opcua-server to do just that.

[tecmatia-dp]

I invite you to try Notify, a software application specifically designed to collect data from SCADA systems (including OPC UA) and in addition to notifying alarms, allows registration in InfluxDB.
http://tecmatia.com/index.php/en/products/notify

[mjkl-gh]

Any news on this? Our company has got plenty of machines out in the field with OPC-UA and we're looking into building a test dashboard. This is what I'm currently using, but if telegraf could handle it that would be much better IMHO

[RavilN]

Hi Maarten,
Would be storing of OPC UA data into InfluxDB sufficient for you to start?
If so, probably I could add support for InfluxDB target into ogamma Visual Logger for OPC (https://cloud.docker.com/u/ogamma/repository/docker/ogamma/logger - this is development version, beta version will be released soon).
Time-frame can be 3-4 weeks.
If you are interested, please let me know, we can discuss mapping details.
Regards, Ravil

[yonas124]

Any update on influxdb to opcua server?
My setup is that I have an opc-ua scada and I'd like to browse my influxdb db/measurement as opc items. The db is populated by mqtt subscriber.

[loonwong]

If someone is interested then we offer a cross-platform OPC UA server with the ability to present InfluxDB fields as OPC UA items and store data in InfluxDB (https://monokot.io)

[RavilN]

Hello,

Visual Logger with support of InfluxDB has been released! Both version 1.7 and 2.0 are supported.
How to deploy community edition is described here: https://onewayautomation.com/visual-logger-docs/html/deploy.html

There is Docker image available, as well as installation packages for Windows, Ubuntu and Debian.

Major difference from other OPC UA to InfluxDB data moving solutions: high throughput and performance, and web based configuration GUI. In test setup throughput rate was around 100,000 values/second.

Best Regards, Ravil

[alexmc1510]

I am very interested on it. Thanks to ogamma and @RavilN we are able to store data in PostgreSQL and influxDB and I would like to give public thanks to RavilN for his effort. From my point of view, the last and best functionality would be to convert OPC UA to Kafka or include OPC UA as telegraf input.

[RavilN]

Hi alexmc1510,
You are welcome, I am glad that it is useful.
Please feel free to create issue at GitHub page if you need new features or encounter bugs, or need different mapping from OPC UA to TSDB.
Also, would be interesting to know details of your project, if you could send me some info to ravil @ onewayautomation.com, would be really appreciated.
Adding support for Kafka is on our radar, can be accelerated if you create issue at GitHub.
Thank you, Ravil

[PWSys]

Is there a reason this was removed from the 1.14.0 milestone? Has it been pushed to a later release?

[danielnelson]

We have pushed this back to a yet to be determined future release.

[PWSys]

Thanks for the update. Can you share if this is a reflection of a reduced priority from the Telegraf team or lack of understanding of the protocol or implementation?

[sjwang90]

@PWSys With Telegraf 1.14 targeted for release next month, we removed the OPC-UA plugin as a priority as we are focusing on a few other IoT input/output plugins that are currently actively being worked on. We understand the importance of OPC-UA and have designated some resources to begin developing an OPC-UA plugin for a future release.

[PWSys]

Thanks for the update @sjwang90. An OPC-UA plug-in is a requirement for my organization and customers to move over to a fully integrated TICK stack for Industrial Control System applications.

[alexmc1510]

Hello, I have the same opinion as @PWSys , OPC UA is a requirement for my organization. By the moment I am using a workaround but it would be good to have it into TICK stack system.

[ThiruKumar]

Hello, I have the same opinion as @PWSys , @alexmc1510 OPC UA is a requirement for my organization too. By the moment I am using a workaround but it would be good to have it into TICK stack system.

[xkilian]

Grafana has an OPC-UA datasource plugin, it is currently being updated to work with Grafana 7. It is not currently published to the Grafana plugin directory, so it is hard to find. See https://github.com/srclosson/grafana-opcua-datasource

[danielnelson]

Please don't advertise other solutions on this issue tracker, it's off topic. Let's keep any discussion focused on Telegraf support for OPC-UA.

[RavilN]

Sorry, I have deleted it now.

[mrhb]

@woodsb02 @RavilN @gisfinder @dprice809 @BrentHumphreys thank you all for the great discussion here. I'm the product manager for both Telegraf and Chronograf and I agree with many of the points here.

To summarize what I'm hearing:

• A Telegraf plugin in the new OPC UA format would be valuable
• Telegraf can be run on Windows machines so legacy OPC protocol support might be possible, but it is probably better to focus on UA now, with legacy support added later
• The data model of InfluxDB, with tags and fields, can support storing the field value and all the metadata such as unit, range, etc
• Chronograf is not a viable alternative to existing solutions unless it could leverage this metadata in it's UI in some way to facilitate creating dashboards or exploring data
• The https://github.com/coussej/node-opcua-logger project has been running for a number of years successfully writing OPC-UA data into InfluxDB.

We have created custom experiences in Chronograf based on specific plugins in Telegraf. See the syslog plugin as an example (the log viewer in Chronograf 1.6).

I have limited knowledge of OPC UA, but on the surface it sounds like the https://github.com/coussej/node-opcua-logger might be a good model to mimic.

Once the data is being written to InfluxDB, we can work on building custom experiences in Chronograf for something like this.

as a control Engineer, I should say OPC-UA is designed to control the industrial process. but we need industrial data in a higher level of management in a management dashboard. who look at dashboard needs to see overall process performance, not details.
nowadays I'm looking for such plugIns to integrate field data into InfluxDB. please continue development... I'm waiting for it

[gavioto]

I think opcua support for telegraf is here: #7213

https://github.com/influxdata/telegraf/tree/master/plugins/inputs/opcua

[sjwang90]

OPC UA input plugin was merged last week and will be included in the 1.16 release.

There's a PR #7195 for the OPC UA output plugin if anyone would like to review.

I'll close this for now. I've opened another issue #8083 around an events base input plugin for OPC UA (new plugin polls OPC UA metrics) if anyone would like to a place for any discussion around there.