net: config: handle autoconfiguration of multiple interfaces

[cfriedt]

Introduction

This is a proposal for a minimal subset of Devicetree bindings to allow for autoconfiguration of multiple network interfaces via DT_INST_FOREACH_STATUS_OKAY().

The primary reason for using DT is to allow for relatively easy scaling in comparison to Kconfig. The network stack was added to Zephyr before Devicetree came to Zephyr, when most devices had (at most!) 1 network interface. Now, we run on devices with multiple Ethernet ports, WiFi, BLE, etc. Imagine trying to use Kconfig to configure a 32-port router or something along those lines.

Problem description

Zephyr is beginning to see some increased need to support multiple network interfaces on a single device. Obvious examples would include dual-radio devices (with 5 GHz and 2.4 GHz transceivers) and dual-mac devices (multiplexing 2.4 GHz IEEE 802.15.4 and BLE).

We also need to support Ethernet, IP over CAN, SLIP, PPP, USB CDC ACM, among others.

The current issue is that our network stack only supports auto-configuration of a single network interface (and additionally, one IPv6 or IPv4 address per interface).

E.g. from samples/net/sockets/echo_server/prj.conf

CONFIG_NET_CONFIG_SETTINGS=y
CONFIG_NET_CONFIG_NEED_IPV6=y
CONFIG_NET_CONFIG_MY_IPV6_ADDR="2001:db8::1"
CONFIG_NET_CONFIG_PEER_IPV6_ADDR="2001:db8::2"
CONFIG_NET_CONFIG_NEED_IPV4=y
CONFIG_NET_CONFIG_MY_IPV4_ADDR="192.0.2.1"
CONFIG_NET_CONFIG_PEER_IPV4_ADDR="192.0.2.2"

Proposed change

• Add Devicetree bindings for minimal subset
• Enumerate special-case interfaces that require DT bindings above the minimal subset
• Annotate Kconfig options for global network address configuration as sooon-to-be-deprecated
• Update applicable drivers / network modules to use DT_FOR_EACH_INST_STATUS_OKAY()
• Update samples, tests, boards, etc
• Deprecate Kconfig options for global network address bindings.

Detailed RFC

Proposed change (Detailed)

How to associate IP addresses with a particular device? Either

• node itself (common use case - e.g. 802.15.4 device init checks for presence of network if bindings)
• phandles (possibly for bridging?)
• nesting (refer to parent device)

For X in addr, netmask, and route, dns:

Either of the following (should debate the relative usefulness and choose 1):

• ipv4-X = <ip1 ip2 ip3 ip4>; (array of uint32's, can be validated at compile-time)
• ipv4-X = "ip1", "ip2", "ip3", "ip4"; (string-list, easier for users, probably requires runtime validation)

Either of the following (should debate the relative usefulness and choose 1):

• ipv6-X = [ip1 ip2 ip3]; (giant byte array, can be validated at compile-time)
• ipv6-X = "ip1", "ip2", "ip3", "ip4"; (string-list, easier for users, probably requires runtime validation)

Address mode selection (an enumeration):

• STATIC
• DHCP
• DHCPv6
• "auto" maybe?

There are some special-case network interface types that might require additional DT bindings - e.g. PPP requires a peer address, IEEE 802.15.4 has short and long addresses, it may be desirable to specify an alternate MAC address for Ethernet, some configurations may require multicast addresses, etc. In the future, we may add TAP adaptors for packet sniffing, or ethernet bridging, or virtual interfaces.

Specification should be handled via inheritance in DT bindings.

An interesting aspect of this storage-wise, is that the number of IP addresses per interface would be exactly the number that are configured in Devicetree - no more, no less, which is nice in terms of code / memory usage.

Dependencies

Devicetree bindings are additive. However, the network subsystem might have some instances that will need some refactoring in the case that Kconfig values are used directly.

List specific areas of the network subsystem here that will require attention:
*

Concerns and Unresolved Questions

Currently, there are many samples and tests that depend on Kconfig as the primary means of network interface configuration. Those samples and tests would need to be updated.

Additionally, many board definitions and products rely on Kconfig to configure network interfaces.

Alternatives

Intentionally blank

Is your enhancement proposal related to a problem? Please describe.

Yes!

This was conceptualized when working on #29598.

When more than 1 IEEE 802.15.4 address is present, the echo_client / echo_server applications misbehave.

*** Booting Zephyr OS build zephyr-v2.4.0-1206-g98490c598ed7  ***
[00:00:00.001,983] <wrn> net_if: You have 1 IPv6 net_if addresses but 2 network interfaces
[00:00:00.002,014] <wrn> net_if: Consider increasing CONFIG_NET_IF_MAX_IPV6_COUNT value.
[00:00:00.010,650] <err> net_config: Cannot setup IEEE 802.15.4 interface (-22)
[00:00:00.010,650] <inf> net_config: Initializing network
[00:00:00.010,650] <inf> net_config: Waiting interface 1 (0x20001438) to be up...

I tried the following which should allow more than 1 network interface

CONFIG_NET_IF_MAX_IPV6_COUNT=2

And appended -DCONFIG_NET_CONFIG_IEEE802154_DEV_NAME="\"IEEE802154_1\"" to indicate that network autoconfiguration should use the secondary IEEE 802.15.4 network interface, but then it simply hangs indefinitely until there is a timeout.

*** Booting Zephyr OS build zephyr-v2.4.0-1206-g98490c598ed7  ***
[00:00:00.010,681] <err> net_config: Cannot setup IEEE 802.15.4 interface (-22)
[00:00:00.010,681] <inf> net_config: Initializing network
[00:00:00.010,711] <inf> net_config: Waiting interface 1 (0x20001438) to be up...
[00:01:33.013,763] <err> net_config: Timeout while waiting network interface
[00:01:33.013,793] <err> net_config: Network initialization failed (-62)
[00:01:33.013,885] <inf> net_echo_client_sample: Run echo client
[00:01:33.013,916] <inf> net_echo_client_sample: Network disconnected
[00:01:33.013,977] <inf> net_echo_client_sample: Network disconnected

I think it's attempting to initialize "IEEE802154_0" but I'm not 100% certain.

The particular code in play here is:

subsys/net/lib/config/ieee802154_settings.c: int z_net_config_ieee802154_setup(void);
subsys/net/lib/config/init.c: static void setup_ipv6(struct net_if *iface, uint32_t flags);

Describe the solution you'd like
I think rather than "hard-coding" the network interfaces in Kconfig, we should probably introduce DT bindings for network configuration. That way, any number of interfaces can be instantiated and their IP addresses can be set in .dts rather than in a .conf file.

The DT bindings would need to cover a number of uses cases:

• both IPv4 and IPv6 addresses
• more than one of each type of address (particularly IPv6)
• network masks
• default routes
• statically assigned addresses
• dynamically assigned addresses ('auto' maybe?)
• maybe links to MAC and PHY entities as well
• dt bindings for mac / phy types? e.g. ble, 15.4, can, lora?
• peer address for point-to-point links?

Describe alternatives you've considered
See above

Additional context
This issue was discovered in the process of #29598. We would ideally like to demonstrate IEEE 802.15.4 at 2.4 GHz working simultaneously with Sub GHz IEEE 802.15.4. Technically speaking, I could set -DCONFIG_IEEE802154_CC13XX_CC26XX=n on the command line here when building with overlay-802154-subg.conf but if that were the case, I would most likely also want an overlay-802154-multi.conf where both the 2.4 GHz and SubGHz PHYs are active.

Suggestions welcome. I also don't like manually setting CONFIG_IEEE802154_CC13XX_CC26XX=n - maybe there is a generic way to disable 2.4 GHz IEEE 802.15.4? There seems to be only one option for CONFIG_IEEE802154=y, maybe it would be wise to add CONFIG_IEEE802154_SUB_GHZ=y as well.

[jukkar]

The subsys/net/lib/config was never designed to support multiple network interfaces. That is why for example echo-server Kconfig file has separate config options for the other network interfaces which are in use if VLAN is enabled. So in this case the configuration problem is pushed to the application as it has better knowledge of the desired functionality.
It would be great if we could have a simple way to configure multiple network interfaces automatically.

[cfriedt]

It would be great if we could have a simple way to configure multiple network interfaces automatically.

It's surprising how many times "Yes" is the answer to the question "Can DT fix that?"

[jukkar]

It's surprising how many times "Yes" is the answer to the question "Can DT fix that?"

Yes, DT could probably be used here. As always, patches are welcome.

[cfriedt]

@jukkar @rlubos

Just had a great use case for this 😁 I lent out my wrt54gl (an original!) a long time ago, and I might finally see it returned soon. Would be a fun retro-computing project to Zephyr running on it and work on getting all features supported. Might provide some interesting new features for Zephyr along the way.

[cfriedt]

@gmarull - PTAL when you have a moment, would be great to get your input here.

[mbolivar-nordic]

I generally agree that the right place for this in Zephyr is devicetree. This has always been our escape mechanism when Kconfig doesn't scale. I don't have enough networking knowledge to comment in depth, but I'd really love to see the networking subsystem become more DT aware in general. I do have some comments on the detailed RFC:

ipv4-X = (less-than) ip1 ip2 ip3 ip4 (greater-than); (array of uint32's, can be validated at compile-time)

I think you meant something like ipv4-X = <ip1 ip2 ip3 ip4>;, right? I'm not sure why you typed out "(less than)" and "(greater than)", haha.

If so, note that helper macros are always possible, and you can use a slightly different syntax, to get a bit more readable (but completely equivalent) results. Example:

ipv4-addr = <IPV4_ADDR(192, 168, 0, 1)>,
                   <IPV4_ADDR(...)>,
                   ...;

ipv6-X = [ip1], [ip2], [ip3]; (list of byte-arrays, does this work in DT?,

It will all just be concatenated into a single byte array.

Devicetree bindings are additive.

... by default, but you can do this too to just take exactly what you want from an include:

include:
  - name: foo.yaml
    property-allowlist:
      - i-want-this-one
      - and-this-one
  - name: bar.yaml
    property-blocklist:
      - do-not-include-this-one
      - or-this-one

[cfriedt]

@mbolivar-nordic - thanks for the suggestions (esp the ipv6 version). Had the same idea in mind with the IPv4 addresses.

Was just kind of rushing, talking, taking notes while I noticed <> did not render.

I like the idea of using binary versions of IP addresses but am also worried that they might be annoying for users.

It's not a problem for IPv4, but kind of annoying for IPv6 (unless we had some similar macro).

strings would generally make things a bit bloated, otoh.

@rlubos, do you know of a convenient macro for specifying IPv6 addresses in binary? I can imagine one way to do it, but it would involve some level of macrobatocs (h/t @henrikbrixandersen)

[rlubos]

@rlubos, do you know of a convenient macro for specifying IPv6 addresses in binary? I can imagine one way to do it, but it would involve some level of macrobatocs (h/t @henrikbrixandersen)

Nothing comes to my mind, unfortunately. Personally, I'd lean towards string representation of IP addresses, especially for IPv6, a commonly used test address 2001:db8::1 for example, the string representation is pretty neat, but imagine putting all those zeroes if we wanted to represent this in binary...

[cfriedt]

@rlubos, do you know of a convenient macro for specifying IPv6 addresses in binary? I can imagine one way to do it, but it would involve some level of macrobatocs (h/t @henrikbrixandersen)

Nothing comes to my mind, unfortunately. Personally, I'd lean towards string representation of IP addresses, especially for IPv6, a commonly used test address 2001:db8::1 for example, the string representation is pretty neat, but imagine putting all those zeroes if we wanted to represent this in binary...

Exactly - I occasional remind myself that this could probably be done at compile time via C++ and constexpr. In fact, I think that the majority of our ELF hacks could be done this way too, which would simplify the macOS port (cc @galak)

[jadonk]

Is there at least a way to dynamically switch between the multiple interfaces if I cannot get them both enabled at the same time? Having different software builds for this seems unreasonable.

[rlubos]

Is there at least a way to dynamically switch between the multiple interfaces if I cannot get them both enabled at the same time? Having different software builds for this seems unreasonable.

I'm not sure I follow, you can have mutliple interfaces enabled at the same time, the issue is about the net_config lib not being able to configure multiple interfaces at boot. But you still can configure them manually with net_if_* APIs if needed.

The IP stack will choose the right interface based on the destination IP address, or you can also use the SO_BINDTODEVICE socket option to bind a socket to a specific interface.

[fabiobaltieri]

• @cfriedt: there's currently no change associated with this but it's been demanded by multiple people
  • tl;dr: we have systems with multiple interfaces, multi MAC etc, and other use cases as well (CAN, SLIP...), Kconfig does not scale well with multiple devices/pseudo-devices
  • it may be a good idea to have a network interface pseudo-device to use the parent-child hierarchy
• @luca-fancellu: also interested in using DT for device configuration, faced the issue with multiple interfaces and only one can be configured, willing to contribute
• @cfriedt: let's start simple first (IPv4/IPv6)
• @bjarki-trackunit: should we support all config on both Kconfig and devicetree or limit the scolpe
• @cfriedt: we should narrow the scope, DT should be about the hardware configuration
• @keith-zephyr: this feels similar to, for example, setting a baudrate, which is a software configuration but is indeed coupled to the hardware, asking if there's anyone advocating against it?
• @jfischer-no: things like IP settings should not be in the devicetree, it's an application detail so it does not belong to DT, maybe zephyr,user properties could be used instead.
• @galak: zephyr,user was meant for convenience would not build on top of it
• @cfriedt: IPs are a software thing but other network features involving address are closer to the hardware and some of those details cross over into being hardware specific
• @galak: no big issue with this (software details in DT at some extent), where do we draw the line? It would be interesting into seeing another model for this (u-boot...). We need to think broader than just networking. One of the concern is about tooling/ease of use. It makes sense to use DT beyond hardware in these cases.
  • In linux we pass a command line, it's not a very robust system, maybe there should be a root level config node
• @carlocaione: like the idea of having a Zephyr specific config root node in the tree, maybe we should separate the devicetree for hardware and configuration, maybe we could think about splitting the two at a file level.
• @nashif: agree with Kumar we should define the line between hw/sw, everything we do in Zephyr is within hardware but we need to define the boundary. At some point we'll be programming in devietree.
  • We need a plan for configuration, but that does not have to necessarily be devicetree, as it is not what it was meant for
• @cfriedt: cannot drive the effort, need help! :-)
• @gmarull: agree with Anas, we should not abuse DT, we may end up diverging too much with Linux, causing confusion
• @bjarki-trackunit: if we use a different system we should at least use something like yaml, that may become our metaprogramming language, but it may also be simpler to just do the configuration in the application
• @nashif: there's room for a system just for that
• @jfischer-no: retention subsystem could be an option

Notes from the chat:

• https://github.com/devicetree-org/dt-schema/blob/main/dtschema/schemas/options.yaml

[mkschreder]

What about extending settings subsystem with features necessary for storing hierarchical network config data and then providing a way to compile in default settings based on yaml definitions?

[jukkar]

Hi all,
It seems that we cannot use device tree for storing this configuration, and yaml has been mentioned in various places as a way to express the configuration data. So I was thinking this network subsystem configuration problem and came to this that the user could create yaml file that would describe network configuration data, then that data could be preprocessed to a settings db (and saved to a partition) which could then be loaded at runtime by a network config module which could then apply the configuration. The said configuration could be saved and then re-loaded later in the next boot.

Below is a simple example yaml file that illustrates how the config data could be used.

# Example of one interface selected by its name
- net_if: &main-interface
    name: eth0
    set-name: my-eth0
    link-address: 01:02:03:04:05:06
    MTU: 1500
    set-default: true
    IPv6:
      status: enabled
      addresses:
        - address: 2001:db8:110/64
      multicast-addresses:
        - address: ff05::114/16
        - address: ff15::115/16
      prefixes:
        - address: 2001:db8::/64
          lifetime: 1024
    IPv4:
      status: enabled
      addresses:
        - address: 192.0.2.10/24
      multicast-addresses:
        - address: 234.0.0.10/8
      gateway: 192.0.2.1
    DHCPv4: enabled
    DHCPv6: disabled
    IPv4-autoconf: disabled

# Example of another interface selected by its device
- net_if:
    device: ETH_DEVICE
    set-name: my-eth1
    flags: no-auto-start
    IPv4:
      status: enabled
      addresses:
        - address: 192.168.1.2/24
      multicast-addresses:
        - address: 234.0.0.10/8
      gateway: 192.168.110.1
    DHCPv4: enabled

# Example of virtual interface tied to the first one
- net_if:
    name: virt0
    set-name: virt0
    bind-to: *main-interface
    IPv6:
      status: enabled
      addresses:
        - address: 2001:db8:110/64
    IPv4:
      status: disabled

# Example of VLAN interface that attaches to eth0
- net_if: &vlan-interface
    set-name: vlan0
    bind-to: *main-interface
    VLAN:
      status: enabled
      tag: 1234
    DHCPv4: enabled

# IEEE 802.15.4 configuration data
- ieee-802.15.4:
    pan-id: 0xabcd
    channel: 26
    tx-power: 1
    security-key: foobar
    security-key-mode: 0
    security-level: 1
    ack-required: true

- sntp:
    server: sntp.foo.bar
    timeout: 30
    bind-to: *vlan-interface

WDYT?

[cfriedt]

@jukkar - thanks so much for being the first one to take the plunge 😁 At first glance, this looks fine to me, but a lot of things look fine at first glance.

And of course, in terms of the network elements you captured here, I think they make sense.

YAML makes complete sense from a high level - it is the glue that binds a lot of things together, and is supported quite well in many places. At the same time, I wonder if there is an actual standard schema of this sort (i.e. an IEEE spec or standard, or IETF RFC that captures best practices).

Food for thought:

• Debian / Ubuntu
• OpenWrt

The OpenWrt solution is (maybe not coincidentally?) like Zephyr's Settings - a key-value format (with extra "quirks") that is necessarily string based. Also, OpenWrt makes heavy use of Lua (which is actually pretty good for embedded DSLs and also structured data).

Devicetree

There is a YAML to DT converter maintained by friends at Konsulko. Yes, I know it sounds funny, but in all seriousness, a lot of markup languages can usually be translated from one format to another. With that, at least, we would be able to re-use some existing DT mechanisms where we get the compile-time const aspect of config data that (IMO) really serves Zephyr quite well.

So it's a +1 for the secondary SW configuration root node proposed by @carlocaione, but at the same time, it opens up a lot of other possibilities if combined with the YAML approach proposed by @jukkar.

At the same time, we don't yet incorporate any runtime DT (DTB) or YAML parsers, and storage is a bit of a concern.

Settings DB

I have concerns about the Settings DB. It is a key-value store and is necessarily string-based. Sure, it's human-readable, but so is JSON or YAML, or XML and those are standardized.

Concerns:

• code size / speed
• flash / ram requirements
• maintainability - i.e. we do not necessarily want to maintain a bunch of custom code for random parsing quirks and special cases
• Ad-hoc formats always (always) end up being technical debt.

[jukkar]

Devicetree

There is a YAML to DT converter maintained by friends at Konsulko. Yes, I know it sounds funny, but in all seriousness, a lot of markup languages can usually be translated from one format to another. With that, at least, we would be able to re-use some existing DT mechanisms where we get the compile-time const aspect of config data that (IMO) really serves Zephyr quite well.

That is an interesting idea to use DT as an intermediate layer that passes the configuration data to zephyr instance. I just wonder the tooling to get this work during the build.

So it's a +1 for the secondary SW configuration root node proposed by @carlocaione, but at the same time, it opens up a lot of other possibilities if combined with the YAML approach proposed by @jukkar.

Indeed 👍

At the same time, we don't yet incorporate any runtime DT (DTB) or YAML parsers, and storage is a bit of a concern.

I think we do not want to incorporate any new parsers to zephyr image. IMHO all the processing of yaml should be done in host side.

Settings DB

I have concerns about the Settings DB. It is a key-value store and is necessarily string-based. Sure, it's human-readable, but so is JSON or YAML, or XML and those are standardized.

I was mainly thinking Settings DB here as that could be used as a provision db inside zephyr. So the initial settings db could be provided by a yaml file, then that could be mounted in zephyr as read-write and any modifications to it could be remembered and restored in next boot. But that is kind of system level thing and I am not sure if we want to go there here and perhaps that is best left for the vendors to tackle.
Bluetooth subsystem already has provision data using settings db, it is just not created / populated beforehand but during when device runs.

[jukkar]

I did some experimenting with yaml2dts which can be found at https://github.com/pantoniou/yamldt with these results:

• The tool seems to be abandoned, last update is from 2017.
• The tool compiled ok (with warnings), and it had some runtime issues and crashes. This required couple of trivial changes to pointers / array usage in the code.
• The tool was not able to accept yaml list syntax for maps, this is probably related to dts syntax conversion

Example:
This is valid yaml code:

net:
  network-interfaces:
    # Example of one interface selected by its name
    - net-if: &main-interface
      name: eth0
      set-name: my-eth0
      set-default: true
      DHCPv4: enabled
      DHCPv6: disabled
      IPv4-autoconf: disabled
    # Example of second interface
    - net-if:
      name: eth1
      set-name: my-eth1
      DHCPv4: enabled
      IPv4-autoconf: disabled

Which when "pretty-printed" produces this output

$ ./pretty-print.sh < eth0-valid.yaml
{'net': {'network-interfaces': [{'DHCPv4': 'enabled',
                                 'DHCPv6': 'disabled',
                                 'IPv4-autoconf': 'disabled',
                                 'name': 'eth0',
                                 'net-if': None,
                                 'set-default': True,
                                 'set-name': 'my-eth0'},
                                {'DHCPv4': 'enabled',
                                 'IPv4-autoconf': 'disabled',
                                 'name': 'eth1',
                                 'net-if': None,
                                 'set-name': 'my-eth1'}]}}

I had to change this to:

net:
  network-interfaces:
    # Example of one interface selected by its name
    net-if@a: &main-interface
      name: eth0
      set-name: my-eth0
      set-default: true
      DHCPv4: enabled
      DHCPv6: disabled
      IPv4-autoconf: disabled
    # Example of second interface
    net-if@b:
      name: eth1
      set-name: my-eth1
      DHCPv4: enabled
      IPv4-autoconf: disabled

which "pretty-printed" looks like this

$ ./pretty-print.sh < eth0.yaml
{'net': {'network-interfaces': {'net-if@a': {'DHCPv4': 'enabled',
                                             'DHCPv6': 'disabled',
                                             'IPv4-autoconf': 'disabled',
                                             'name': 'eth0',
                                             'set-default': True,
                                             'set-name': 'my-eth0'},
                                'net-if@b': {'DHCPv4': 'enabled',
                                             'IPv4-autoconf': 'disabled',
                                             'name': 'eth1',
                                             'set-name': 'my-eth1'}}}}

which was then accepted by the tool yamldt eth0.yaml -o eth0.dts and it generated this output

/dts-v1/;
/ {
    net {
        network-interfaces {
            main-interface: net-if@a {
                name = "eth0";
                set-name = "my-eth0";
                set-default;
                DHCPv4 = "enabled";
                DHCPv6 = "disabled";
                IPv4-autoconf = "disabled";
            };
            net-if@b {
                name = "eth1";
                set-name = "my-eth1";
                DHCPv4 = "enabled";
                IPv4-autoconf = "disabled";
            };
        };
    };
};

[cfriedt]

I did some experimenting with yaml2dts which can be found at https://github.com/pantoniou/yamldt with these results:

• The tool seems to be abandoned, last update is from 2017.

Hard to say. I would guess that it is in use in Yocto-related things. The author would likely accept PRs and has also actively contributed todtc for a long time.

https://github.com/pantoniou/dtc

The yaml2dts output looks a little unexpected. Would be good to ensure that we get the preferred / correct syntax.

cc @pantoniou

[carlescufi]

What about extending settings subsystem with features necessary for storing hierarchical network config data and then providing a way to compile in default settings based on yaml definitions?

The settings subsystem is entirely optional, and adding a dependency to it is probably not a good idea. Also, this would then need pre-baking the settings which would require yet more tooling.