[SWP] coap plugtests

[sgso]

This contains the coap-plugtests application. It passes the sections "Base CoAP", "Block", and "Link format" as specified here: https://github.com/cabo/td-coap4/

It depends on a patched libcoap that is referenced in the pkg/libcoap/Makefile and can be found here: https://github.com/sgso/libcoap/tree/riot-pkg. The patches are based on the current HEAD of libcoap. I worked with upstream to include most of the patches that are currently included in the libcoap pkg directory (most of them were due to RIOTs pedantic compiler flags).

There are no handlers programmed for the "Observe" part of the plugtest but my libcoap branch proposes a variant with timer enabled socket functions to compensate for the missing select/*poll.

[OlegHahm]

Coding conventions.

[OlegHahm]

I don't quite get why the linker is unable to find _gettimeofday().

[sgso]

I'll never find out why git hates me....

[OlegHahm]

@sgso, will you try to fix it or should I adopt this PR?

[sgso]

@OlegHahm, I think I addressed the stylistic problems. For me it compiles on native. This: "I don't quite get why the linker is unable to find _gettimeofday()." I never encountered.
If there is any interest left in it, I'd be happy to bring this whole thing in a better shape but not on native only (I'm fed up with that). If you give me board that can theoretically run it (or a recommendation for one to buy) I'd be far more motivated.

[OlegHahm]

@authmillenon, you offered to help out with reviewing - thanks. :)

[miri64]

Are you really commiting to update it regularly in the future? Otherwise: how about adding a fork to RIOT-OS? What do you think @OlegHahm?

[sgso]

I could create a new patch set on top of the current version (like we do now with the older version) which seems to be in a good state and hasn't changed since August/September. Should libcoap development pick up some momentum in the future I'd recommend to ask its maintainer to include the patches together with a RIOT flag rather than fork it. But currently it's not worth the effort, in my opinion.

[miri64]

I could create a new patch set on top of the current version (like we do now with the older version) which seems to be in a good state and hasn't changed since August/September.

👍 please do so.

[miri64]

Apart from the repo situation I'm fine with this PR.

[OlegHahm]

@sgso, are you planning to do so?

[sgso]

@OlegHahm, Tomorrow evening, I'll do.

[OlegHahm]

Great.

[OlegHahm]

Please squash and we're ready to go!

[sgso]

Added cpp compatible headers & removed cppcheck errors.

Squashed.

[OlegHahm]

Applying: adapt libcoap for RIOT
Applying: Add RIOT Makefile
Applying: Add config.h
Applying: remove contiki example programs
Applying: add makefile
error: Makefile: already exists in index
Patch failed at 0005 add makefile

[sgso]

@authmillenon I don't know anything about all the new network stuff (is there an overview of what has changed somewhere?), but is there some select/epoll-like thing for networking now? Last time I asked, I was told to wait for "the new stuff".

[miri64]

As for the ng_tapnet (#2558) thing ("which is basically nativenet without all the RIOT-proprietary headers (so pure Ethernet frames) but with full ng_netdev support."), I don't get it. Currently we have Ethernet[ Riot[ 802145[ 6lowpan[ UDP ]]]], and with ng_tapnet that changes into what? What are "all the RIOT-proprietary headers"? The 2 (?) bytes Riot[ ... ] part? That was never the problem.

ng_tapnet changes this to pure ethernet: <Ethernet|IPv6|UDP|…>. So no proprietary RIOT header, no 802.15.4, no 6LoWPAN. Just IPv6 over Ethernet. To test 6LoWPAN on native regardless I started porting ZEP (Wireshark's Zigbee Encapsulation Protocol) in #2627 which allows you to send IEEE 802.15.4 frames over UDP. Since 6LoWPAN on native is needed only for testing purposes anyway I think that is a better solution, than hooking into some Linux 6LoWPAN implementation through yet another interface ;-).

But regardless: CoAP is not dependent on 6LoWPAN and neither are the plugtests (at least that's what I remember, because that led to a lot of frustration for us on the last Plugtest in London, because everybody else used plain IPv4 and v6 while we were only speaking 6LoWPAN). So I guess you don't need ZEP.

@authmillenon I don't know anything about all the new network stuff (is there an overview of what has changed somewhere?), but is there some select/epoll-like thing for networking now? Last time I asked, I was told to wait for "the new stuff".

The lack of asynchronous IO (which is required for select()) is more of a scheduler problem for us (at least that's what I gathered from discussions in the past). To make things short: this is not in the scope of the current refactoring. An overview of the current progress can be found in #2278.

[sgso]

@authmillenon Thank you, that cleared it up. I tried the ng_tapdev branch and experimented a little with ng_udp. The tap driver works well for me already.

@OlegHahm I'd like to port this to ng_udp in a separate branch and give it some love in the process so it can be included with complete instructions how to test it locally with ng_tapdev against Californium. Ok?

[OlegHahm]

@sgso, sounds perfect to me. Thanks.

[miri64]

@sgso keep in mind that ng_tapnet is more or less obsoleted by @kaspar030's dev_eth_tap (#2776).

[sgso]

I put together a tree of dev_eth_net/ipv6/udp in https://github.com/sgso/RIOT/tree/ng_ipv6/dev_eth_netdev/ng_udp/fixes and started working on the coap plugtests in https://github.com/sgso/applications/tree/master/ng_coap_plugtests

There is no coap yet involved but could somebody take a look at it and tell me if I initialize and use the network stack correctly?

[miri64]

Normally all the initialization you do in main() should be done by auto-init (except the device and nomac initialization), so you can leave them out. You can also leave out the coap-thread, unless you plan something special with it, except dumping, and just register the pktdump module to NG_NETTYPE_UDP with demux_type 5683, instead of the registration to NG_NETTYPE_UNDEF (if you don't want to, I don't know what you need pktdump for here anyways).

[miri64]

The address auto-initialization will come as soon as I find some time for this :-)

[sgso]

I replicated the functionality of this PR with the new network stack in:
https://github.com/sgso/applications/tree/coap-plugtests/coap-plugtests

It needs a RIOT tree with ng_udp & dev_eth_tap and of course a patched libcoap. Locations and instructions can be found in README.md.

Because I'm not using sockets (of any kind) this time, the asynchronous parts integrate nicely and I'll probably start to implement the Observe part of the plugtests as well.

I'd like comments

1. on the vtimer use in coap_thread.c:coap_run_context(): Is it okay to use 2 timers like I do there or should I calculate the minimum of timeouts for one timer?

2. For now I made a lazy port which doesn't try to optimize much. I'd like to get rid of the mallocs. Is something like this legal/intended:

while (packet->next) {
    packet = ng_pktbuf_remove_snip(packet, packet->next);
}
/* packet->next == NULL */

/* declare our package as payload */
packet = ng_pktbuf_start_write(packet);

if (packet) {
    packet->type = NG_NETTYPE_UNDEF;
}

Then "carry" around packet through the libcoap functions and let the pktbuf realloc it before adding data/sending as needed. For a lot of requests the answer will probably be smaller or equal the already allocated packet (certainly for simple ACKS). This would avoid copying buffers for nothing in many cases.

[miri64]

while (packet->next) {
     packet = ng_pktbuf_remove_snip(packet, packet->next);
}
/* packet->next == NULL */

/* declare our package as payload */
packet = ng_pktbuf_start_write(packet);

if (packet) {
    packet->type = NG_NETTYPE_UNDEF;
}

Sure, this is absolute legitimate. If you are in your own thread, there is nothing from stopping you doing stuff with the pkt snips as you like, as long as you make sure (what you do) you have exclusive rights on it.

[miri64]

@sgso ping?

[sgso]

Sorry, I got a little fascinated by real hardware and this suffered under it. I'll polish it within the next week.

The main question now is how to provide the library. I'm not happy with the current patch-set method because updating is tiresome and we'll look at differences of patches here at github.
And even though the changes to libcoap are not fundamental they will still affect a majority of the files with RIOT-specific changes for the network stack, random numbers, logging, etc.

I'd prefer to keep a patched libcoap in a separate repository and periodically update the commit/tag the pkg is pointing at. Upstream has been inactive since last summer, so we wouldn't miss out on anything by having a separate version. Would a RIOT repository be suitable for that? Maybe find one or two other people interested in CoAP besides me to maintain it there?

[miri64]

I'm still uncertain about the separate repository, but since we are more git-dependent than other operating systems which use similar mechanisms for packages, it may be a better solution for now.

[miri64]

Any other opinions on this?

[jnohlgard]

I think keeping a separate repository can help when reviewing changes. Meta diffs (diffs of diffs) are hard to read and to maintain.
I am interested in libcoap because CoAP is our most commonly used protocol in our Contiki sensor applications. From the looks of the libcoap mailing list and repo commits it does not receive a lot of patches so I guess I could try to help out in reviewing if we decide to produce our own fork.

[OlegHahm]

I agree with @gebart and @authmillenon here.

[jnohlgard]

@sgso Ping!

needs rebase

[sgso]

@gebart I'll commit a cleaned & polished libcoap version suitable for integration or separate repository when my exams are over next week. It's pretty much ready but I need to fix some formalities.

[OlegHahm]

Any news?

[sgso]

Yes. I worked on this with the 'old' libcoap base. This (https://github.com/sgso/libcoap/commits/netapio-riot-clean) contains a netapi port that I cleaned from all the cruft, awful style and platform specific code as a basis for a riot specific version like mentioned above. It's probably outdated but in principle it's there.

After libcoap moved to github, I found out about the 'develop' branch which is active and seems to address a lot of the issues that I loathed about libcoap. Under the assumption that libcoap is developed actively again, above branch does not make sense anymore.

I tried to get the develop branch working with RIOT. But I will not go to GNU/autoconf-University just for integrating this in my free time. I have better things to do than figure out Makefiles and I don't know why an embedded library needs libtool and other crap. I'd suggest to hand over the project to the Eclipse foundation. They would love to provide some more additional layers of build tools. [/sarcasm]

Check https://github.com/sgso/applications/tree/coap-plugtests/coap-plugtests for the latest working plugtests application (for some RIOT/libcoap combination, don't know which). It contains some glue code in coap_thread.c which might be interesting.

Provide me with the Makefile to build the develop-branch within RIOT and I can supply some of my modifications. Apart from that, I'm out.

(closing this)

[dimgogos]

Hello,

I am new to RIOT OS and do not have much experience with embedded programming. I am trying to flash the client.c code provided by libcoap examples into my board. However I am facing compilation problems. Is including the libcoap pkg into my makefile not enough?

I am sorry if my question is not relevant to this topic.

[biboc]

Libcoap examples have been tested with RIOT yet. For the moment, you can compile libcoap package with an example as it is shown here:
https://github.com/RIOT-OS/RIOT/tree/master/tests/coap
Feel free to complete this example with a working example of libcoap