GNU radio 3.8 update meta issue #82263

doronbehar · 2020-03-10T17:08:32Z

Gnuradio 3.8 was released quiet some time ago. There are many changes needed to add it to NixOS at the best possible manner. This issue will be the place to discuss all the details with the maintainers of every Gnuradio related package. Some efforts have been made already in #47707 (cc @tomberek) but I hope work done from this issue will supersede it.

I have already thought about most of the details but I reached the conclusion these are too many changes to put in 1 PR. Hence this issue. Here are the main topics I'd like to address:

Incompatibilities between 3.7 and 3.8

There are 2 types of packages that depend on Gnuradio. Some of which are GUI, end user like programs such as gqrx and some of which are Gnuradio extra components such as gr-gsm. Following are the results of my findings in regard to incompatibilities:

End user packages

click to expand

qradiolink

Maintainers: @markuskowa

Info

Doesn't support Gnuradio 3.8 though there's a patch open for that which seem to be working: qradiolink/qradiolink#65

Version status

Needs update 0.5.0 -> 0.8.2-3 , link: https://github.com/qradiolink/qradiolink/releases

gnss-sdr

Maintainers: (None)

Version status

Updated.

Info

Supports both 3.8 and 3.7. Ref: https://github.com/gnss-sdr/gnss-sdr/search?q=3.8&type=Commits

gqrx

Maintainers: @bjornfor @the-kenny @fpletz

Version status

Updated.

Info

Supports both 3.8 and 3.7. Ref: gqrx-sdr/gqrx#705

inspectrum

Maintainers: @mog

Info

Doesn't seem to support both or only 3.8 as there are no open issue nor commits referencing 3.8 api changes. Refs:

Version status

Needs update: unstable-2017-05-31 -> 0.2.2, link: https://github.com/miek/inspectrum/releases

Components / Libraries type packages

click to expand

gr-nacl

Maintainers: @mog

Version status

Hasn't been updated since 2017-04-10. I'm most certain it won't be compatible with 3.8.

gr-limesdr

Maintainers: @markuskowa

Version status

Needs update: 2.0.0 -> 3.0.1, link: https://github.com/myriadrf/gr-limesdr/releases

Info

Current version supports only 3.7, next version supports only 3.8.

gr-osmosdr

Maintainers: @the-kenny @bjornfor

Version status

Needs update: 0.1.5 -> 0.2.0, link: https://github.com/osmocom/gr-osmosdr/releases

Info

Current version supports only 3.7, next version supports only 3.8.

gr-rds

Maintainers: @mog

Version status

Needs update: 1.1.0 -> 3.8.0, link: https://github.com/bastibl/gr-rds/releases

Info

Current version supports only 3.7, next version supports only 3.8.

gr-ais

Maintainers: @mog

Version status

Hasn't been updated since 2015-12-20. I'm most certain it won't be compatible with 3.8.

gr-gsm

Maintainers: @mog

Version status

Needs update: 2016-08-25 -> 0.42.2, link: https://github.com/ptrkrysik/gr-gsm/releases

Info

When testing the update with Gnuradio 3.7, it seemed to pass the configurePhase but the build failed with:

click to expand

Warning: Can't parse color code "#000" for domain "gr_message"
Warning: Can't parse color code "#000" for domain "gr_message"
Warning: Can't parse color code "#000" for domain "gr_stream"
Warning: Can't parse color code "#000" for domain "gr_stream"
Block key "rtlsdr_source" not found
Block key "rtlsdr_source" not found
Validation failed:

Block - blocks_head_0 - Head(blocks_head):
        Sink - in(0):
                Port is not connected.

Block - gsm_receiver_0 - GSM Receiver(gsm_receiver):
        Param - Cell allocation(cell_allocation):
                Value "[arfcn.downlink2arfcn(fc)]" cannot be evaluated:
                name 'arfcn' is not defined

Block - import_1 - Import(import):
        Param - Import(import):
                Import "from grgsm import arfcn" failed.
Error during file compilation.
make[2]: *** [apps/CMakeFiles/pygen_apps.dir/build.make:90: apps/grgsm_livemon_headless] Error 1
make[2]: *** Waiting for unfinished jobs....
Validation failed:

Block - blocks_rotator_cc_0 - Rotator(blocks_rotator_cc):
        Sink - in(0):
                Port is not connected.

Block - gsm_receiver_0 - GSM Receiver(gsm_receiver):
        Param - Cell allocation(cell_allocation):
                Value "[arfcn.downlink2arfcn(fc)]" cannot be evaluated:
                name 'arfcn' is not defined

Block - import_1 - Import(import):
        Param - Import(import):
                Import "from grgsm import arfcn" failed.
Error during file compilation.
make[2]: *** [apps/CMakeFiles/pygen_apps.dir/build.make:85: apps/grgsm_livemon] Error 1
make[1]: *** [CMakeFiles/Makefile2:1261: apps/CMakeFiles/pygen_apps.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
builder for '/nix/store/1cl0252wg3vrdn113w0h8ndqkk43qkzi-gr-gsm-0.42.2.drv' failed with exit code 2
error: build of '/nix/store/1cl0252wg3vrdn113w0h8ndqkk43qkzi-gr-gsm-0.42.2.drv' failed

Haven't tested with 3.8.

Gnuradio 3.7 branch

Gnuradio is still maintaining a 3.7 branch and it had a release, see: https://www.gnuradio.org/news/2020-02-16-gnu-radio-v3-7-14-0-release/ . I guess we should update that as well.

Other Improvements

Introduce `enable` / `disable` flags

Gnuradio has a lot of features that can be enabled and disabled via Cmake flags. I'd like to make them easily turned on / off declaratively - via enableThis and enableThat arguments to the derivation. I'm completely open to other ideas as for handling this idea. But I think it can bring great improvement to closure size of packages that use Gnuradio as a library (such as gqrx, qradiolink ..). The idea is that this will enable such packages to use build of Gnuradio that doesn't include the GUI or the companion, which bloat the closure.

Note

uhd Which is a dependency of Gnuradio, has many optional features as well. One of them is liberio support, which needs: #82259 . After #82259 is merged, I'd like to introduce a PR with many new enable / disable flags for uhd. EDIT: #84183 .

Use a Wrapper Derivation

One of the issues @tomberek encountered when writing #47707, was wrapping the executables. This is certainly a complicated issue as some executables need wrapGAppsHook and some need qt style wrapping. I can imagine how frustrating that might be since the build can take as long as half an hour and if the wrapping fails, all of the build is lost.

I'd like to address this issue by introducing gnuradio-unwrapped and gnuradio (wrapped) derivations. Programs that use Gnuradio as a library shouldn't care about the executables being wrapped or not. Hence they'll be able to use gnuradio-unwrapped and spare excessive builds.

Rewrite Plugin System

Along with using wrapper derivations, I'd like to introduce a new plugin system inspired by urxvt's (see #77347 ). This way, a user would be able to choose to setup a gnuradio development environment with a selection of gr- plugins (declaratively), and the wrapping will be done accordingly. You can read weechat's plugin system for inspiration: https://nixos.org/nixpkgs/manual/#sec-weechat

Technical details

Currently, these are the files in pkgs/applications/radio/Gnuradio/:

pkgs/applications/radio/gnuradio
├── ais.nix
├── default.nix
├── gsm.nix
├── limesdr.nix
├── nacl.nix
├── osmosdr.nix
├── rds.nix
└── wrapper.nix

I'd like to arrange them like this:

pkgs/applications/radio/gnuradio
├── default.nix
├── plugins
│   ├── ais.nix
│   ├── default.nix
│   ├── gsm.nix
│   ├── limesdr.nix
│   ├── nacl.nix
│   ├── osmosdr.nix
│   └── rds.nix
├── unwrapped.nix
└── wrapper.nix

While all-packages.nix will have something like this:

  gnuradio-unwrapped = callPackage ../applications/radio/gnuradio/unwrapped.nix {  };
  gnuradio-plugins = import ../applications/radio/gnuradio/plugins { inherit callPackage; };
  gnuradio = callPackage ../applications/radio/gnuradio/wrapper.nix { };

This is roughly the idea. While I guess 3.7 will still be there.

Unanswered Questions

Should we drop gr-* packages whose upstraem don't show activity regarding Gnuradio 3.8?
Related to the above: We'll probably keep Gnuradio 3.7 in the near future. If we'll keep those very old gr-* packages that only work with Gnuradio3.7, how should we handle the plugins / wrapping system of Gnuradio/wrapper.nix?
Should we rewrite a plugin system like I proposed for 3.8 but for 3.7? With the old available plugins of today?
What are the incompatibilities for Nix users we will introduce?
How to name the new derivations? Should gnuradio point to gnuradio3_7? or to gnuradio3_8?

The text was updated successfully, but these errors were encountered:

nixos-discourse · 2020-03-10T17:13:48Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/gnu-radio-3-8-meta-issue/6189/1

markuskowa · 2020-03-11T12:14:25Z

Thanks for putting this Issue together. We probably will need to keep 3.7 around for quite a while and also keep around the old gr-* plugins as there is no replacement for some of the unmaintained ones.

doronbehar · 2020-03-11T16:13:37Z

We probably will need to keep 3.7 around for quite a while and also keep around the old gr-* plugins...

OK. Some such plugins have a new version available which doesn't support 3.7. Should we keep 2 versions of these plugins? The older suitable for Gnuradio 3.7 and the newer suitable for Gnuradio 3.8 ?

markuskowa · 2020-03-11T20:42:01Z

We may want to keep source plugins, such as limesdr and osmosdr, for 3.7.

doronbehar · 2020-03-11T20:45:52Z

Source plugins?

markuskowa · 2020-03-11T23:14:41Z

Sorry, to clarify: the mentioned plugins provide "signal source blocks", while others like gr-ais provide signal decoding blocks.

doronbehar · 2020-03-28T10:00:54Z

I see. Please tell me @markuskowa what do you think about the following arrangement of files attributes:

Files in pkgs/applications/radio/gnuradio/ will look like this:

pkgs/applications/radio/gnuradio
├── 3.7
│   ├── default.nix
│   └── unwrapped.nix
├── 3.8
│   ├── default.nix
│   └── unwrapped.nix
├── plugins
│   ├── ais.nix
│   ├── default.nix
│   ├── gsm.nix
│   ├── limesdr.nix
│   ├── nacl.nix
│   ├── osmosdr.nix
│   └── rds.nix
└── wrapper.nix

While in all-packages, we'll have:

  gnuradio_3_7 = recurseIntoAttrs(callPackage ../applications/radio/gnuradio/3.7 { });
  gnuradio_3_8 = recurseIntoAttrs(callPackage ../applications/radio/gnuradio/3.8 { });

And every plugin derivation will be called with a specific gnuradio package which'll get compiled with it. A user would be able to access every gnuradio plugin via e.g gnuradio_3_7.limesdr or gnuradio_3_8.limesdr. I'll outline what behaviour a user would get for gr-* packages per version status:

`gr-foo` has a new version compatible with both gnuradio versions

You'd be able to simply access gnuradio_3_8.foo and gnuradio_3_7.foo while each was compiled with the relevant gnuradio version.

`gr-bar` has a new version compatible only with 3.8 while the previous version works with 3.7 only

You'd be able to access both gnuradio_3_8.bar and gnuradio_3_7.bar while gnuradio_3_7.bar will use the latest version of gr-bar that supported gnuradio 3.7.

`gr-bla` hasn't had an update compatible 3.8

You'd be able to access only gnuradio_3_7.bla and trying to access gnuradio_3_8.bla will spit an error such as:

gr-bla is compatible only with gnuradio 3.7, please use gnuradio_3_7.bla

wirew0rm · 2020-03-29T19:26:01Z

I also started experimenting with gnuradio 3.8 and had somehow missed this issue, great to see that more people are working on this.
I like the idea with the unwrapped package, being able to fix wrapper issues without recompiling sounds good.

Regarding enable/disable flags, If it is only about closure size maybe having separate outputs (or even separate derivations, but not sure if that's feasible with gnuradio's cmake) for core, gnu radio companion and the builtin plugins might be better, because it would only need one build and users/dependent applications would still only get what they need.
Maybe this could also help cleaning up the way the wrappers handle PYTHONPATH/switch to pythonWithPackages.

Another new feature in 3.8 is that the companion also can generate c++ code in addition to python code. I guess to support this, gnuradio would have to provide some stdenv to compile these gnuradio c++ files.

doronbehar · 2020-03-31T15:51:35Z

I also started experimenting with gnuradio 3.8 and had somehow missed this issue, great to see that more people are working on this.

Good to have you with us @wirew0rm :).

I like the idea with the unwrapped package, being able to fix wrapper issues without recompiling sounds good.
[...]
Another new feature in 3.8 is that the companion also can generate c++ code in addition to python code. I guess to support this, gnuradio would have to provide some stdenv to compile these gnuradio c++ files.

Another improvement wrapping can introduce is to allow users to add their own packages to the wrapper so their gnuradio will include stdenv or whatever a user may wish to add to the environment GNUradio sees.

Regarding enable/disable flags, If it is only about closure size maybe having separate outputs (or even separate derivations, but not sure if that's feasible with gnuradio's cmake) for core, gnu radio companion and the builtin plugins might be better, because it would only need one build and users/dependent applications would still only get what they need.

I have had once experienced issues with splitting outputs of packages using cmake which provide both development files needed for other packages. I think that having build flavors will be enough but once PRs will start flooding I'd be glad if anyone will test splitting. The main issue not taken care by split outputs in my view is the closure size increase of dependencies - all of the python libraries needed for the GUI will be downloaded no matter what output is used.

doronbehar · 2020-04-05T07:56:53Z

I'm working on most of the ideas presented here at https://github.com/doronbehar/nixpkgs/tree/update-gnuradio3_7 .

Hoverbear · 2020-05-18T23:26:01Z

@doronbehar That fixed gnuradio for me. Thanks so much. :)

andrew-d · 2020-07-17T05:26:56Z

@doronbehar - I just came across this PR; anything I can do to help get this all merged? Would love to help out!

doronbehar · 2020-07-17T16:05:32Z

@andrew-d thanks for the encouragement. My goal is to rewrite an expression for GNU radio 3.8 that will be even nicer to look at compared to what I did in #84401 . I'm (not actively) working on it here:

https://github.com/doronbehar/nixpkgs/blob/gnuradio-rewrite/pkgs/applications/radio/gnuradio/default.nix

And it's not ready yet. Feel free to open a PR to my fork against the gnuradio-rewrite branch if you'd like to collaborate on this, or open your own. Perhaps we shouldn't strive to write 1 expression that will fit both 3.7 and 3.8, but the only thing that's important to me is to have that features attrset that will smartly manage all inputs that are needed by all features. The inspiration is by mpd's expression: https://github.com/NixOS/nixpkgs/blob/b6eca9a2afa143bcbee3b4ae1bdc13993bc71784/pkgs/servers/mpd/default.nix

doronbehar · 2020-10-06T01:44:30Z

Here's the promised improved version of #84401 which adds 3.8 and 3.7 builds with the turn on and off features via an gnuradio.override { features = {}; };: #99685

I've also made sure all reverse deps build and changes to their inputs are unavoidable. Some other changes required are not included in that PR, to ease the review.

The PR also introduces a new wrapper.nix that acts as an external wrapper, hence this fixes issues such as #87510 - making gnuradio-companion work again.

Upcoming changes should be:

Put all gr- plugins in a separate attrset.
Update all plugins to build with either gnuradio 3.7/3.8 (use an outdated version if it's 3.7 being built with).
Cosnider upgrading qradiolink with 3.8 and osmosdr built with 3.8.
Update gqrx to build with 3.8 and qradiolink if possible (fatal error: gnuradio/vocoder/freedv_api.h: No such file or directory qradiolink/qradiolink#67) .

qradiolink can't be built with 3.8 yet (qradiolink/qradiolink#67) and gqrx can, but gqrx needs osmosdr and it's osmosdr should be built with the same 3.8 gnuradio, so this is not taken care of at the moment.

doronbehar · 2020-12-10T21:04:39Z

All work left to do as I see it was done in #106599 . Reviews are welcome :).

wirew0rm · 2021-02-15T18:58:07Z

As gnuraido 3.9 was released last month, how does this fit in the grand scheme of things?
As it changes the python binding generator from SWIG to PyBind, I suspect it will also have varying support in the out of tree modules in the beginning. So the question is, can we reasonably support 3 different gnuradio versions? Do we want to test the 3.9 build against gnuradio.pkgs (#106599) PR? Or do we want to get that merged first? I did some experiments on top of the branch and it seems to not require that many changes. There are still a few issues, but I wanted to check here before investing too much time into it.

doronbehar · 2021-02-15T20:42:36Z

@wirew0rm thanks for mentioning the 3.9 release. I tried putting some work into it now as well, on top of #106599 at: master...doronbehar:pkg/gnuradio/3.9 . And I encountered these errors after configurePhase:

gnuradio> CMake Error at /nix/store/5d7dgha0nl6hixhszb9x5pfczpblch1a-python3.8-pybind11-2.6.1/share/cmake/pybind11/pybind11Tools.cmake:151 (add_library):
gnuradio>   Target "vocoder_python" links to target "CODEC2::CODEC2" but the target was
gnuradio>   not found.  Perhaps a find_package() call is missing for an IMPORTED
gnuradio>   target, or an ALIAS target is missing?
gnuradio> Call Stack (most recent call first):
gnuradio>   cmake/Modules/GrPybind.cmake:120 (pybind11_add_module)
gnuradio>   gr-vocoder/python/vocoder/bindings/CMakeLists.txt:44 (GR_PYBIND_MAKE_CHECK_HASH)
gnuradio> CMake Error at /nix/store/5d7dgha0nl6hixhszb9x5pfczpblch1a-python3.8-pybind11-2.6.1/share/cmake/pybind11/pybind11Tools.cmake:151 (add_library):
gnuradio>   Target "vocoder_python" links to target "GSM::GSM" but the target was not
gnuradio>   found.  Perhaps a find_package() call is missing for an IMPORTED target, or
gnuradio>   an ALIAS target is missing?
gnuradio> Call Stack (most recent call first):
gnuradio>   cmake/Modules/GrPybind.cmake:120 (pybind11_add_module)
gnuradio>   gr-vocoder/python/vocoder/bindings/CMakeLists.txt:44 (GR_PYBIND_MAKE_CHECK_HASH)
gnuradio> CMake Error at gr-vocoder/lib/CMakeLists.txt:11 (add_library):
gnuradio>   Target "gnuradio-vocoder" links to target "CODEC2::CODEC2" but the target
gnuradio>   was not found.  Perhaps a find_package() call is missing for an IMPORTED
gnuradio>   target, or an ALIAS target is missing?
gnuradio> CMake Error at gr-vocoder/lib/CMakeLists.txt:11 (add_library):
gnuradio>   Target "gnuradio-vocoder" links to target "GSM::GSM" but the target was not
gnuradio>   found.  Perhaps a find_package() call is missing for an IMPORTED target, or
gnuradio>   an ALIAS target is missing?
gnuradio> -- Generating done

wirew0rm · 2021-02-15T23:14:43Z

ah yes, that looks familiar, that's where I'm stuck, too. I'll try to merge my branch with yours and add try to find out if i can get further with it if I find the time this week.

doronbehar · 2021-03-13T15:16:03Z

#106599 will close this. The codec2 issue is not critical and most of the issues discussed above are fixed.

doronbehar added the 0.kind: packaging request Request for a new package to be added label Mar 10, 2020

veprbl added 0.kind: enhancement Add something new 9.needs: package (update) This needs a package to be updated and removed 0.kind: packaging request Request for a new package to be added labels Mar 10, 2020

doronbehar mentioned this issue Apr 4, 2020

comedilib: init at 0.11.0 #84243

Merged

10 tasks

This was referenced Apr 5, 2020

gnuradio: 3.7.13.4 -> 3.7.14.0 & more #84401

Closed

PyGTK deprecation tracking issue #75478

Open

doronbehar mentioned this issue May 16, 2020

gnuradio: incomplete dependencies for gnuradio-companion #87510

Closed

doronbehar mentioned this issue May 31, 2020

gnuradio: 3.7.13.4 -> 3.7.14.0 #89244

Merged

10 tasks

doronbehar mentioned this issue Oct 6, 2020

gnuradio: rewrite #99685

Merged

10 tasks

doronbehar mentioned this issue Dec 10, 2020

gnuradio.pkgs: init + 3.8 -> 3.9 #106599

Merged

10 tasks

doronbehar mentioned this issue Feb 23, 2021

Codec2 isn't detected properly (at least on NixOS) gnuradio/gnuradio#4278

Closed

doronbehar closed this as completed in #106599 Mar 13, 2021

GNU radio 3.8 update meta issue #82263

GNU radio 3.8 update meta issue #82263

Comments

doronbehar commented Mar 10, 2020 • edited Loading

Incompatibilities between 3.7 and 3.8

End user packages

Info

Version status

Version status

Info

Version status

Info

Info

Version status

Components / Libraries type packages

Version status

Version status

Info

Version status

Info

Version status

Info

Version status

Version status

Info

Gnuradio 3.7 branch

Other Improvements

Introduce enable / disable flags

Note

Use a Wrapper Derivation

Rewrite Plugin System

Technical details

Unanswered Questions

nixos-discourse commented Mar 10, 2020

markuskowa commented Mar 11, 2020

doronbehar commented Mar 11, 2020 • edited Loading

markuskowa commented Mar 11, 2020

doronbehar commented Mar 11, 2020

markuskowa commented Mar 11, 2020 • edited Loading

doronbehar commented Mar 28, 2020

gr-foo has a new version compatible with both gnuradio versions

gr-bar has a new version compatible only with 3.8 while the previous version works with 3.7 only

gr-bla hasn't had an update compatible 3.8

wirew0rm commented Mar 29, 2020

doronbehar commented Mar 31, 2020

doronbehar commented Apr 5, 2020

Hoverbear commented May 18, 2020

andrew-d commented Jul 17, 2020

doronbehar commented Jul 17, 2020

doronbehar commented Oct 6, 2020

doronbehar commented Dec 10, 2020

wirew0rm commented Feb 15, 2021

doronbehar commented Feb 15, 2021

wirew0rm commented Feb 15, 2021

doronbehar commented Mar 13, 2021

doronbehar commented Mar 10, 2020 •

edited

Loading

Introduce `enable` / `disable` flags

doronbehar commented Mar 11, 2020 •

edited

Loading

markuskowa commented Mar 11, 2020 •

edited

Loading

`gr-foo` has a new version compatible with both gnuradio versions

`gr-bar` has a new version compatible only with 3.8 while the previous version works with 3.7 only

`gr-bla` hasn't had an update compatible 3.8