[Bug] Pinky4 default keymap fails to build on Guix System

[Soundtoxin]

Describe the Bug

I'm following along with the newbs guide and when it has me build the default keymap to test my environment works, it fails with error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?

System Information

• Keyboard: Pinky4

  • Revision (if applicable): v0.2

• Operating system: Guix System GNU/Linux

• AVR GCC version: avr-gcc (GCC) 7.5.0

• ARM GCC version: arm-none-eabi-gcc (GCC) 7.3.1 20180622 (7-2018-q2-update)

• QMK Firmware version: 0.11.64

• Any keyboard related software installed?
  • AutoHotKey
  • Karabiner
  • Other:

Additional Context

brad@anpan:~/ > ~/.local/bin/qmk compile -kb pinky/4 -km default
Ψ Compiling keymap with make -j 1 pinky/4:default


QMK Firmware 0.11.64
Making pinky/4 with keymap default

avr-gcc (GCC) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiling: keyboards/pinky/pinky.c                                                                 In file included from quantum/quantum.h:19:0,
                 from keyboards/pinky/pinky.h:19,
                 from keyboards/pinky/pinky.c:16:
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1456:25: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern size_t strlen_PF(uint_farptr_t src) __ATTR_CONST__; /* program memory can't change */
                         ^~~~~~~~~~~~~
                         uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1472:26: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern size_t strnlen_PF(uint_farptr_t src, size_t len) __ATTR_CONST__; /* program memory can't change */
                          ^~~~~~~~~~~~~
                          uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1487:36: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern void *memcpy_PF(void *dest, uint_farptr_t src, size_t len);
                                    ^~~~~~~~~~~~~
                                    uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1502:36: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern char *strcpy_PF(char *dest, uint_farptr_t src);
                                    ^~~~~~~~~~~~~
                                    uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1522:37: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern char *strncpy_PF(char *dest, uint_farptr_t src, size_t len);
                                     ^~~~~~~~~~~~~
                                     uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1538:36: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern char *strcat_PF(char *dest, uint_farptr_t src);
                                    ^~~~~~~~~~~~~
                                    uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1559:37: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern size_t strlcat_PF(char *dst, uint_farptr_t src, size_t siz);
                                     ^~~~~~~~~~~~~
                                     uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1576:37: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern char *strncat_PF(char *dest, uint_farptr_t src, size_t len);
                                     ^~~~~~~~~~~~~
                                     uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1592:38: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern int strcmp_PF(const char *s1, uint_farptr_t s2) __ATTR_PURE__;
                                      ^~~~~~~~~~~~~
                                      uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1609:39: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern int strncmp_PF(const char *s1, uint_farptr_t s2, size_t n) __ATTR_PURE__;
                                       ^~~~~~~~~~~~~
                                       uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1625:42: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern int strcasecmp_PF(const char *s1, uint_farptr_t s2) __ATTR_PURE__;
                                          ^~~~~~~~~~~~~
                                          uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1643:43: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern int strncasecmp_PF(const char *s1, uint_farptr_t s2, size_t n) __ATTR_PURE__;
                                           ^~~~~~~~~~~~~
                                           uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1659:40: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern char *strstr_PF(const char *s1, uint_farptr_t s2);
                                        ^~~~~~~~~~~~~
                                        uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1671:37: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern size_t strlcpy_PF(char *dst, uint_farptr_t src, size_t siz);
                                     ^~~~~~~~~~~~~
                                     uint_fast8_t
/home/brad/.guix-profile/avr/include/avr/pgmspace.h:1687:36: error: unknown type name ‘uint_farptr_t’; did you mean ‘uint_fast8_t’?
 extern int memcmp_PF(const void *, uint_farptr_t, size_t) __ATTR_PURE__;
                                    ^~~~~~~~~~~~~
                                    uint_fast8_t
 [ERRORS]
 |
 |
 |
make[1]: *** [tmk_core/rules.mk:412: .build/obj_pinky_4_default/keyboards/pinky/pinky.o] Error 1
Make finished with errors
make: *** [Makefile:523: pinky/4:default] Error 1

[aadcg]

Are you still facing this issue? I can't reproduce.

Make sure you have dfu-util, dfu-programmer, arm-none-easi-toolchain, avr-toolchain an avrdude installed in your guix profile.

[Soundtoxin]

I had given up and done the building on another machine running Arch in the end, but I just tried this again and it seemed to work. I guess I'll consider it solved. Thanks for the reminder.

[Soundtoxin]

I was still on the same QMK version as in my initial report. I tried to update it first with pip, which seemed to print the same version upon recompiling, and then I just did a git pull in my qmk_firmware dir. Now I get errors compiling again and I've broken everything. I'm not sure I even care to fix it, but I'm just gonna post it here anyway.

brad@anpan:~/qmk_firmware/ > ~/.local/bin/qmk compile -kb pinky/4 -km default
Error: %s: %s ('ImportError', ImportError('Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll'))
Traceback (most recent call last):
  File "/home/brad/.local/lib/python3.8/site-packages/qmk_cli/script_qmk.py", line 76, in main
    import qmk.cli  # noqa
  File "/home/brad/qmk_firmware/lib/python/qmk/cli/__init__.py", line 224, in <module>
    __import__(subcommand)
  File "/home/brad/qmk_firmware/lib/python/qmk/cli/console.py", line 9, in <module>
    import hid
  File "/home/brad/.local/lib/python3.8/site-packages/hid/__init__.py", line 30, in <module>
    raise ImportError(error)
ImportError: Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll

A shame qmk isn't packaged properly in guix. pip is truly horrible and I'd ideally never touch it again.

[aadcg]

What if you try to add that local bin directory to your path? (export PATH="/home/brad/.local/bin:$PATH")

EDIT: Also, you can ditch the qmk CLI wrapper if you're comfortable with the codebase. For instance, running ~/.local/bin/qmk compile -kb pinky/4 -km default is equivalent to make pinky/4:default when the current directory is the root of your qmk git checkout.

From the hacker's perspective, the CLI is just yet another layer of unnecessary complexity. But I do acknowledge its value and necessity, since not all keyboard nerds are hackers.

[Soundtoxin]

What if you try to add that local bin directory to your path? (export PATH="/home/brad/.local/bin:$PATH")

I seem to get an identical error after sorting out my PATH issues

EDIT: Also, you can ditch the qmk CLI wrapper if you're comfortable with the codebase.

I wouldn't say I am, haha. I'm more comfortable with sysadmin-type stuff than programming.

For instance, running ~/.local/bin/qmk compile -kb pinky/4 -km default is equivalent to make pinky/4:default when the current directory is the root of your qmk git checkout.

brad@anpan:~/qmk_firmware/ > make pinky/4:default
QMK Firmware 0.14.23
Traceback (most recent call last):
  File "/home/brad/.local/lib/python3.8/site-packages/qmk_cli/script_qmk.py", line 76, in main
    import qmk.cli  # noqa
  File "/home/brad/qmk_firmware/lib/python/qmk/cli/__init__.py", line 224, in <module>
    __import__(subcommand)
  File "/home/brad/qmk_firmware/lib/python/qmk/cli/console.py", line 9, in <module>
    import hid
  File "/home/brad/.local/lib/python3.8/site-packages/hid/__init__.py", line 30, in <module>
    raise ImportError(error)
ImportError: Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll
Makefile:539: target 'Error' doesn't match the target pattern
Traceback (most recent call last):
  File "/home/brad/.local/lib/python3.8/site-packages/qmk_cli/script_qmk.py", line 76, in main
    import qmk.cli  # noqa
  File "/home/brad/qmk_firmware/lib/python/qmk/cli/__init__.py", line 224, in <module>
    __import__(subcommand)
  File "/home/brad/qmk_firmware/lib/python/qmk/cli/console.py", line 9, in <module>
    import hid
  File "/home/brad/.local/lib/python3.8/site-packages/hid/__init__.py", line 30, in <module>
    raise ImportError(error)
ImportError: Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll
ERROR: Cannot run "qmk hello"!

 Please run qmk setup to install all the dependencies QMK requires.

make: *** [Makefile:492: pinky/4:default] Error 1

brad@anpan:~/qmk_firmware/ > qmk setup                                                                                                                        Warning: Could not import qmk.cli.console: ImportError, Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll                                  Ψ Found qmk_firmware at /home/brad/qmk_firmware.                                                                                                              Warning: Could not import qmk.cli.console: ImportError, Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll                                  Ψ QMK Doctor is checking your environment.                                                                                                                    Ψ CLI version: 1.0.0                                                                                                                                          Ψ QMK home: /home/brad/qmk_firmware                                                                                                                           Ψ Detected Linux.                                                                                                                                             ⚠ Missing or outdated udev rules for 'atmel-dfu' boards. Run 'sudo cp /home/brad/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.                     ⚠ Missing or outdated udev rules for 'kiibohd' boards. Run 'sudo cp /home/brad/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.                       ⚠ Missing or outdated udev rules for 'stm32' boards. Run 'sudo cp /home/brad/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.                         ⚠ Missing or outdated udev rules for 'bootloadhid' boards. Run 'sudo cp /home/brad/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.                   ⚠ Missing or outdated udev rules for 'usbasploader' boards. Run 'sudo cp /home/brad/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.                  ⚠ Missing or outdated udev rules for 'massdrop' boards. Run 'sudo cp /home/brad/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.                      ⚠ Missing or outdated udev rules for 'caterina' boards. Run 'sudo cp /home/brad/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.                      ⚠ Missing or outdated udev rules for 'hid-bootloader' boards. Run 'sudo cp /home/brad/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.                Ψ Git branch: master
Ψ All dependencies are installed.
Ψ Found arm-none-eabi-gcc version 7.3.1
Ψ Found avr-gcc version 7.5.0
Ψ Found avrdude version 6.3
Ψ Found dfu-util version 0.10
Ψ Found dfu-programmer version 0.7.2
Ψ Submodules are up to date.
Ψ QMK is ready to go, but minor problems were found

[aadcg]

What's the output of which pip?

[sigprof]

Also, you can ditch the qmk CLI wrapper if you're comfortable with the codebase

Actually you can't, because recent qmk_firmware versions actually require that the qmk tool would be present somewhere in $PATH.

ImportError: Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll

This error happens because the hid Python module which is installed by pip is a generic version that assumes a “normal” system, where most shared libraries are located in standard places like /usr/lib or /usr/lib64, and therefore can be found with a plain dlopen() call specifying just the file name without a full path. However, Nix and Guix work differently — there is no global search path for shared libraries, and any shared libraries must be loaded using a full path (which changes when anything in the source or the build environment changes).

For Nix the qmk_firmware repository has the shell.nix file and more supporting files in util/nix/; you need “just” to port that to Guix. However, this may be nontrivial, because the underlying code depends on poetry and poetry2nix to install the required Python dependencies, and I'm not sure whether Guix has any equivalents for that. You may need to investigate how the software using Python should be packaged for Guix.

If someone decides to work on this, here are the poetry2nix overrides that I needed to add for the hid and pyusb modules while updating the Nix support in #12855 (then @purcell pushed then to the poetry2nix upstream): nix-community/poetry2nix#300; you would need to do something similar to make those Python modules work in Guix.

[Soundtoxin]

What's the output of which pip?

In the term where I changed my PATH:

brad@anpan:~/ > which pip
/home/brad/.local/bin/pip

normally:

brad@anpan:~/ > which pip
/home/brad/.guix-profile/bin/pip

I recently tried to update my pip by doing

python3 -m pip install --upgrade pip --user

So, I probably have two pips on my system with different versions I guess.

[sigprof]

So I decided to spend a whole day on this, and the result is guix-qmk — for now it contains only the guix-manifest.scm file, which is intended to be used as:

guix environment --manifest=path/to/guix-manifest.scm

The resulting environment will provide the qmk command (QMK CLI) and its Python dependencies, therefore the recent qmk_firmware code won't complain that it's missing. It will also provide the required toolchains for Arm-based and AVR-based MCUs, and some flashing tools (avrdude, dfu-programmer, dfu-util, teensy_loader_cli). After entering this environment the QMK firmware compilation should work as documented (both qmk compile and make should work as expected); qmk flash and qmk console should also work if the udev rules are set up properly (but I did not have time to actually test that yet).

When testing this, you should remove any Python modules installed by pip in ~/.local — the manifest includes the whole chain of Python dependencies, including the problematic hid module (which needed a patch to work in Guix, similar to what was required to make it work in Nix; the pyusb module turned out to be already present in Guix).

And the compile error that was mentioned in the original message apparently was noticed back in #8234 (comment), and is caused by including both the Arm and AVR toolchains in the environment — the underlying problem in Guix is still not fixed, even though that bug report was closed in 2020. However, my guix-manifest.scm has a complicated workaround for that problem, therefore both Arm and AVR toolchains in that environment should work without conflicts.

[sigprof]

If anyone is still watching this issue, I finally updated guix-qmk for compatibility with recent Guix (including the guix shell command) and QMK CLI 1.1.1, so it should be mostly usable. The only potential problem is that the avr-gcc version provided by Guix is now 10.3.0, which is known to produce excessively large binaries in some cases; on the other hand, arm-none-eabi-gcc is still 7.3.1 (7-2018-q2-update).

[aadcg]

@sigprof amazing work! I've been happily using guix-qmk :)

[sigprof]

Note that after #21656 had been merged to the develop branch, the Chibios USB code is no longer compatible with GCC < 8.x due to the usage of _Alignas in compound literals. So if anybody wants to use GNU Guix to compile QMK firmware for ARM-based MCUs, the ARM cross toolchain in Guix needs to be updated — the 7-2018-q2-update version is too old.