Get '.BTF' rejected: Invalid argument (22)! when run basic01-xdp-pass. #38

kwjjyn · 2019-05-10T02:14:37Z

When I use the command below to inject the xdp_pass_kern.o to kernel :

ip link set dev eth1 xdpgeneric obj xdp_pass_kern.o sec xdp

it occered that:

BTF debug data section '.BTF' rejected: Invalid argument (22)!
 - Length:       554
Verifier analysis:

magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 256
str_off: 256
str_len: 274
btf_total_size: 554
[1] FUNC_PROTO (anon) return=2 args=(3 (anon))
[2] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] PTR (anon) type_id=4
[4] STRUCT xdp_md size=20 vlen=5
        data type_id=5 bits_offset=0
        data_end type_id=5 bits_offset=32
        data_meta type_id=5 bits_offset=64
        ingress_ifindex type_id=5 bits_offset=96
        rx_queue_index type_id=5 bits_offset=128
[5] TYPEDEF __u32 type_id=6
[6] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)
[7] FUNC xdp_prog_simple type_id=1
[8] INT char size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
[9] ARRAY (anon) type_id=8 index_type_id=10 nr_elems=4
[10] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none)
[11] VAR _license type_id=9 linkage=1
[12] DATASEC license size=0 vlen=1 size == 0

And then I use the command below to check whether it's attached to the kernel , the answer is yes.

# ip link show dev eth1
6: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:32:3b:e0 brd ff:ff:ff:ff:ff:ff
    prog/xdp id 131163 tag b8a375b5b20c00
    [...]

SO, why would that Invalid argument(22) happened ?
And if I use libbpf library ,the print log is below (I added some print log)

# ./xdp_pass_user --dev eth1 --skb-mode
libbpf: Error loading ELF section .BTF: -22. Ignored and continue.
BPF-prog is only loaded by the kernel and get the prog_fd.
prog_fd is : 3
start attaching the prog_fd to a kernel hook point
the err code of xdp_link_attach() is 0
xdp_link_attach() success
Success: Loading XDP prog name:xdp_prog_simple(id:131169) on device:eth1(ifindex:6)

it seems the same probelm.
But when I run the samples/bpf in the kernel tree , there's no probelm.
I searched the google but didn't get any useful info. I have no idea how to solve this probelm.
could you give me some advice?

more info:
the kernel version is 5.1.0-rc7+ .
the llvm version is 9.0.0

tohojo · 2019-05-10T08:16:50Z

Does this only happen with a 5.1 kernel? Can't reproduce it on my machine (running 5.0)... Also, what's the output of `llvm-objdump -s xdp_pass_kern.o` ?

kwjjyn · 2019-05-10T08:58:45Z

@tohojo Hi，the output of 'llvm-objdump -S xdp_pass_kern.o' is :

xdp_pass_kern.o:        file format ELF64-BPF

Disassembly of section xdp:

0000000000000000 xdp_prog_simple:
; {
       0:       7b 1a f8 ff 00 00 00 00 *(u64 *)(r10 - 8) = r1
;     return XDP_PASS;
       1:       b7 00 00 00 02 00 00 00 r0 = 2
       2:       95 00 00 00 00 00 00 00 exit

and the output of the command you said is :

# llvm-objdump -s xdp_pass_kern.o
xdp_pass_kern.o:        file format ELF64-BPF

Contents of section .strtab:
 0000 002e6465 6275675f 61626272 6576002e  ..debug_abbrev..
 0010 74657874 002e7265 6c2e4254 462e6578  text..rel.BTF.ex
 0020 74002e64 65627567 5f737472 00786470  t..debug_str.xdp
 0030 002e6465 6275675f 6d616369 6e666f00  ..debug_macinfo.
 0040 2e72656c 2e646562 75675f69 6e666f00  .rel.debug_info.
 0050 5f6c6963 656e7365 002e7265 6c2e6465  _license..rel.de
 0060 6275675f 6c696e65 002e7265 6c2e6465  bug_line..rel.de
 0070 6275675f 6672616d 65007864 705f7072  bug_frame.xdp_pr
 0080 6f675f73 696d706c 65007864 705f7061  og_simple.xdp_pa
 0090 73735f6b 65726e2e 63002e73 74727461  ss_kern.c..strta
 00a0 62002e73 796d7461 62002e72 656c2e42  b..symtab..rel.B
 00b0 544600                               TF.
Contents of section xdp:
 0000 7b1af8ff 00000000 b7000000 02000000  {...............
 0010 95000000 00000000                    ........
Contents of section license:
 0000 47504c00                             GPL.
Contents of section .debug_str:
 0000 636c616e 67207665 7273696f 6e20392e  clang version 9.
 0010 302e302d 73766e33 36303132 362d317e  0.0-svn360126-1~
 0020 65787031 2b307e32 30313930 35303730  exp1+0~201905070
 0030 39353433 342e3132 33327e31 2e676270  95434.1232~1.gbp
 0040 65353732 32642028 7472756e 6b290078  e5722d (trunk).x
 0050 64705f70 6173735f 6b65726e 2e63002f  dp_pass_kern.c./
 0060 686f6d65 2f6b6f75 6a69616e 7975616e  home/koujianyuan
 0070 2f786470 2d747574 6f726961 6c2f6261  /xdp-tutorial/ba
 0080 73696330 312d7864 702d7061 7373005f  sic01-xdp-pass._
 0090 6c696365 6e736500 63686172 005f5f41  license.char.__A
 00a0 52524159 5f53495a 455f5459 50455f5f  RRAY_SIZE_TYPE__
 00b0 00756e73 69676e65 6420696e 74005844  .unsigned int.XD
 00c0 505f4142 4f525445 44005844 505f4452  P_ABORTED.XDP_DR
 00d0 4f500058 44505f50 41535300 5844505f  OP.XDP_PASS.XDP_
 00e0 54580058 44505f52 45444952 45435400  TX.XDP_REDIRECT.
 00f0 7864705f 61637469 6f6e0078 64705f70  xdp_action.xdp_p
 0100 726f675f 73696d70 6c650069 6e740063  rog_simple.int.c
 0110 74780064 61746100 5f5f7533 32006461  tx.data.__u32.da
 0120 74615f65 6e640064 6174615f 6d657461  ta_end.data_meta
 0130 00696e67 72657373 5f696669 6e646578  .ingress_ifindex
 0140 0072785f 71756575 655f696e 64657800  .rx_queue_index.
 0150 7864705f 6d6400                      xdp_md.
Contents of section .debug_abbrev:
 0000 01110125 0e130503 0e10171b 0e110112  ...%............
 0010 06000002 3400030e 49133f19 3a0b3b0b  ....4...I.?.:.;.
 0020 02180000 03010149 13000004 21004913  .......I....!.I.
 0030 370b0000 05240003 0e3e0b0b 0b000006  7....$...>......
 0040 2400030e 0b0b3e0b 00000704 01491303  $.....>......I..
 0050 0e0b0b3a 0b3b0500 00082800 030e1c0f  ...:.;....(.....
 0060 0000092e 01110112 06401803 0e3a0b3b  .........@...:.;
 0070 0b271949 133f1900 000a0500 0218030e  .'.I.?..........
 0080 3a0b3b0b 49130000 0b0f0049 1300000c  :.;.I......I....
 0090 1301030e 0b0b3a0b 3b050000 0d0d0003  ......:.;.......
 00a0 0e49133a 0b3b0538 0b00000e 16004913  .I.:.;.8......I.
 00b0 030e3a0b 3b0b0000 00                 ..:.;....
Contents of section .debug_info:
 0000 13010000 04000000 00000801 00000000  ................
 0010 0c000000 00000000 00000000 00000000  ................
 0020 00000000 00001800 00000200 0000003f  ...............?
 0030 00000001 0d090300 00000000 00000003  ................
 0040 4b000000 04520000 00040005 00000000  K....R..........
 0050 06010600 00000008 07078500 00000000  ................
 0060 00000402 ef0b0800 00000000 08000000  ................
 0070 00010800 00000002 08000000 00030800  ................
 0080 00000004 00050000 00000704 09000000  ................
 0090 00000000 00180000 00015a00 00000001  ..........Z.....
 00a0 07b40000 000a0291 00000000 000107bb  ................
 00b0 00000000 05000000 0005040b c0000000  ................
 00c0 0c000000 001402fa 0b0d0000 00000b01  ................
 00d0 000002fb 0b000d00 0000000b 01000002  ................
 00e0 fc0b040d 00000000 0b010000 02fd0b08  ................
 00f0 0d000000 000b0100 0002ff0b 0c0d0000  ................
 0100 00000b01 00000200 0c10000e 85000000  ................
 0110 00000000 031b00                      .......
Contents of section .rel.debug_info:
 0000 06000000 00000000 0a000000 1a000000  ................
 0010 0c000000 00000000 0a000000 02000000  ................
 0020 12000000 00000000 0a000000 03000000  ................
 0030 16000000 00000000 0a000000 1c000000  ................
 0040 1a000000 00000000 0a000000 04000000  ................
 0050 1e000000 00000000 01000000 19000000  ................
 0060 2b000000 00000000 0a000000 05000000  +...............
 0070 37000000 00000000 01000000 1d000000  7...............
 0080 4c000000 00000000 0a000000 06000000  L...............
 0090 53000000 00000000 0a000000 07000000  S...............
 00a0 5e000000 00000000 0a000000 0e000000  ^...............
 00b0 67000000 00000000 0a000000 09000000  g...............
 00c0 6d000000 00000000 0a000000 0a000000  m...............
 00d0 73000000 00000000 0a000000 0b000000  s...............
 00e0 79000000 00000000 0a000000 0c000000  y...............
 00f0 7f000000 00000000 0a000000 0d000000  ................
 0100 86000000 00000000 0a000000 08000000  ................
 0110 8d000000 00000000 01000000 19000000  ................
 0120 9b000000 00000000 0a000000 0f000000  ................
 0130 a9000000 00000000 0a000000 11000000  ................
 0140 b5000000 00000000 0a000000 10000000  ................
 0150 c1000000 00000000 0a000000 18000000  ................
 0160 ca000000 00000000 0a000000 12000000  ................
 0170 d7000000 00000000 0a000000 14000000  ................
 0180 e4000000 00000000 0a000000 15000000  ................
 0190 f1000000 00000000 0a000000 16000000  ................
 01a0 fe000000 00000000 0a000000 17000000  ................
 01b0 10010000 00000000 0a000000 13000000  ................
Contents of section .debug_macinfo:
 0000 00                                   .
Contents of section .BTF:
 0000 9feb0100 18000000 00000000 00010000  ................
 0010 00010000 12010000 00000000 0100000d  ................
 0020 02000000 00000000 03000000 83000000  ................
 0030 00000001 04000000 20000001 00000000  ........ .......
 0040 00000002 04000000 87000000 05000004  ................
 0050 14000000 8e000000 05000000 00000000  ................
 0060 93000000 05000000 20000000 9c000000  ........ .......
 0070 05000000 40000000 a6000000 05000000  ....@...........
 0080 60000000 b6000000 05000000 80000000  `...............
 0090 c5000000 00000008 06000000 cb000000  ................
 00a0 00000001 04000000 20000000 d8000000  ........ .......
 00b0 0000000c 01000000 e8000000 00000001  ................
 00c0 01000000 08000001 00000000 00000003  ................
 00d0 00000000 08000000 0a000000 04000000  ................
 00e0 ed000000 00000001 04000000 20000000  ............ ...
 00f0 01010000 0000000e 09000000 01000000  ................
 0100 0a010000 0100000f 00000000 0b000000  ................
 0110 00000000 04000000 00786470 002f686f  .........xdp./ho
 0120 6d652f6b 6f756a69 616e7975 616e2f78  me/koujianyuan/x
 0130 64702d74 75746f72 69616c2f 62617369  dp-tutorial/basi
 0140 6330312d 7864702d 70617373 2f786470  c01-xdp-pass/xdp
 0150 5f706173 735f6b65 726e2e63 00696e74  _pass_kern.c.int
 0160 20207864 705f7072 6f675f73 696d706c    xdp_prog_simpl
 0170 65287374 72756374 20786470 5f6d6420  e(struct xdp_md
 0180 2a637478 29002020 20207265 7475726e  *ctx).    return
 0190 20584450 5f504153 533b0069 6e740078   XDP_PASS;.int.x
 01a0 64705f6d 64006461 74610064 6174615f  dp_md.data.data_
 01b0 656e6400 64617461 5f6d6574 6100696e  end.data_meta.in
 01c0 67726573 735f6966 696e6465 78007278  gress_ifindex.rx
 01d0 5f717565 75655f69 6e646578 005f5f75  _queue_index.__u
 01e0 33320075 6e736967 6e656420 696e7400  32.unsigned int.
 01f0 7864705f 70726f67 5f73696d 706c6500  xdp_prog_simple.
 0200 63686172 005f5f41 52524159 5f53495a  char.__ARRAY_SIZ
 0210 455f5459 50455f5f 005f6c69 63656e73  E_TYPE__._licens
 0220 65006c69 63656e73 6500               e.license.
Contents of section .rel.BTF:
 0000 10010000 00000000 0a000000 1d000000  ................
Contents of section .BTF.ext:
 0000 9feb0100 18000000 00000000 14000000  ................
 0010 14000000 2c000000 08000000 01000000  ....,...........
 0020 01000000 00000000 07000000 10000000  ................
 0030 01000000 02000000 00000000 05000000  ................
 0040 45000000 001c0000 08000000 05000000  E...............
 0050 6e000000 05280000                    n....(..
Contents of section .rel.BTF.ext:
 0000 24000000 00000000 00000000 19000000  $...............
 0010 38000000 00000000 00000000 19000000  8...............
 0020 48000000 00000000 00000000 19000000  H...............
Contents of section .debug_frame:
 0000 0c000000 ffffffff 04000800 087c0b00  .............|..
 0010 14000000 00000000 00000000 00000000  ................
 0020 18000000 00000000                    ........
Contents of section .rel.debug_frame:
 0000 14000000 00000000 0a000000 1b000000  ................
 0010 18000000 00000000 01000000 19000000  ................
Contents of section .debug_line:
 0000 85000000 04006a00 00000801 01fb0e0d  ......j.........
 0010 00010101 01000000 01000001 2f757372  ............/usr
 0020 2f696e63 6c756465 2f6c696e 7578002f  /include/linux./
 0030 7573722f 696e636c 7564652f 61736d2d  usr/include/asm-
 0040 67656e65 72696300 00786470 5f706173  generic..xdp_pas
 0050 735f6b65 726e2e63 00000000 6270662e  s_kern.c....bpf.
 0060 68000100 00696e74 2d6c6c36 342e6800  h....int-ll64.h.
 0070 02000000 00090200 00000000 00000019  ................
 0080 05050a22 02020001 01                 ...".....
Contents of section .rel.debug_line:
 0000 77000000 00000000 01000000 19000000  w...............
Contents of section .symtab:
 0000 00000000 00000000 00000000 00000000  ................
 0010 00000000 00000000 8a000000 0400f1ff  ................
 0020 00000000 00000000 00000000 00000000  ................
 0030 00000000 00000500 00000000 00000000  ................
 0040 00000000 00000000 00000000 00000500  ................
 0050 4f000000 00000000 00000000 00000000  O...............
 0060 00000000 00000500 5f000000 00000000  ........_.......
 0070 00000000 00000000 00000000 00000500  ................
 0080 8f000000 00000000 00000000 00000000  ................
 0090 00000000 00000500 98000000 00000000  ................
 00a0 00000000 00000000 00000000 00000500  ................
 00b0 9d000000 00000000 00000000 00000000  ................
 00c0 00000000 00000500 b1000000 00000000  ................
 00d0 00000000 00000000 00000000 00000500  ................
 00e0 be000000 00000000 00000000 00000000  ................
 00f0 00000000 00000500 ca000000 00000000  ................
 0100 00000000 00000000 00000000 00000500  ................
 0110 d3000000 00000000 00000000 00000000  ................
 0120 00000000 00000500 dc000000 00000000  ................
 0130 00000000 00000000 00000000 00000500  ................
 0140 e3000000 00000000 00000000 00000000  ................
 0150 00000000 00000500 f0000000 00000000  ................
 0160 00000000 00000000 00000000 00000500  ................
 0170 fb000000 00000000 00000000 00000000  ................
 0180 00000000 00000500 0b010000 00000000  ................
 0190 00000000 00000000 00000000 00000500  ................
 01a0 0f010000 00000000 00000000 00000000  ................
 01b0 00000000 00000500 13010000 00000000  ................
 01c0 00000000 00000000 00000000 00000500  ................
 01d0 18010000 00000000 00000000 00000000  ................
 01e0 00000000 00000500 1e010000 00000000  ................
 01f0 00000000 00000000 00000000 00000500  ................
 0200 27010000 00000000 00000000 00000000  '...............
 0210 00000000 00000500 31010000 00000000  ........1.......
 0220 00000000 00000000 00000000 00000500  ................
 0230 41010000 00000000 00000000 00000000  A...............
 0240 00000000 00000500 50010000 00000000  ........P.......
 0250 00000000 00000000 00000000 03000300  ................
 0260 00000000 00000000 00000000 00000000  ................
 0270 00000000 03000600 00000000 00000000  ................
 0280 00000000 00000000 00000000 03000e00  ................
 0290 00000000 00000000 00000000 00000000  ................
 02a0 00000000 03001000 00000000 00000000  ................
 02b0 00000000 00000000 50000000 11000400  ........P.......
 02c0 00000000 00000000 04000000 00000000  ................
 02d0 7a000000 12000300 00000000 00000000  z...............
 02e0 18000000 00000000                    ........

Another thing:
I could run the /samples/bpf/xdp_*.c and found the version of libbpf between the kernel tree's /tools/lib/bpf/ and xdp-tutorial/libbpf/src is different.

tohojo · 2019-05-10T10:02:47Z

Great, thanks! Does this also happen on older kernels, or is it specific to 5.1?

kwjjyn · 2019-05-13T01:23:49Z

@tohojo oh，I didn't try it on older kernels. I don't know either

tohojo · 2019-05-13T12:33:47Z

kwjjyn <[email protected]> writes:

@tohojo oh，I didn't try it on older kernels. I don't know either

Right, I'll see if I can't get this tested on a 5.1 kernel and see if I can reproduce, then...

kwjjyn · 2019-05-13T14:36:52Z

@tohojo Thanks! Do you think I really attach the xdp program to the hook and run correctly ? maybe the problem is only about BTF and it has no effect on the xdp program?

tohojo · 2019-05-13T14:48:10Z

kwjjyn <[email protected]> writes:

@tohojo Thanks! Do you think I really attach the xdp program to the hook and run correctly ? maybe the problem is only about BTF and it has no effect on the xdp program?

Yeah, it looks like it is attaching the program, but fails to load the BTF values associated with it (which is just debug stuff in this case).

netoptimizer · 2019-06-05T11:35:44Z

AFAIK this might have been fixed in libbpf ... trying to find the libbpf commit

florianl · 2019-06-30T11:49:24Z

Maybe not directly related to xdp-roject/xdp-tuorial, but I'm facing the very same error BTF debug data section '.BTF' rejected: Invalid argument (22)!, when following the instructions on [1].

Some words on my environment:

$ clang --version
clang version 8.0.0 (Fedora 8.0.0-1.fc30)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
$ uname -a
Linux x250 5.2.0-0.rc6.git1.2.fc31.x86_64 #1 SMP Tue Jun 25 16:06:42 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[1] https://github.com/borkmann/clsact/blob/master/test_bpf.sh#L79-L105

SPYFF · 2019-07-16T08:50:35Z

Try to compile with the following flags: -g -c -O2
With Clang 8.0 this (the -O2 flag) solved the problem.

florianl · 2019-07-17T17:18:33Z

thanks a lot for the hint @SPYFF
with the flag, everything is just fine!

simonhf · 2020-02-10T19:45:27Z

I also got the same BTF issue with a newer kernel:

$ clang --version
clang version 9.0.0-2 (tags/RELEASE_900/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
$ uname -a
Linux ubuntu 5.3.0-26-generic #28-Ubuntu SMP Wed Dec 18 05:37:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Note: xdp_pass_kern.c already seems to be compiled with flags -g -c -O2 so this wasn't a fix for me.

Also, after loading the following output is missing the tag info. Is that to be expected?

$ ip link show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 xdpgeneric qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    prog/xdp id 258

The output of llvm-objdump is:

$ llvm-objdump -S xdp_pass_kern.o 

xdp_pass_kern.o:        file format ELF64-BPF


Disassembly of section xdp:

0000000000000000 xdp_prog_simple:
;       return XDP_PASS;
       0:       b7 00 00 00 02 00 00 00 r0 = 2
       1:       95 00 00 00 00 00 00 00 exit

Any ideas how to fix this?

simonhf · 2020-02-10T20:31:38Z

Maybe due to this [1]?

We can't yet convert BPF programs that are loaded with iproute2 to new
BTF-defined maps, until iprout2 uses libbpf as a loader. I missed that
sample_map_ret0.c is used with iproute2, will undo conversion for it.

[1] https://www.spinics.net/lists/netdev/msg584085.html

tohojo · 2020-02-10T21:31:32Z

@simonhf yes, if you are using iproute2 to attach an eBPF program, you will get the BTF errors. However, they shouldn't be fatal, so you can just ignore them until iproute2 is fixed - or you can use another loader, such as that in xdp-tools: https://github.com/xdp-project/xdp-tools

mestery · 2020-06-03T13:52:36Z

This still seems to be a problem, I'm trying to run the tutorials with this version of kernel:

[vagrant@xdp-tutorial basic01-xdp-pass]$ uname -a
Linux xdp-tutorial 5.6.13-300.fc32.x86_64 #1 SMP Thu May 14 22:51:37 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[vagrant@xdp-tutorial basic01-xdp-pass]$

This is the basic01-xdp-pass test, and when I try to load this with iproute2 from Fedora 32 I get the following:

[vagrant@xdp-tutorial basic01-xdp-pass]$ sudo ip link set dev lo xdpgeneric obj xdp_pass_kern.o sec xdp

BTF debug data section '.BTF' rejected: Invalid argument (22)!
 - Length:       510
Verifier analysis:

magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 256
str_off: 256
str_len: 230
btf_total_size: 510
[1] PTR (anon) type_id=2
[2] STRUCT xdp_md size=20 vlen=5
	data type_id=3 bits_offset=0
	data_end type_id=3 bits_offset=32
	data_meta type_id=3 bits_offset=64
	ingress_ifindex type_id=3 bits_offset=96
	rx_queue_index type_id=3 bits_offset=128
[3] TYPEDEF __u32 type_id=4
[4] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)
[5] FUNC_PROTO (anon) return=6 args=(1 ctx)
[6] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[7] FUNC xdp_prog_simple type_id=5
[8] INT char size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
[9] ARRAY (anon) type_id=8 index_type_id=10 nr_elems=4
[10] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none)
[11] VAR _license type_id=9 linkage=1
[12] DATASEC license size=0 vlen=1 size == 0

[vagrant@xdp-tutorial basic01-xdp-pass]$

Is there a specific version of tools that is required to use these tutorials which is not referenced in the tutorial itself?

tohojo · 2020-06-03T14:08:46Z

Kyle Mestery <[email protected]> writes:

This still seems to be a problem, I'm trying to run the tutorials with this version of kernel: ``` ***@***.*** basic01-xdp-pass]$ uname -a Linux xdp-tutorial 5.6.13-300.fc32.x86_64 #1 SMP Thu May 14 22:51:37 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ***@***.*** basic01-xdp-pass]$ ``` This is the basic01-xdp-pass test, and when I try to load this with iproute2 from Fedora 32 I get the following: ``` ***@***.*** basic01-xdp-pass]$ sudo ip link set dev lo xdpgeneric obj xdp_pass_kern.o sec xdp BTF debug data section '.BTF' rejected: Invalid argument (22)! - Length: 510 Verifier analysis: magic: 0xeb9f version: 1 flags: 0x0 hdr_len: 24 type_off: 0 type_len: 256 str_off: 256 str_len: 230 btf_total_size: 510 [1] PTR (anon) type_id=2 [2] STRUCT xdp_md size=20 vlen=5 data type_id=3 bits_offset=0 data_end type_id=3 bits_offset=32 data_meta type_id=3 bits_offset=64 ingress_ifindex type_id=3 bits_offset=96 rx_queue_index type_id=3 bits_offset=128 [3] TYPEDEF __u32 type_id=4 [4] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none) [5] FUNC_PROTO (anon) return=6 args=(1 ctx) [6] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED [7] FUNC xdp_prog_simple type_id=5 [8] INT char size=1 bits_offset=0 nr_bits=8 encoding=SIGNED [9] ARRAY (anon) type_id=8 index_type_id=10 nr_elems=4 [10] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none) [11] VAR _license type_id=9 linkage=1 [12] DATASEC license size=0 vlen=1 size == 0 ***@***.*** basic01-xdp-pass]$ ``` Is there a specific version of tools that is required to use these tutorials which is not referenced in the tutorial itself?

This is just because iproute2 doesn't understand BTF. Loading the program should still work, though?

mestery · 2020-06-03T15:07:18Z

This is just because iproute2 doesn't understand BTF. Loading the program should still work, though?

It does not:

[vagrant@xdp-tutorial basic01-xdp-pass]$ sudo ./xdp_pass_user --dev lo --skb-mode
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: Error in bpf_object__probe_global_data():Operation not permitted(1). Couldn't create simple array map.
libbpf: load bpf program failed: Operation not permitted
libbpf: failed to load program 'xdp'
libbpf: failed to load object 'xdp_pass_kern.o'
ERR: loading BPF-OBJ file(xdp_pass_kern.o) (-22): Invalid argument
ERR: loading file: xdp_pass_kern.o
[vagrant@xdp-tutorial basic01-xdp-pass]$

tohojo · 2020-06-03T15:29:36Z

That's looks like a different error - check ulimit -l ?

mestery · 2020-06-03T15:31:47Z

ulimit -l

[vagrant@xdp-tutorial basic01-xdp-pass]$ ulimit -l
64
[vagrant@xdp-tutorial basic01-xdp-pass]$

mestery · 2020-06-03T16:36:39Z

So it seems if I set ulimit -l higher as root inside the VM, and then run the loader as root in that same shell, it works.

tohojo · 2020-06-04T09:56:53Z

Kyle Mestery <[email protected]> writes:

So it seems if I set `ulimit -l` higher as root inside the VM, and then run the loader as root in that same shell, it works.

Yup, setting the ulimit doesn't work with sudo (unless you create a wrapper script), so just having a 'real' root shell is probably the easiest way to get around this.

nicklesimba · 2020-06-16T15:16:47Z

Any fix to this problem? I am new to XDP/EBPF and am hitting a wall here. Same issue as OP. And it does seem as if -O2 -c -g are already being used

tohojo · 2020-06-16T16:51:00Z

Any fix to this problem? I am new to XDP/EBPF and am hitting a wall here. Same issue as OP. And it does seem as if -O2 -c -g are already being used

Which one? There are two different issues being described above

nicklesimba · 2020-06-17T18:57:13Z

Hi, sorry for not clarifying. I have the exact outputs that mestery posted when trying to load the program via the tutorial basic01 up until checking my ulimit -l to be 64 as well.

I am not sure how to proceed from here or why the ulimit is an issue by default when trying to follow the tutorial. Thanks!

tohojo · 2020-06-18T17:49:37Z

To fix the ulimit issue you'll need to get a real root shell and run commands there, instead of running each one through sudo. So instead of:

sudo ./xdp_pass_user --dev lo --skb-mode

you'd do:

sudo -i
ulimit -l unlimited
./xdp_pass_user --dev lo --skb-mode

simonhf · 2020-07-16T23:59:13Z

Interestingly I get the same error if I use clang version 10, but using clang version 9 then no error and everything works fine!

Here's some output associated with clang version 10:

$ clang --version | egrep version
Alpine clang version 10.0.0 (https://gitlab.alpinelinux.org/alpine/aports.git 7445adce501f8473efdb93b17b5eaf2f1445ed4c)

$ wc --bytes src/example_kern.*
 1069 src/example_kern.c
10087 src/example_kern.ll
 6616 src/example_kern.o

$ llvm-objdump -S src/example_kern.o

src/example_kern.o:    file format ELF64-BPF


Disassembly of section xdp_sock:

0000000000000000 xdp_sock_prog:
       0:       61 11 10 00 00 00 00 00 r1 = *(u32 *)(r1 + 16)
       1:       63 1a fc ff 00 00 00 00 *(u32 *)(r10 - 4) = r1
       2:       bf a2 00 00 00 00 00 00 r2 = r10
       3:       07 02 00 00 fc ff ff ff r2 += -4
       4:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
       6:       85 00 00 00 01 00 00 00 call 1
       7:       bf 01 00 00 00 00 00 00 r1 = r0
       8:       b7 00 00 00 02 00 00 00 r0 = 2
       9:       15 01 05 00 00 00 00 00 if r1 == 0 goto +5 <LBB0_2>
      10:       61 a2 fc ff 00 00 00 00 r2 = *(u32 *)(r10 - 4)
      11:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
      13:       b7 03 00 00 00 00 00 00 r3 = 0
      14:       85 00 00 00 33 00 00 00 call 51

0000000000000078 LBB0_2:
      15:       95 00 00 00 00 00 00 00 exit

And here's some output associated with clang version 9:

$ clang --version | egrep version
clang version 9.0.0-2 (tags/RELEASE_900/final)

$ wc --bytes src/example_kern.*
 1069 src/example_kern.c
 9514 src/example_kern.ll
 6344 src/example_kern.o

$ llvm-objdump -S src/example_kern.o

src/example_kern.o:    file format ELF64-BPF


Disassembly of section xdp_sock:

0000000000000000 xdp_sock_prog:
;     int index = ctx->rx_queue_index;
       0:       61 11 10 00 00 00 00 00 r1 = *(u32 *)(r1 + 16)
       1:       63 1a fc ff 00 00 00 00 *(u32 *)(r10 - 4) = r1
       2:       bf a2 00 00 00 00 00 00 r2 = r10
       3:       07 02 00 00 fc ff ff ff r2 += -4
;     if (bpf_map_lookup_elem(&xsks_map, &index))
       4:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
       6:       85 00 00 00 01 00 00 00 call 1
       7:       bf 01 00 00 00 00 00 00 r1 = r0
       8:       b7 00 00 00 02 00 00 00 r0 = 2
       9:       15 01 05 00 00 00 00 00 if r1 == 0 goto +5 <LBB0_2>
;         return bpf_redirect_map(&xsks_map, index, 0);
      10:       61 a2 fc ff 00 00 00 00 r2 = *(u32 *)(r10 - 4)
      11:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
      13:       b7 03 00 00 00 00 00 00 r3 = 0
      14:       85 00 00 00 33 00 00 00 call 51

0000000000000078 LBB0_2:
; }
      15:       95 00 00 00 00 00 00 00 exit

@tohojo said above "Yeah, it looks like it is attaching the program, but fails to load the BTF values associated with it (which is just debug stuff in this case)."

In this case, it looks like the source code (AKA debug stuff?) is missing from the llvm-objdump of the .o file compiled with clang version 10, or?

Why could clang version 10 not be putting that in (note: it has the same command line as used with clang version 9)?

And is that the reason for my issue?

simonhf · 2020-07-22T01:00:58Z

Okay. I figured out the issue. Yay!

Using llvm-objdump --section-headers --full-contents -S src/example_kern.o then I could dump the respective .BTF sections and discovered the issue happens for me because the .BTF section contains the path to the original source file. However, I was running clang v10 from a docker Alpine container which embedded a source file path which is unknown outside the container. However, having the unknown source file path seems enough to trigger the BTF errors at run-time.

I tried to invoke clang without the -g option and hoped this would result in no source file path being embedded in the .BTF section, and thus also make the BPF executable more distributable [1] too. Luckily this seemed to work! Without the -g option then no .BTF section is generate at all :-)

[1] #138

Wqrld · 2021-07-14T13:03:52Z

Okay. I figured out the issue. Yay!

Using llvm-objdump --section-headers --full-contents -S src/example_kern.o then I could dump the respective .BTF sections and discovered the issue happens for me because the .BTF section contains the path to the original source file. However, I was running clang v10 from a docker Alpine container which embedded a source file path which is unknown outside the container. However, having the unknown source file path seems enough to trigger the BTF errors at run-time.

I tried to invoke clang without the -g option and hoped this would result in no source file path being embedded in the .BTF section, and thus also make the BPF executable more distributable [1] too. Luckily this seemed to work! Without the -g option then no .BTF section is generate at all :-)

[1] #138

Have you found other ways around this? While it is ofcourse not required it would help to keep in all the debug info if issues occur on another machine.

tohojo · 2021-07-14T13:14:58Z

The path doesn't matter. That is only for printing the line information when dumping the byte code. To load programs built with BTF without errors you'll just need to update iproute2 to a newer version that is linked against libbpf. Or you can ignore the BTF error on load as it's not fatal...

Wqrld · 2021-07-14T15:01:08Z

The path doesn't matter. That is only for printing the line information when dumping the byte code. To load programs built with BTF without errors you'll just need to update iproute2 to a newer version that is linked against libbpf. Or you can ignore the BTF error on load as it's not fatal...

I am using the normal xdp-tutorial tutorial loader which is giving this error and thus exiting the program (the xdp program is not actually loaded if you look in ip a. Compiling without -g is giving me a error about fully missing BTF so that does not seem to work either.

With -g flag the program never actually gets loaded:

libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error loading .BTF into kernel: -22.
ERR: loading BPF-OBJ file(xdp_prog_kern.o) (-2): No such file or directory
ERR: loading file: xdp_prog_kern.o

The program loads fine on my development machine but does crashes on a different VM with the above error.

You said it was related to the iproute2 version. My new VM is indeed running iproute 4.15.0-2ubuntu1 while the dev machine is running 5.5.0-1ubuntu1. Is iproute2 still the cause even though i use a custom loader? The new VM can run its own compiled XDP programs just fine but the dev ones can't be transferred to dev and load

EDIT:
Just tried it on a new ubuntu 20.04 5.5.0 VM and the same invalid argument -22 error occured.
Compiled using Clang-10 on the dev machine and CI for the VM
Loading with IP link gives the same error but also gets the ip command hang after that with 100% cpu. The goal is to have it working with a normal userspace c loader though

tohojo · 2021-07-14T17:07:34Z

The loading error is because your kernel is too old. You need at least 4.18. Not sure why the other error occurs, maybe libbpf needs an update? Will have to look into that, but it should be safe to ignore in the meantime...

I ran into this problem[1] which appears to be caused by linux kernel version skew between the host that compiles the filters and the one that loads them. I'd like to be able to compile on my machine and load into Ubuntu 22.04 hosts. Compiling without "-g" increases the version compatibility so let's remove that flag for now. [1] xdp-project/xdp-tutorial#38

kwjjyn changed the title ~~something wrong happened when run basic01-xdp-pass.~~ Get '.BTF' rejected: Invalid argument (22)! when run basic01-xdp-pass. May 10, 2019

simonhf mentioned this issue Jul 24, 2020

Feature request: Make debug build optional and/or fix BTF debug info issue #139

Open

interduo mentioned this issue Sep 5, 2022

libbpf: Error loading BTF: Invalid argument(22) LibreQoE/LibreQoS#63

Closed

thomascalvert-xlnx mentioned this issue May 2, 2024

Fix missing declaration of custom externs in the generated eBPF/uBPF header file. p4lang/p4c#4644

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get '.BTF' rejected: Invalid argument (22)! when run basic01-xdp-pass. #38

Get '.BTF' rejected: Invalid argument (22)! when run basic01-xdp-pass. #38

kwjjyn commented May 10, 2019

tohojo commented May 10, 2019 via email

kwjjyn commented May 10, 2019

tohojo commented May 10, 2019 via email

kwjjyn commented May 13, 2019

tohojo commented May 13, 2019 via email

kwjjyn commented May 13, 2019

tohojo commented May 13, 2019 via email

netoptimizer commented Jun 5, 2019

florianl commented Jun 30, 2019

SPYFF commented Jul 16, 2019

florianl commented Jul 17, 2019

simonhf commented Feb 10, 2020

simonhf commented Feb 10, 2020

tohojo commented Feb 10, 2020

mestery commented Jun 3, 2020

tohojo commented Jun 3, 2020 via email

mestery commented Jun 3, 2020 •

edited

Loading

tohojo commented Jun 3, 2020

mestery commented Jun 3, 2020

mestery commented Jun 3, 2020

tohojo commented Jun 4, 2020 via email

nicklesimba commented Jun 16, 2020 •

edited

Loading

tohojo commented Jun 16, 2020

nicklesimba commented Jun 17, 2020

tohojo commented Jun 18, 2020

simonhf commented Jul 16, 2020 •

edited

Loading

simonhf commented Jul 22, 2020

Wqrld commented Jul 14, 2021

tohojo commented Jul 14, 2021

Wqrld commented Jul 14, 2021 •

edited

Loading

tohojo commented Jul 14, 2021

Get '.BTF' rejected: Invalid argument (22)! when run basic01-xdp-pass. #38

Get '.BTF' rejected: Invalid argument (22)! when run basic01-xdp-pass. #38

Comments

kwjjyn commented May 10, 2019

tohojo commented May 10, 2019 via email

kwjjyn commented May 10, 2019

tohojo commented May 10, 2019 via email

kwjjyn commented May 13, 2019

tohojo commented May 13, 2019 via email

kwjjyn commented May 13, 2019

tohojo commented May 13, 2019 via email

netoptimizer commented Jun 5, 2019

florianl commented Jun 30, 2019

SPYFF commented Jul 16, 2019

florianl commented Jul 17, 2019

simonhf commented Feb 10, 2020

simonhf commented Feb 10, 2020

tohojo commented Feb 10, 2020

mestery commented Jun 3, 2020

tohojo commented Jun 3, 2020 via email

mestery commented Jun 3, 2020 • edited Loading

tohojo commented Jun 3, 2020

mestery commented Jun 3, 2020

mestery commented Jun 3, 2020

tohojo commented Jun 4, 2020 via email

nicklesimba commented Jun 16, 2020 • edited Loading

tohojo commented Jun 16, 2020

nicklesimba commented Jun 17, 2020

tohojo commented Jun 18, 2020

simonhf commented Jul 16, 2020 • edited Loading

simonhf commented Jul 22, 2020

Wqrld commented Jul 14, 2021

tohojo commented Jul 14, 2021

Wqrld commented Jul 14, 2021 • edited Loading

tohojo commented Jul 14, 2021

mestery commented Jun 3, 2020 •

edited

Loading

nicklesimba commented Jun 16, 2020 •

edited

Loading

simonhf commented Jul 16, 2020 •

edited

Loading

Wqrld commented Jul 14, 2021 •

edited

Loading