Add support for USB HID keyboard over AOAv2

This provides a better input experience, by simulating a physical
keyboard. It converts SDL keyboard events to proper HID events, and send
them over AOAv2.

This is a rewriting and bugfix of the origin code from @amosbird:
<#279 (comment)>

The feature is enabled the command line option -K or --hid-keyboard,
and is only available on Linux, over USB.

Refs <https://source.android.com/devices/accessories/aoa2#hid-support>
Refs <https://www.usb.org/sites/default/files/hid1_11.pdf>

PR #2632 <#2632>

Signed-off-by: Romain Vimont <[email protected]>

BUILD.md

@@ -14,7 +14,8 @@ First, you need to install the required packages:
 # for Debian/Ubuntu
 sudo apt install ffmpeg libsdl2-2.0-0 adb wget \
                  gcc git pkg-config meson ninja-build libsdl2-dev \
-                 libavcodec-dev libavdevice-dev libavformat-dev libavutil-dev
+                 libavcodec-dev libavdevice-dev libavformat-dev libavutil-dev \
+                 libusb-1.0-0 libusb-dev
 ```
 
 Then clone the repo and execute the installation script
@@ -88,11 +89,12 @@ Install the required packages from your package manager.
 
 ```bash
 # runtime dependencies
-sudo apt install ffmpeg libsdl2-2.0-0 adb
+sudo apt install ffmpeg libsdl2-2.0-0 adb libusb-1.0-0
 
 # client build dependencies
 sudo apt install gcc git pkg-config meson ninja-build libsdl2-dev \
-                 libavcodec-dev libavdevice-dev libavformat-dev libavutil-dev
+                 libavcodec-dev libavdevice-dev libavformat-dev libavutil-dev \
+                 libusb-dev
 
 # server build dependencies
 sudo apt install openjdk-11-jdk
@@ -114,7 +116,7 @@ pip3 install meson
 sudo dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm
 
 # client build dependencies
-sudo dnf install SDL2-devel ffms2-devel meson gcc make
+sudo dnf install SDL2-devel ffms2-devel libusb-devel meson gcc make
 
 # server build dependencies
 sudo dnf install java-devel

README.md

@@ -673,6 +673,39 @@ content (if supported by the app) relative to the center of the screen.
 Concretely, scrcpy generates additional touch events from a "virtual finger" at
 a location inverted through the center of the screen.
 
+#### Physical keyboard simulation (HID)
+
+By default, scrcpy uses Android key or text injection: it works everywhere, but
+is limited to ASCII.
+
+On Linux, scrcpy can simulate a USB physical keyboard on Android to provide a
+better input experience (using [USB HID over AOAv2][hid-aoav2]): the virtual
+keyboard is disabled and it works for all characters and IME.
+
+[hid-aoav2]: https://source.android.com/devices/accessories/aoa2#hid-support
+
+However, it only works if the device is connected by USB, and is currently only
+supported on Linux.
+
+To enable this mode:
+
+```bash
+scrcpy --hid-keyboard
+scrcpy -K  # short version
+```
+
+If it fails for some reason (for example because the device is not connected via
+USB), it automatically fallbacks to the default mode (with a log in the
+console). This allows to use the same command line options when connected over
+USB and TCP/IP.
+
+In this mode, raw key events (scancodes) are sent to the device, independently
+of the host key mapping. Therefore, if your keyboard layout does not match, it
+must be configured on the Android device, in Settings → System → Languages and
+input → [Physical keyboard].
+
+[Physical keyboard]: https://github.com/Genymobile/scrcpy/pull/2632#issuecomment-923756915
+
 
 #### Text injection preference
 
@@ -692,6 +725,9 @@ scrcpy --prefer-text
 
 (but this will break keyboard behavior in games)
 
+This option has no effect on HID keyboard (all key events are sent as
+scancodes in this mode).
+
 [textevents]: https://blog.rom1v.com/2018/03/introducing-scrcpy/#handle-text-input
 [prefertext]: https://github.com/Genymobile/scrcpy/issues/650#issuecomment-512945343
 
@@ -707,6 +743,9 @@ To avoid forwarding repeated key events:
 scrcpy --no-key-repeat
 ```
 
+This option has no effect on HID keyboard (key repeat is handled by Android
+directly in this mode).
+
 
 #### Right-click and middle-click
 

app/meson.build

@@ -42,6 +42,14 @@ if v4l2_support
     src += [ 'src/v4l2_sink.c' ]
 endif
 
+aoa_hid_support = host_machine.system() == 'linux'
+if aoa_hid_support
+    src += [
+        'src/aoa_hid.c',
+        'src/hid_keyboard.c',
+    ]
+endif
+
 check_functions = [
     'strdup'
 ]
@@ -62,8 +70,11 @@ if not get_option('crossbuild_windows')
         dependencies += dependency('libavdevice')
     endif
 
-else
+    if aoa_hid_support
+        dependencies += dependency('libusb-1.0')
+    endif
 
+else
     # cross-compile mingw32 build (from Linux to Windows)
     prebuilt_sdl2 = meson.get_cross_property('prebuilt_sdl2')
     sdl2_bin_dir = meson.current_source_dir() + '/../prebuilt-deps/' + prebuilt_sdl2 + '/bin'
@@ -140,6 +151,9 @@ conf.set('SERVER_DEBUGGER_METHOD_NEW', get_option('server_debugger_method') == '
 # enable V4L2 support (linux only)
 conf.set('HAVE_V4L2', v4l2_support)
 
+# enable HID over AOA support (linux only)
+conf.set('HAVE_AOA_HID', aoa_hid_support)
+
 configure_file(configuration: conf, output: 'config.h')
 
 src_dir = include_directories('src')

app/scrcpy.1

@@ -82,6 +82,14 @@ Start in fullscreen.
 .B \-h, \-\-help
 Print this help.
 
+.TP
+.B \-K, \-\-hid\-keyboard
+Simulate a physical keyboard by using HID over AOAv2.
+
+This provides a better experience for IME users, and allows to generate non-ASCII characters, contrary to the default injection method.
+
+It may only work over USB, and is currently only supported on Linux.
+
 .TP
 .B \-\-legacy\-paste
 Inject computer clipboard text as a sequence of key events on Ctrl+v (like MOD+Shift+v).