Extended & human-friendly keys

See the changelog additions for user-visible changes.

Since we enable/disable terminal protocols whenever we pass terminal ownership,
tests can no longer run in parallel on the same terminal.

For the same reason, readline shortcuts in the gdb REPL will not work anymore.
As a remedy, use gdbserver, or lobby for CSI u support in libreadline.

Add sleep to some tests, otherwise they fall (both in CI and locally).

There are two weird failures on FreeBSD remaining, disable them for now
https://github.com/fish-shell/fish-shell/pull/10359/checks?check_run_id=23330096362

Design and implementation borrows heavily from Kakoune.

In future, we should try to implement more of the kitty progressive
enhancements.

Closes fish-shell#10359

CHANGELOG.rst

@@ -12,7 +12,7 @@ fish 3.8.0 (released ???)
    10198 10200 10201 10204 10210 10214 10219 10223 10227 10232 10235 10237 10243 10244 10245
    10246 10251 10260 10267 10281 10347 10366 10368 10370 10371 10263 10270 10272 10276 10277
    10278 10279 10291 10293 10305 10306 10309 10316 10317 10327 10328 10329 10330 10336 10340
-   10345 10346 10353 10354 10356 10372 10373 3299 10360
+   10345 10346 10353 10354 10356 10372 10373 3299 10360 10359
 
 The entirety of fish's C++ code has been ported to Rust (:issue:`9512`).
 This means a large change in dependencies and how to build fish.
@@ -21,6 +21,24 @@ Packagers should see the :ref:`For Distributors <rust-packaging>` section at the
 Notable backwards-incompatible changes
 --------------------------------------
 
+- Fish now decodes keyboard input into human-readable key names.
+  To make this for for a wide range of terminals, fish asks terminals to speak several keyboard protocols,
+  including CSI u, XTerm's ``modifyOtherKeys`` and some progressive enhancements from the [kitty keyboard protocol](https://sw.kovidgoyal.net/kitty/keyboard-protocol/).
+  Depending on terminal support, this allows to bind a lot more key combinations,
+  including arbitrary combinations of modifiers ``ctrl``, ``alt`` and ``shift``.
+
+  This comes with a new syntax for specifying keys to builtin ``bind``.
+  The new syntax introduces modifier names and names for some keys that don't have an obvious and printable Unicode code point.
+  The old syntax remains mostly supported but the new one is preferred.
+  - Existing bindings that use the new names have a different meaning now.
+    For example
+    - ``bind up 'do something'`` binds the up arrow key instead of a two-key sequence.
+    - ``bind ctrl-x,alt-c 'do something'`` binds a sequence of two keys.
+    Since ``,`` and ``-`` act as separators, there are some cases where they need to be written as ``comma`` and ``minus`` respectively.
+  - To minimize gratuitous breakage, the key argument to ``bind`` is parsed using the old syntax in two cases:
+    - If key starts with a raw escape character (``\e``) or a raw ASCII control character (``\c``).
+    - If key consists of exactly two characters, contains none of ``,`` or ``-`` and is not a named key.
+
 - ``random`` now uses a different random number generator and so the values you get even with the same seed have changed.
   Notably, it will now work much more sensibly with very small seeds.
   The seed was never guaranteed to give the same result across systems,
@@ -39,7 +57,7 @@ Notable backwards-incompatible changes
 Notable improvements and fixes
 ------------------------------
 - New function ``fish_should_add_to_history`` can be overridden to decide whether a command should be added to the history (:issue:`10302`).
-- :kbd:`Control-C` during command input no longer prints ``^C`` and a new prompt but merely clears the command line. This restores the behavior from version 2.2. To revert to the old behavior use ``bind \cc __fish_cancel_commandline`` (:issue:`10213`).
+- :kbd:`Control-C` during command input no longer prints ``^C`` and a new prompt but merely clears the command line. This restores the behavior from version 2.2. To revert to the old behavior use ``bind ctrl-c __fish_cancel_commandline`` (:issue:`10213`).
 - The :kbd:`Control-R` history search now uses glob syntax (:issue:`10131`).
 - The :kbd:`Control-R` history search now operates only on the line at cursor, making it easier to quickly compose a multi-line commandline by recalling previous commands.
 
@@ -48,6 +66,12 @@ Deprecations and removed features
 
 - ``commandline --tokenize`` (short option ``-o``) has been deprecated in favor of ``commandline --tokens-expanded`` (short option ``-x``) which expands variables and other shell expressions, removing the need to use "eval" in custom completions (:issue:`10212`).
 - A new feature flag, ``remove-percent-self`` (see ``status features``) disables PID expansion of ``%self`` which has been supplanted by ``$fish_pid`` (:issue:`10262`).
+- Specifying key names as terminfo name (``bind -k``) is deprecated and may be removed in a future version.
+- Flow control -- which if enabled by ``stty ixon ixoff`` allows to pause terminal input with ``ctrl-s`` and resume it with ``ctrl-q`` -- now works only while fish is executing an external command.
+- When a terminal pastes text into fish using bracketed paste, fish used to switch to a special ``paste`` bind mode.
+  This bind mode has been removed. The behavior on paste is currently not meant to be configurable.
+- When fish is stopped or terminated by a signal that cannot be caught (SIGSTOP or SIGKILL), it may leave the terminal in a state where keypresses with modifiers are sent as CSI u sequences instead of traditional control characters or escape sequecnes (that are recognized by bash/readline). If this happens, you can use the ``reset`` command from ``ncurses`` to restore the terminal state.
+- ``fish_key_reader --verbose`` is now ignored, so it no longer shows raw byte values or timing information. Since fish now decodes keys, this should no longer be necessary.
 
 Scripting improvements
 ----------------------
@@ -78,13 +102,14 @@ Interactive improvements
 
 New or improved bindings
 ^^^^^^^^^^^^^^^^^^^^^^^^
-- Bindings can now mix special input functions and shell commands, so ``bind \cg expand-abbr "commandline -i \n"`` works as expected (:issue:`8186`).
+- Bindings can now mix special input functions and shell commands, so ``bind ctrl-g expand-abbr "commandline -i \n"`` works as expected (:issue:`8186`).
 - When the cursor is on a command that resolves to an executable script, :kbd:`Alt-O` will now open that script in your editor (:issue:`10266`).
 - Two improvements to the :kbd:`Alt-E` binding which edits the commandline in an external editor:
   - The editor's cursor position is copied back to fish. This is currently supported for Vim and Kakoune.
   - Cursor position synchronization is only supported for a set of known editors. This has been extended by also resolving aliases. For example use ``complete --wraps my-vim vim`` to synchronize cursors when `EDITOR=my-vim`.
 - ``backward-kill-path-component`` and friends now treat ``#`` as part of a path component (:issue:`10271`).
 - The ``E`` binding in vi mode now correctly handles the last character of the word, by jumping to the next word (:issue:`9700`).
+- If the terminal supports shifted key codes from the [kitty keyboard protocol](https://sw.kovidgoyal.net/kitty/keyboard-protocol/), ``shift-enter`` now inserts a newline instead of executing the command line.
 - Vi mode has seen some improvements but continues to suffer from the lack of people working on it.
   - Insert-mode :kbd:`Control-N` accepts autosuggestions (:issue:`10339`).
   - Outside insert mode, the cursor will no longer be placed beyond the last character on the commandline.
@@ -103,6 +128,8 @@ Completions
 Improved terminal support
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 - Fish now sets the terminal window title unconditionally (:issue:`10037`).
+- Focus reporting is enabled unconditionally, not just inside tmux.
+  To use it, define functions that handle events ``fish_focus_in`` and ``fish_focus_out``.
 
 Other improvements
 ------------------

build_tools/pexpect_helper.py

@@ -34,7 +34,12 @@ def get_prompt_re(counter):
     """Return a regular expression for matching a with a given prompt counter."""
     return re.compile(
         r"""(?:\r\n?|^)   # beginning of line
-            (?:\x1b[\d\[KB(m]*)* # optional colors
+            (?:\x1b[\d[KB(m]*)* # optional colors
+            (?:\x1b[\?2004h) # Bracketed paste
+            (?:\x1b[>4;1m) # XTerm's modifyOtherKeys
+            (?:\x1b[>5u) # CSI u with kitty progressive enhancement
+            (?:\x1b=) # set application keypad mode, so the keypad keys send unique codes
+            (?:\x1b[\?1004h)? # enable focus notify
             (?:\[.\]\ )?  # optional vi mode prompt
          """
         + (r"prompt\ %d>" % counter)  # prompt with counter

cmake/Tests.cmake

@@ -1,13 +1,6 @@
 # This adds ctest support to the project
 enable_testing()
 
-# By default, ctest runs tests serially
-if(NOT CTEST_PARALLEL_LEVEL)
-  include(ProcessorCount)
-  ProcessorCount(CORES)
-  set(CTEST_PARALLEL_LEVEL ${CORES})
-endif()
-
 # Put in a tests folder to reduce the top level targets in IDEs.
 set(CMAKE_FOLDER tests)
 
@@ -24,16 +17,14 @@ set(SKIP_RETURN_CODE 125)
 #    running `make test` does not require any of the binaries to be built before testing.
 #  * The only way to have a test depend on a binary is to add a fake test with a name like
 #    "build_fish" that executes CMake recursively to build the `fish` target.
-#  * It is not possible to set top-level CTest options/settings such as CTEST_PARALLEL_LEVEL from
-#    within the CMake configuration file.
 #  * Circling back to the point about individual tests not being actual Makefile targets, CMake does
 #    not offer any way to execute a named test via the `make`/`ninja`/whatever interface; the only
 #    way to manually invoke test `foo` is to to manually run `ctest` and specify a regex matching
 #    `foo` as an argument, e.g. `ctest -R ^foo$`... which is really crazy.
 
 # The top-level test target is "fish_run_tests".
 add_custom_target(fish_run_tests
-  COMMAND env CTEST_PARALLEL_LEVEL=${CTEST_PARALLEL_LEVEL} FISH_FORCE_COLOR=1
+  COMMAND env FISH_FORCE_COLOR=1
           FISH_SOURCE_DIR=${CMAKE_SOURCE_DIR}
           ${CMAKE_CTEST_COMMAND} --force-new-ctest-process # --verbose
           --output-on-failure --progress

doc_src/cmds/bind.rst

@@ -7,49 +7,67 @@ Synopsis
 
 .. synopsis::
 
-    bind [(-M | --mode) MODE] [(-m | --sets-mode) NEW_MODE] [--preset | --user] [-s | --silent] [-k | --key] SEQUENCE COMMAND ...
-    bind [(-M | --mode) MODE] [-k | --key] [--preset] [--user] SEQUENCE
-    bind (-K | --key-names) [-a | --all] [--preset] [--user]
+    bind [(-M | --mode) MODE] [(-m | --sets-mode) NEW_MODE] [--preset | --user] [-s | --silent] KEYS COMMAND ...
+    bind [(-M | --mode) MODE] [--preset] [--user] [KEYS]
+    bind [-a | --all] [--preset] [--user]
     bind (-f | --function-names)
     bind (-L | --list-modes)
-    bind (-e | --erase) [(-M | --mode) MODE] [--preset] [--user] [-a | --all] | [-k | --key] SEQUENCE ...
+    bind (-e | --erase) [(-M | --mode) MODE] [--preset] [--user] [-a | --all] | KEYS ...
 
 Description
 -----------
 
-``bind`` manages bindings.
+``bind`` manages key bindings.
 
-It can add bindings if given a SEQUENCE of characters to bind to. These should be written as :ref:`fish escape sequences <escapes>`. The most important of these are ``\c`` for the control key, and ``\e`` for escape, and because of historical reasons also the Alt key (sometimes also called "Meta").
+If both ``KEYS`` and ``COMMAND`` are given, ``bind`` adds (or replaces) a binding in ``MODE``.
+If only ``KEYS`` is given, any existing binding in the given ``MODE`` will be printed.
 
-For example, :kbd:`Alt`\ +\ :kbd:`W` can be written as ``\ew``, and :kbd:`Control`\ +\ :kbd:`X` (^X) can be written as ``\cx``. Note that Alt-based key bindings are case sensitive and Control-based key bindings are not. This is a constraint of text-based terminals, not ``fish``.
+``KEYS`` is a comma-separated list of key names.
+Modifier keys can be specified by prefixing a key name with a combination of ``ctrl-``, ``alt-`` and ``shift-``.
+For example, :kbd:`Alt`\ +\ :kbd:`w` is written as ``alt-w``.
+Key names are case-sensitive; for example ``alt-W`` is the same as ``alt-shift-w``.
 
-The generic key binding that matches if no other binding does can be set by specifying a ``SEQUENCE`` of the empty string (``''``). For most key bindings, it makes sense to bind this to the ``self-insert`` function (i.e. ``bind '' self-insert``). This will insert any keystrokes not specifically bound to into the editor. Non-printable characters are ignored by the editor, so this will not result in control sequences being inserted.
+Some keys have names, usually because they don't have an obvious printable character representation.
+They are:
 
-If the ``-k`` switch is used, the name of a key (such as 'down', 'up' or 'backspace') is used instead of a sequence. The names used are the same as the corresponding curses variables, but without the 'key\_' prefix. (See ``terminfo(5)`` for more information, or use ``bind --key-names`` for a list of all available named keys). Normally this will print an error if the current ``$TERM`` entry doesn't have a given key, unless the ``-s`` switch is given.
+``plus`` (``+``),
+``minus`` (``-``),
+``comma`` (``,``),
+``backspace``,
+``delete``,
+``escape``,
+``enter``,
+the arrow keys ``up``, ``down``, ``left`` and ``right``,
+``pageup``,
+``pagedown``,
+``home``,
+``end``,
+``insert``,
+``tab``,
+``space`` and
+``F1`` through ``F12``.
 
-To find out what sequence a key combination sends, you can use :doc:`fish_key_reader <fish_key_reader>`.
+An empty value (``''``) for ``KEYS`` designates the generic binding. For most bind modes, it makes sense to bind this to the ``self-insert`` function (i.e. ``bind '' self-insert``). This will insert any keystrokes not specifically bound to into the editor. Non-printable characters are ignored by the editor, so this will not result in control sequences being inserted.
+
+To find the name of a key combination you can use :doc:`fish_key_reader <fish_key_reader>`.
 
 ``COMMAND`` can be any fish command, but it can also be one of a set of special input functions. These include functions for moving the cursor, operating on the kill-ring, performing tab completion, etc. Use ``bind --function-names`` or :ref:`see below <special-input-functions>` for a list of these input functions.
 
 .. note::
     If a script changes the commandline, it should finish by calling the ``repaint`` special input function.
 
-If no ``SEQUENCE`` is provided, all bindings (or just the bindings in the given ``MODE``) are printed. If ``SEQUENCE`` is provided but no ``COMMAND``, just the binding matching that sequence is printed.
+If no ``KEYS`` argument is provided, all bindings (in the given ``MODE``) are printed. If ``KEYS`` is provided but no ``COMMAND``, just the binding matching that sequence is printed.
 
 Key bindings may use "modes", which mimics vi's modal input behavior. The default mode is "default". Every key binding applies to a single mode; you can specify which one with ``-M MODE``. If the key binding should change the mode, you can specify the new mode with ``-m NEW_MODE``. The mode can be viewed and changed via the ``$fish_bind_mode`` variable. If you want to change the mode from inside a fish function, use ``set fish_bind_mode MODE``.
 
+To bind a sequence of keys, separate them with ``,``.
+
 To save custom key bindings, put the ``bind`` statements into :ref:`config.fish <configuration>`. Alternatively, fish also automatically executes a function called ``fish_user_key_bindings`` if it exists.
 
 Options
 -------
 The following options are available:
 
-**-k** or **--key**
-    Specify a key name, such as 'left' or 'backspace' instead of a character sequence
-
-**-K** or **--key-names**
-    Display a list of available key names. Specifying **-a** or **--all** includes keys that don't have a known mapping
-
 **-f** or **--function-names**
     Display a list of available input functions
 
@@ -69,7 +87,7 @@ The following options are available:
     Specifying **-a** or **--all** without **-M** or **--mode** erases all binds in all modes regardless of sequence.
 
 **-a** or **--all**
-    See **--erase** and **--key-names**
+    See **--erase**
 
 **--preset** and **--user**
     Specify if bind should operate on user or preset bindings.
@@ -349,7 +367,7 @@ Examples
 
 Exit the shell when :kbd:`Control`\ +\ :kbd:`D` is pressed::
 
-    bind \cd 'exit'
+    bind ctrl-d 'exit'
 
 Perform a history search when :kbd:`Page Up` is pressed::
 

doc_src/cmds/fish_key_reader.rst

@@ -15,16 +15,11 @@ Description
 
 :program:`fish_key_reader` is used to explain how you would bind a certain key sequence. By default, it prints the :doc:`bind <bind>` command for one key sequence read interactively over standard input.
 
-If the character sequence matches a special key name (see ``bind --key-names``),  both ``bind CHARS ...`` and ``bind -k KEYNAME ...`` usage will be shown. In verbose mode (enabled by passing ``--verbose``), additional details about the characters received, such as the delay between chars, are written to standard error.
-
 The following options are available:
 
 **-c** or **--continuous**
     Begins a session where multiple key sequences can be inspected. By default the program exits after capturing a single key sequence.
 
-**-V** or **--verbose**
-    Tells fish_key_reader to output timing information and explain the sequence in more detail.
-
 **-h** or **--help**
     Displays help about using this command.
 
@@ -34,8 +29,6 @@ The following options are available:
 Usage Notes
 -----------
 
-In verbose mode, the delay in milliseconds since the previous character was received is included in the diagnostic information written to standard error. This information may be useful to determine the optimal ``fish_escape_delay_ms`` setting or learn the amount of lag introduced by tools like ``ssh``, ``mosh`` or ``tmux``.
-
 ``fish_key_reader`` intentionally disables handling of many signals. To terminate ``fish_key_reader`` in ``--continuous`` mode do:
 
 - press :kbd:`Control`\ +\ :kbd:`C` twice, or
@@ -51,12 +44,4 @@ Example
    > fish_key_reader
    Press a key:
    # press up-arrow
-   bind \e\[A 'do something'
-
-   > fish_key_reader --verbose
-   Press a key:
-   # press alt+enter
-              hex:   1B  char: \e
-   (  0.027 ms)  hex:    D  char: \cM  (or \r)
-   bind \e\r 'do something'
-
+   bind up 'do something'