gh-102130: Support tab completion in cmd for Libedit.

[Constantin1489]

After I dug the rabbit hole, I wanted to share something and commit it to CPython.

My commit allows tab completion in cmd for Libedit. (EDIT AFTER MERGE: this allows tab completion for pdb with libedit)

tab of readline.parse_and_bind("tab: complete") is a symbolic character name. Libedit doesn't support a number of symbolic character names(like TAB, ESC, DEL).

FYI, for better compatibility, Python Documents should suggest using backslashed escape sequences(e.g. \t, \a) or the ASCII character corresponding to the octal number(or octal excape sequences).(e.g. \011), It's because the only thing GNU readline(man bash and /Readline Key Bindings to search the part.) and Libedit(man editrc) have in common.

Also, unlike GNU emacs software, GNU readline doesn't support control characters of the form '^character' (e.g. '^I' it's a tab).
Both GNU readline(man bash and /Readline Key Bindings to search the part.) and Libedit(man editrc) support the following backslashed escape sequences and octal escape sequences.

\a     bell
\b     backspace
\f     form feed
\n     newline
\r     carriage return
\t     horizontal tab
\v     vertical tab
\nnn   octal escape sequences

GNU readline's predefined commands are in the '/Readline Command Names' of man bash.
Libedit's predefined commands are in the section "EDITOR COMMANDS" of man editrc. But there are other undocumented commands in its source code.
Or you can get a list by import readline; readline.parse_and_bind('bind -l') (Libedit only. This shows undocumented commands too).

• Issue: cmd.py command completion does not work with libedit #102130

---

📚 Documentation preview 📚: https://cpython-previews--107748.org.readthedocs.build/

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[Constantin1489]

There is something that Python core developers decide.

1. Supporting "TAB" symbolic character name for libedit in "cmd.py".

readline_doc = getattr(readline, '__doc__', '')
if readline_doc is not None and 'libedit' in readline_doc: 
    if self.completekey is 'tab':
        self.completekey = r'"\t"'

OR just

if self.completekey is 'tab':
    self.completekey = r'"\t"'

If this is supported, then other third-party libraries can support Libedit tab completion without changing their code.
I don't want this support because it will give a false impression of common key sequences and the requirement to support other symbolic character names.

2. Removing editrc examples in readline.rst.

Because of the absence of the GNU inputrc example in readline.rst, the editrc example may be just for hotfix to do a tab complete in cmd or pdb.

For Vim keybinding in the editrc example(python:bind -v), I think it's valid. For this reason, Vim keybinding for GNU inputrc example is needed.

3. Changing keybindings in related test files.
   There are ^I and tab in related test files. If you need me to change them, I will do it.

Just so you know, if you want to debug this problem, please check what I found below.

Use one of these lines to enable tab completion in Python using ~/.editrc for libedit.

python:bind "\t" rl_complete
python:bind "\011" rl_complete
python:bind \\t rl_complete
python:bind "	" rl_complete
# "	" is a tab.

To use them withreadline.parse_and_bind() in Python, remove 'python:' prefix. To make clear, If you did import readline; readline.parse_and_bind('bind "\t" rl_complete') in pdb interactively (e.g., $python -m pdb somefile.py), it will autocomplete file names of the current directory. It's normal because the binding statement wasn't called in the pdb library.

The key sequences("\t", "\011") of the first two lines in the code block above are documented in man editrc. The first one is called a backslashed escape sequence. The second one is called the ASCII character corresponding to the octal number(or octal escape character).

But the last two lines are not. I stumble across them.

To do another complete command in Libedit, this is possible.
import readline; readline.parse_and_bind('bind -s "\011" ee')
In this case, If you press a tab, 'ee' will be typed.
Other possible options are appeared in man editrc.

I found several ways to debug it. (these are examples.)

• In Python interpreter, import readline; readline.parse_and_bind('bind -s "\011" ee')
• Replace 'tab' with '' in the pdb.py and cmd.py of your python directory in $PATH. e.g. /usr/local/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/pdb.py.

How to check the keybinding setting applied with Libedit.

import readline
readline.parse_and_bind('bind')

This shows the user's current applied libedit readline setting.

[gaogaotiantian]

I don't think we should change the default and hope the completekey passed in is correct. It could break users code, for example, if they compare self.completekey == tab. Also, if the user subclassed the cmd.Cmd class and did something pdb does like

class MyCmd(cmd.Cmd):
    def __init__(self, completekey='tab'):
        cmd.Cmd.__init__(self, completekey)

They will still have the issue.

We should convert the completekey internally and just do a different binding command with libedit. Minimal code changes and fix things for good.

Also, why do you want to change the site.py file? It works for now I believe. Python REPL has the complete feature with libedit.

I believe the right way to do this is just add an extra check in cmd.Cmd for libedit, and do the correct thing (convert tab to "\t" or the equivalent in libedit, then use a different binding command).

Also, if you want to finish this, you'd need a regression test on at least cmd.Cmd, it would be easier after #110945 is merged so you'll have a helper function for pty.

I can help you with the review and find the core-dev for approval if you want to work on this.

[Constantin1489]

Yes, what you pointed is valid.
So you want Python should change the tab internally for portability.

Minimal code changes and fix things for good.

Because I was unsure about the decision I made. This is what I asked in number 1 #107748 (comment).
In that case, a user who uses libedit with your custom object will experience when press 't' will be ignored sometimes.

At the time I wrote this PR, I thought, that if a user did something wrong, the wrong result must be thrown. And the docs must give a pledge. (You know when learning the linux thing there must be some reason I don't know and some result is unintuitive or counterintuitive for newcomer.).

Thank you for giving me direction for what I consider. I will do that within a week as soon as possible.

The second thing you mention(site.py) is just for a tone and manner thing(second and third thing that I asked in #107748 (comment)). I will revert it.

Thank you for helping me.

[gaogaotiantian]

I believe you only need to change cmd.py, which would make this PR compact and easy to review. I'd recommend you to wait for #110945 so you'll have a test framework for completion. Actually enabling the test in that PR on libedit would just do.

BTW, you should not use is to compare strings. In this very specific case it actually works because of the python internal cache for intern strings, but that's definitely not what you should rely on. Always use == to check the string equality.

[gaogaotiantian]

I have the hardest time believing that commit can cause the completion issue. The code should not be executed at all for completion. Are you sure you did the test correctly?

[Constantin1489]

I'm sorry. You are right. Maybe my environment goes something wrong.

[gaogaotiantian]

Hi @Constantin1489 , now that #111826 is merged, you have the utility to write tests for the feature. Could you:

• Add a test in test_cmd.py to do some basic testing for command completion? It does not need to be super fancy, refer to gh-110944: Move pty helper to test.support and add basic pdb completion test #111826 for the usage of run_pty().
• Remove the check for libedit in gh-110944: Move pty helper to test.support and add basic pdb completion test #111826 so it would be another confirmation that the feature works with libedit? Do not remove the import_module piece because that's the magic to skip the test if readline is not available.

[encukou]

Alright!
All that's left is very small clean-ups. Do you want to do them? If not, just let me know, I'll update & merge and you can move to the next thing :)

• Merge in the branch (using the GitHub Update branch button), to get the new readline.backend attribute
  • Use readline.backend rather than getattr(readline, '__doc__', '')
  • Make sure docs build without the warning about readline.backend
• Add a blank line after the added documentation paragraph (I think I added one in my Suggestions, but GitHub ate it!)

[encukou]

And one more thing I always forget to point out (sorry for that)!

At the end of the Cmd class documentation (before .. _cmd-objects:), add a versionchanged note like

   .. versionchanged:: 3.11
   ``completekey='tab'`` is replaced by ``'^I'`` for ``editline``.

Also, please ignore the free-threaded CI failures; they're not your fault and they're allowed to fail for now.

[encukou]

change the NEWS text, the commit names don't matter

On December 1, 2023 5:02:04 PM GMT+01:00, Constantin Hong ***@***.***> wrote: @Constantin1489 commented on this pull request. > @@ -0,0 +1 @@ +Support tab completion in :mod:`cmd` for Libedit. Should I change Libedit to editline of this and this commit post's name? ``` gh-102130: Support tab completion in cmd for Libedit. -> gh-102130: Support tab completion in cmd for editline. ``` -- Reply to this email directly or view it on GitHub: #107748 (review) You are receiving this because you were mentioned. Message ID: ***@***.***>

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit 652100f 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[encukou]

Thank you for your contribution!