Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken when piped to. #502

Open
Granitosaurus opened this issue Apr 16, 2017 · 11 comments
Open

Broken when piped to. #502

Granitosaurus opened this issue Apr 16, 2017 · 11 comments

Comments

@Granitosaurus
Copy link

When piping to python script that uses prompt() everything breaks:

File "/home/user/lib/python3.6/site-packages/prompt_toolkit/input.py", line 67, in __init__
    assert self.stdin.isatty()
AssertionError

To reproduce:

#test.py
from prompt_toolkit import prompt
text = prompt('> ')

Then:

$ echo "foo" | python test.py
<...>
File "/home/user/lib/python3.6/site-packages/prompt_toolkit/input.py", line 67, in __init__
    assert self.stdin.isatty()
AssertionError
@jonathanslenders
Copy link
Member

Hi @Granitosaurus,

Thanks for reporting the issue. This is however something which won't be fixed. (I guess I definitely need to given better error message as feedback.)

The reason is that prompt_toolkit is meant for user interaction, not for machine-interaction like a pipe. If you want to support pipe input on the other hand, it's best to test for sys.stdin.isatty() yourself, and if that is true, then use sys.stdin.read/readline to retrieve the input.

The same is true for piping the output to something else. If you'd pipe stdout to a file, the result will be a meaningless sequence of ANSI escape characters. If sys.stdout.isatty() is true, don't use prompt_toolkit.

@Granitosaurus
Copy link
Author

@jonathanslenders is there a technical reason behind this? There seems to be a bit of weirdness going on with stdin/out when piping is involved.
I think piping doesn't mean machine-interaction - it's a very common way to pass arguments to an application.
The application I'm working on currently takes in a file from a pipe or an argument and uses user input to analyze the file's content. So piping in this case very much makes sense.

@jonathanslenders
Copy link
Member

Yes, a Posix pipe is not the same as a PTY (pseudo terminal). Both are I/O devices, but they have different properties.

  • For instance, for a pipe between two processes, you want to have a big buffer, because otherwise, you end up switching between these process all the time. For user interaction, you want immediate feedback, so no buffering at all.
  • Further, prompt_toolkit will discard the input I/O when too much was read from stdin and the input gets accepted (enter is pressed). You will loose this input data if you would pipe certain input. For user interaction, this doesn't matter, because after pressing "enter", the user waits to see the feedback. (The reason this works that way, is because it's the only way to make processing pasted text on stdin efficient. - There is no way to read until a "newline" character, and there is also no way to push stdin, back into the input device. And reading one character at a time during paste events is slow.)
  • A pipe does not respond to a CPR (cursor position request). There is actually no cursor.
  • A pipe does not have raw or canonical input mode.

Probably there are other differences. There is a lot of non-obvious low level I/O underneath. But in any case, prompt_toolkit won't support pipes as stdin. Maybe with some hacks it will work partly, but seriously, it'll take only a few lines of code to read stdin yourself, if the input doesn't appear to be a tty.

if not sys.stdin.isatty():
    data = sys.stdin.read()
    ...

@chrisgoddard
Copy link

chrisgoddard commented Feb 23, 2019

Apologies for replying on an old thread, but I spent the better part of the day figuring out a workaround for this. And when I say workaround - I definitely mean hack.

If you want to start a command by piping in input and then have an interactive prompt appear, e.g.

'''data-producer | data-processor''' - where data-processor would then have an interactive prompt (I'm actually using PyInquirer to create a few checkbox interfaces, and it uses prompt-toolkit)

I found that if you take the data from stdin before the creation of the prompt-toolkit application, you can do the following:

data = sys.stdin.read()
sys.stdin = sys.stdout

app = Application()
...

Now I'm sure this would create other issues for other applications and might not even work in all environments, but it worked for my purposes.

I had tried setting sys.stdin = open('dev/tty') but that caused a weird bad file descriptor error.

Totally understand this isn't the intended use, but there are a few folks that seem to be trying to do something similar so if it's helpful, great.

Also feel free to tell me why this is a terrible idea. I'm sure there's a reason why it hasn't been suggested.

@ciphergoth
Copy link

Adding my "me too" to this. It's sometimes very useful to be able to write a script which uses stdout/stderr for data, but also presents a user interface, so it would be great if there were a flag which caused prompt-toolkit to use /dev/tty for the UI. This is similar to the way eg "whiptail" produces a result on "stdout".

@jonathanslenders
Copy link
Member

Hi @ciphergoth,

That's interesting. I didn't knew that "whiptail" was doing that.

The way it works is as follows. Normally /dev/stdin is the input device and /dev/stdout is the output device. The file descriptors correspond to 0 and 1 respectively. However, when a program is attached to a TTY, /dev/stdin and /dev/stdout will refer to the same device - the slave side of the TTY - even though the file descriptors remain 0 and 1. This means, we can actually also open FD 1 for reading or FD 0 for writing.

It is what every pager does. I discovered this actually when writing pypager, a $PAGER implementation using prompt_toolkit. The relevant line is the following where we pass sys.stdout to the create_input function: https://github.com/prompt-toolkit/pypager/blob/master/pypager/pager.py#L121

So, here's a question to all of you:

  • Should prompt_toolkit by default use the /dev/stdout device for reading input key strokes if it appears that /dev/stdin is not a tty?

I know some people try to pipe data into the stdin, and they expect prompt_toolkit to handle that (even though it's not always working perfect). So, I'm not sure what's the best default.

@ciphergoth
Copy link

I am not sure that this should be the only option, but there should be a documented way to get this behavior. As people have said, attempts to do this by hand result in a bad file descriptor error.

My application is that I want a script that helps me set up environment variables, so I have

function set_vars { source <( ./ask_for_vars ) }

and this doesn't work because stdout is redirected.

@jonathanslenders
Copy link
Member

@ciphergoth,
So, that's the opposite. Here we send the output in a pipe, rather than receiving the input from a pipe.
In this case, we would have to use /dev/stderr as the output device.
This PR should fix the default output: https://github.com/prompt-toolkit/python-prompt-toolkit/pull/899/files

I'm tempted to say that it's a reasonable choice to switch the output to /dev/stderr automatically if stdout is not a tty, but stderr is. The output that prompt_toolkit generates is not really meant for consumption by anything other then a terminal emulator.

@asmeurer: What are your opinions on this matter? You have been scripting prompt_toolkit with pexect, and trying to pipe data into a prompt_toolkit application.

@anki-code
Copy link

anki-code commented Apr 17, 2024

it would be great if there were a flag which caused prompt-toolkit to use /dev/tty for the UI

Just want to add another modern real life example - fzf. As we know:

Normally programs read and write stdin/stdout to communicate with the user, but fzf uses a clever trick: It opens /dev/tty which is always the terminal (and not stdin/stdout which might not be seen by the user, as is here the case). This way fzf can be used in pipes e.g. find . -name '*.mp3' | fzf | xargs vlc.

@anki-code
Copy link

JFYI I got answer from fzf owner:

fzf:
* Input list: Read from STDIN
* User input: Read from /dev/tty
* UI: Print to STDERR
* Selected item: Print to STDOUT

@ErikBjare
Copy link

ErikBjare commented Dec 16, 2024

Came to this issue having the same issue as #1943

I used to have code like:

# if stdin is not a tty, we might be getting piped input, which we should include in the prompt
was_piped = False
if not sys.stdin.isatty():
    # fetch prompt from stdin
    prompt_stdin = _read_stdin()
    if prompt_stdin:
        initial_msgs += [Message("system", f"```stdin\n{prompt_stdin}\n```")]
        was_piped = True

        # Attempt to switch to interactive mode
        sys.stdin.close()
        try:
            sys.stdin = open("/dev/tty")
        except OSError:
            # if we can't open /dev/tty, we're probably in a CI environment, so we should just continue
            logger.warning(
                "Failed to switch to interactive mode, continuing in non-interactive mode"
            )

But after switching from rich+readline to prompt-toolkit, I got the same issue as #1943.

The workaround by @chrisgoddard (#502 (comment)) seems to have worked: ErikBjare/gptme@b3974c1

Would appreciate knowledge about any known footguns with this workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants