readLine() ignores any text in the buffer when OEF is reached

[bdw429s]

CommandBox allows you to pipe a file of commands into it and readLine() will read the lines one at a time off the standard input just as though a human had typed them.

box.exe < commands.txt

If there's no trailing line break at the end of the file, then the last command is read in but the OEF fires before the last line is handed back for execution. I get that this makes some deal of sense since I can press Ctrl-D while typing at the shell to exit immediately, but in the case of reading an actual text file input, I'd like to be able to execute that last line even if there's no line break at the end.

The org.jline.reader.UserInterruptException exception provides getPartialLine() to see if the user typed something. Is the correct fix to add the partial line to the org.jline.reader.EndOfFileException as well?

[gnodet]

You're right. When the terminal is not an interactive terminal (with a human being), it does not make sense to try to read Ctrl-D in advance. I'll try to investigate.

[gnodet]

On unix systems, the TerminalBuilder creates a DumbTerminal when the input is not a real terminal, such as a file redirect. I think the same should be true on windows too. The DumbTerminal does not read the stream in advance, so the problem does not appear on unix.
In theory, the input and output should be better decorrelated so that the output could be cleanly usable even if there's no input (or the opposite), but that's really a corner case imho.

[bdw429s]

Just a note on the dumb terminal idea-- just because I've piped input to the process doesn't mean that I'm not running it in a terminal capable of ANSI formatting, etc. In fact, in my cases I likely would be which I reported as another issue here; #299

[gnodet]

Yes, this is the same issue. But not a simple bug to fix, it may require some important changes in order to split the input from the output.

[bdw429s]

@gnodet Any progress on this? What if, when the OEF is reached, if there is input that's been read into the buffer, the line is read (returned from readLine() for processing) and then any subsequent call of readLine() will fire OEF now that the input is truly empty.

[bdw429s]

@mattirn I see this ticket has been closed. Can you please elaborate on what change you made or if I need to modify any code on my end to take advantage of it?

[mattirn]

You have to catch EndOfFileException, get exception property partialLine, execute it and exit from REPL loop:

jline3/demo/src/main/java/org/jline/demo/Repl.java

Lines 341 to 351 in fd11775

	catch (EndOfFileException e) {
	String pl = e.getPartialLine();
	if (pl != null) { // execute last line from redirected file (required for Windows)
	try {
	consoleEngine.println(systemRegistry.execute(pl));
	} catch (Exception e2) {
	systemRegistry.trace(e2);
	}
	}
	break;
	}

[bdw429s]

@mattirn Thanks, I'll make these changes in my console app.

[bdw429s]

@mattirn @gnodet Oops, it seems something has gotten broken since I originally put this ticket in. This simple example of piping commands to a text file into my console app no longer works:

box.exe < commands.txt

On Windows, using the JansiWinSysTerminal, I get the following error that appears in the console:

Dec 11, 2020 2:24:28 PM org.jline.utils.Log logr
WARNING: Error in WindowsStreamPump
java.io.IOException: ReadConsoleInputW failed
        at org.fusesource.jansi.internal.Kernel32.readConsoleInputHelper(Kernel32.java:816)
        at org.jline.terminal.impl.jansi.win.JansiWinSysTerminal.processConsoleInput(JansiWinSysTerminal.java:134)
        at org.jline.terminal.impl.AbstractWindowsTerminal.pump(AbstractWindowsTerminal.java:460)
        at java.base/java.lang.Thread.run(Unknown Source)

The EndOfFileException is thrown right away on the first readLine() call (even if I have more than one line of text in the file) and the getPartialLine() method returns an empty string.

I am not able to confirm this ticket is fixed.

[bdw429s]

I attempted another approach to this, with not piping input to the process, but instead typing some text into the buffer and then pressing Ctrl-D. Interestingly enough, when I press Ctrl-D, instead of throwing the EndOfFileException it behaves as though I've pressed Tab and displays completion results.

I'm not sure if this is an inteded behavior or not. I can confirm that when I press Ctrl-D without any text in the buffer, it throws the EndOfFileException and I exit my shell.

[mattirn]

Also Ubuntu/xterm zsh will complete when you press Ctrl-D if the buffer is not empty.

The ticket I have tested only using Windows/CMD and it does work also with JLine version 3.18.0.

[gnodet]

Ctrl-D is mapped to the DELETE_CHAR_OR_LIST widget (see https://github.com/jline/jline3/blob/master/reader/src/main/java/org/jline/reader/impl/LineReaderImpl.java#L6014) which is mapped to the deleteCharOrList method.
The Ctrl-D from the terminal is usually mapped to the VEOF control char on the terminal, and that's what causes the EndOfFileException, see

jline3/reader/src/main/java/org/jline/reader/impl/LineReaderImpl.java

Lines 653 to 655 in 29d72f8

	if (buf.length() == 0 && getLastBinding().charAt(0) == originalAttributes.getControlChar(ControlChar.VEOF)) {
	throw new EndOfFileException();
	}

.

[bdw429s]

Also Ubuntu/xterm zsh will complete when you press Ctrl-D if the buffer is not empty.

Interesting, that doesn't make any sense to me, but no matter. I've admittedly never tried it before so let's just ignore that behavior and focus on the error above that's preventing me from using the changes in this ticket.

The ticket I have tested only using Windows/CMD and it does work also with JLine version 3.18.0.

Ok, so "works on your machine" but let's figure out why it's not working for me. I've pasted the exception I received above. Can you comment on what that exception might mean or why I'd be getting it?

Dec 11, 2020 2:24:28 PM org.jline.utils.Log logr
WARNING: Error in WindowsStreamPump
java.io.IOException: ReadConsoleInputW failed
        at org.fusesource.jansi.internal.Kernel32.readConsoleInputHelper(Kernel32.java:816)
        at org.jline.terminal.impl.jansi.win.JansiWinSysTerminal.processConsoleInput(JansiWinSysTerminal.java:134)
        at org.jline.terminal.impl.AbstractWindowsTerminal.pump(AbstractWindowsTerminal.java:460)
        at java.base/java.lang.Thread.run(Unknown Source)

[gnodet]

@bdw429s hard to tell the reason as the underlying problem is hidden. I've raised an issue in jansi to report the real problem. That should help understanding the issue.

[bdw429s]

@mattirn Thanks for spending some time on this. Is the commit above meant to be a fix, or simply some additional debugging code? May I recommend adding a comment to tickets when adding commits or closing them? I see a lot of mystery commits on tickets in this repo where I'm not sure what the current status of the ticket really is or what has been done against it. :)

[mattirn]

Fixed and tested using ConEmu with Jansi and JNA.

[bdw429s]

@mattirn Excellent, and thanks for the update. I'll give this a test soon on my end.