Pasting long lines does not wrap the text correctly

[hecerinc]

Versions

ConEmu build: 2021.9.12 x64
OS version: Windows 11 Enterprise 10.0.22000 Build 22000 x64
Used shell version (Far Manager, git-bash, cmd, powershell, cygwin, whatever): cmd

Problem description

When I try to paste a long string in the text won't wrap correctly.

Steps to reproduce

1. Open ConEmu
2. Paste a text that exceeds the width of window

Actual results

Text starts wrapping on same line

Expected results

Text starts wrapping on new line

Additional files

[Maximus5]

This is clink problem. Please report that to cmder issue tracker.
https://chrisant996.github.io/clink/
https://conemu.github.io/en/cmder.html

[chrisant996]

This does not appear to be a Clink problem:

ConEmu is not wrapping to the next line when Clink prints a character at the end of the line. It is a variation on the "width of lambda char in Chinese locales" problems.

I've also confirmed that the output from Clink is identical between computers where this occurs and computers where it does not. In another recent report in Cmder repo, logging was obtained that shows the output is identical between computers where this occurs and computers where it does not.

This was reported recently in another post in Cmder. But this does not appear to be a Clink issue.

[hecerinc]

Thanks for the context @chrisant996 !
@Maximus5 should we consider reopening then?

[chrisant996]

If there is someone willing to try a couple custom versions of Clink and report back on results, then it will be possible to determine more precisely what is going wrong and where.

I have been unable to reproduce the issue myself.

[hecerinc]

I'm willing to try it out :)

[chrisant996]

I'm willing to try it out :)

Awesome, thanks. First I'll code up a diagnostic test to get more detailed information about the issue. It could be a couple days, but I'll post it here when it's ready.

[chrisant996]

@Maximus5 real console shows the lambda character taking two cells. ConEmu uses only one cell to render it.

If the terminal is 80 cells wide, then Clink and real console end up believing 79 characters fit on the line.

The Readline library inside Clink prints the lambda, a space, and 77 other characters.

That should wrap the cursor to the next line. In standalone Cmd.exe it does. In ConEmu it does not.

Readline then prints an extra space to ensure wrapping. But even then the line does not wrap.

When ConEmu renders differently than real console, that creates a discrepancy between what the OS console APIs report versus how ConEmu renders the text. Programs can't realistically overcome the discrepancies on their own -- they rely on the OS APIs return results being accurate.

[chrisant996]

@Maximus5 here is a log excerpt from Clink, showing what GetConsoleScreenBufferInfo reports:

23e4 rl_module::on_input      2457 INPUT "a", 1
23e4 LOGCURSORPOS              369 CURSORPOS 116,6
23e4 puts_face_func            754 PUTSFACE "�[ma�[m", 7
23e4 rl_module::on_input      2457 INPUT "a", 1
23e4 LOGCURSORPOS              369 CURSORPOS 117,6
23e4 puts_face_func            754 PUTSFACE "�[ma�[m", 7
23e4 LOGCURSORPOS              369 CURSORPOS 118,6
23e4 terminal_log_write       1925 RL_OUTSTREAM " ", 1
23e4 LOGCURSORPOS              369 CURSORPOS 118,6
23e4 terminal_log_write       1925 RL_OUTSTREAM "
", 1
23e4 rl_module::on_input      2457 INPUT "a", 1
23e4 LOGCURSORPOS              369 CURSORPOS 0,6
23e4 puts_face_func            754 PUTSFACE "�[ma�[m", 7

You can see that the second to last a input prints an a and the cursor moves from col 117 row 6 to col 118 row 6. The last a on the line prints an a and the cursor does not move. That is the point at which things get messed up. The character width math in Real console and Clink both expect the cursor to wrap at that point, but it does not. I don't know exactly why not. In standalone Cmd.exe it does wrap there.

The only thing I can do about this inside Clink is to add a setting to let the user choose how Clink determines the width of East Asian Ambiguous characters. I'm adding such a setting in the next build of Clink.

I believe there is a discrepancy in how ConEmu handles these East Asian Ambiguous characters. I will continue to try to work around ConEmu's behavior. It would be great if you could also investigate how ConEmu behaves under these conditions.

[chrisant996]

@hecerinc can you please try this version of Clink: clink.1.2.51.25aa54.zip

This has a new terminal.east_asian_ambiguous setting.
Please try the new Clink with the default setting (auto) and see if the problem still occurs.

If it does, then please try each of the following, to see which one(s), if any, resolve the problem:

• clink set terminal.east_asian_ambiguous font
• clink set terminal.east_asian_ambiguous one
• clink set terminal.east_asian_ambiguous two

If none of these resolve the problem, then I will produce a diagnostic test that can be used in different terminal hosts to report details about how each terminal host behaves with respect to the East Asian Ambiguous characters.

Thanks!

[hecerinc]

Unfortunately still happens with all 3 settings (and now there's interesting artifacts like the C: on top of the prompt when Ctrl+C or the double c in cclink when pressing arrow+up)

[chrisant996]

@hecerinc The screen shot only show leftover artifacts from the first 4 attempts, so it's very difficult to tell anything from the screen shot. Once the math is different between ConEmu and the OS, then there will definitely be some kind of artifacts left over. But seeing the artifacts doesn't give any indication about where the math went wrong.

Also, I am still unable to reproduce the problem.

So here are a few more diagnostic questions:

1. What code page are you using? Run chcp to find out the code page.
2. What language installation of Window are you using?
3. Does the width of the terminal display affect what happens? What happens if you make the terminal screen 100 or fewer characters wide?
4. Does the problem occur if you hold down the A key until the input line is longer than one full line? Or is it only happening when pasting hyperlinks? If it's exclusively when pasting hyperlinks, can you please share an example hyperlink that reproduces the problem?

[chrisant996]

I just noticed this was reported on Win11...

I cannot repro on Win10.
I CAN repro on Win11.
I can even repro it with code page 437 on Win11, but only in ConEmu.

Investigating...

[chrisant996]

@Maximus5 can you please reopen this issue? I am able to reproduce this by running a standalone exe in ConEmu, without Clink involved at all. Something is going wrong with how ConEmu disables wrapping (see below). It looks like maybe Win11 has exposed a latent bug in ConEmu's line wrapping logic.

@Maximus5 how does ConEmu decide when to wrap at the end of a line?
@miniksa has Win11 changed how it responds to ENABLE_WRAP_AT_EOL_OUTPUT?

I pulled latest ConEmu sources, built Debug x64, added a diagnostic test function into Clink, and spent several hours stepping through code in the debugger.

I am able to reproduce this problem without any Clink code running. I.e. Win 11 with code page 437, with just pure WriteConsoleW calls writing exactly 1 "a" character at a time to fill up the line. But only in ConEmu and only on Win11. No ANSI escape codes are involved, no Clink code running, and the WriteConsoleW calls are handled by ConEmu. Writing multiple characters does not exhibit the problem -- only writing single characters at the end of the line fails to wrap.

The reason pasting encounters the problem is because ConEmu produces a stream of single character inputs to effectively type the pasted text as input. And so each character is printed separately, which falls into the more general case above, i.e. writing single characters at the end of a line does not wrap.

It seems the reason Clink is involved is because the problem only occurs in the process that ConEmu hooks (i.e. the cmd.exe process). Standalone console programs work fine because ConEmu does not seem hook them.
UPDATE: I figured out how to reproduce the problem in a standalone exe running in ConEmu, without injecting Clink. Will share details tonight.

Here ExtWriteText() disables wrapping to the next line:

ConEmu/src/ConEmuHk/ExtConsole.cpp

Line 1104 in aab1d90

SetConsoleMode(h, Mode & ~ENABLE_WRAP_AT_EOL_OUTPUT);

And then here ExtWriteText() chooses not to wrap:

ConEmu/src/ConEmuHk/ExtConsole.cpp

Line 1253 in aab1d90

&& (!bRevertMode || ((pCur+1) < pEnd)))

I've stepped through the working and failing cases.
I don't see any difference in execution in ConEmu code!
On Win10 the text wraps, even though it looks as though ConEmu is disabling wrapping.
On Win11 the text does not wrap, no matter how many times I call WriteConsoleW(L"a") at the end of the line.

I can't tell if this is a ConEmu issue or a Windows 11 issue, but I don't see any way this could be a Clink issue.

Question 1:

@Maximus5 How does ConEmu decide when a WriteConsoleW(L"a") call needs to wrap to the next line?
Has it been wrapping due to some quirk/bug in the OS implementation that didn't always respect disabling wrapping?
Has Win11 maybe changed such that now disabling wrapping really does reliably disable wrapping?

Question 2:

@Maximus5 Is there a way to get ConEmu to hook WriteConsoleW in a standalone console program, not just in the shell?
If so then I can try to provide a minimal repro program.
UPDATE: I figured it out how to do this after I got some sleep. I am able to reproduce the problem in a simple standalone exe file in ConEmu, without Clink involved. I will share source code, executable, and symbols later tonight.

Question 3:

@Maximus5 have you used Time Travel Tracing before? Would it be helpful if I provided a Time Travel Trace with matching sources, binaries, and symbols?
UPDATE: unnecessary, now that I can reproduce the issue in a simple standalone exe.

[chrisant996]

@Maximus5 @miniksa Standalone repro program and details are available in the chrisant996/conemu2404 repo.

See the README.md there for repro instructions and details.

Interesting:

• Repro requires the ENABLE_VIRTUAL_TERMINAL_PROCESSING console mode to be set. (The repro program has flags to set it or to clear it.)
• Repro requires making ConEmu run the program as though it were a shell.
  • It looks like ConEmu has a bug that causes it to always turn off ENABLE_WRAP_AT_EOL_OUTPUT when printing a single character and the cursor X position is width - 1. I.e. printing a single character at the end of the line without that flag doesn't move the cursor, ConEmu keeps thinking it hasn't reached the end of the line, and keeps disabling wrapping.
  • Surprisingly, that has worked for years, i.e. it seems the OS somehow had some logic that respected the absence of the flag for first single character printed at the end of line, but wrapped anyway on subsequent single characters printed at the end of line (even though the flag was absent).
  • It appears Win11 has some change that now always respects ENABLE_WRAP_AT_EOL_OUTPUT, thus exposing what appears to be a latent bug in ConEmu.
• For me, repro occurs only on Win11, but in Weird input cursor positioning problem cmderdev/cmder#2628 a user reported that it occurs for them in both Win10 20H2 and 21H1.

[chrisant996]

@Maximus5 in addition to the preceding info, it looks as though microsoft/terminal#3943 may have exposed this long-standing latent issue in ConEmu's source code.

Feel free to contact me at my profile email address if you'd like to discuss and/or strategize how to fix the issue in ConEmu. It appears fixable with a status flag and some additional logic for when to set/clear the status flag.

[chrisant996]

@Maximus5 let me rephrase: this is not a clink or bash bug. This is a ConEmu bug that has been exposed by changes in the OS. The ConEmu bug will affect any program that writes one character at a time and reaches/passes the right edge of the terminal.

Would you like me to produce a sample program that reproduces the problem, without clink or bash involved?

[Maximus5]

@chrisant996 That would be nice. For now I guess that is a conhost bug (as ConEmu does not handle the prompt input). So with sample program the bug may be easily reported.

[chrisant996]

@chrisant996 That would be nice. For now I guess that is a conhost bug (as ConEmu does not handle the prompt input). So with sample program the bug may be easily reported.

@Maximus5 Sure, I will make a tiny repro program.

It's not a conhost bug, it is definitely a ConEmu bug. After I get home from the New Year festivities, I will share a link to the specific code, and describe the specific conditions.

In the meantime, a quick description is:

• ConEmu always turns off console auto-wrapping.
• When the cursor reaches the right edge, ConEmu sets the cursor back one position.
• But it doesn't set any other state to keep track that the next character output should wrap.
• So, if a program repeatedly writes one character at a time, then ConEmu keeps setting the cursor back one position, and never realizes it needs to wrap.
• Instead ConEmu tests whether the emitted characters go past the right edge, so it would take writing 2 characters at once for ConEmu to recognize the need to wrap.
• Clink and bash both write a single character past the end of the line, and so they trigger the ConEmu bug.

A quirk/bug in the OS had been failing to respect that the console mode said not to wrap, and was wrapping anyway. Now that the OS is respecting the console mode, it exposes the ConEmu logic problem above.

[chrisant996]

@Maximus5 the following program reproduces the problem in ConEmu:
ReproConEmu2404.zip

The enclosed Readme.txt has the instructions.

As you will see from its source code, the repro steps are simple.

I believe the relevant ConEmu source code is here:

Here ExtWriteText() disables wrapping to the next line:

ConEmu/src/ConEmuHk/ExtConsole.cpp

Line 1104 in aab1d90

SetConsoleMode(h, Mode & ~ENABLE_WRAP_AT_EOL_OUTPUT);

And then here ExtWriteText() chooses not to wrap:

ConEmu/src/ConEmuHk/ExtConsole.cpp

Line 1253 in aab1d90

&& (!bRevertMode || ((pCur+1) < pEnd)))

[chrisant996]

@Maximus5 wow, I totally missed that I had already shared a repro program on Dec 20. So now there are two repro programs.

The latest one is slightly simpler, but the earlier one is arguably even better, because it even pops up a message box right before the call that will malfunction, and it also captures a bunch of data about console state along the way.

Either sample program is reasonable to use.

I am happy to discuss further. I am willing to attempt making a fix in ConEmu, but the issue requires keeping track of persistent state across ExtWriteText calls, and resetting the state in response to some other calls (such as SetConsoleCursorPosition). I am not sure where to store persistent state in ConEmu. If you can describe where such state should be kept, then I am willing to share a PR with a proposed fix.

Ideally I would appreciate if you could take a look in the debugger to observe the issue, and make a statement about how you would recommend fixing the problem.

[chrisant996]

@Maximus5 can you please reopen this issue?

[rcmonitor]

Is there going to be a fix for this?
Maybe, some temporary quick-fix available right now?
It seemed that a lot of issues relates to this one.

[chrisant996]

@Maximus5 your help is needed.

This is not an OS bug. Or, rather, the OS fixed a bug, which exposed this ConEmu bug.

ConEmu's logic for deferring wrapping does not work as intended when only 1 character is written after wrapping has been deferred.

I am willing to do the work of producing a fix, but I need some help from you:

• Why is ConEmu disabling wrapping? It seems like that would inherently cause problems.
• Or does ConEmu already have some mechanism for storing console state between WriteConsole calls? Can you point me at it?

[Maximus5]

Definitely I need to check. Thanks for the tests.

[Maximus5]

https://conemu.github.io/blog/2022/03/08/Build-220308.html

[Maximus5]

https://conemu.github.io/blog/2022/04/18/Build-220418.html
Fix was reworked

[hecerinc]

This works for me now! Thanks 😄 somebody had mentioned that #2384 was related to this, but that's still reproing for me, in case that's useful.

[yuki-tm]

Verifying that the latest version of ConEmu fixes this issue.

If you're working with a terminal like Cmder and having this problem just overwrite the installed ConEmu files in vendor\conemu-maximus5, located in your root Cmder folder.