tmux: enable utf8proc

[joshuarubin]

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same formula update/change?
• Have you built your formula locally prior to submission with brew install <formula> (where <formula> is the name of the formula you're submitting)?
• Does your submission pass brew audit --new-formula <formula> (after doing brew install <formula>)?

---

This enables the use of utf8proc for calculating character widths. This fixes most issues with tmux on mac.

Here is the PR this originated from: tmux/tmux#524

[mistydemeo]

If this is recommended for all users, do you think it's worth revisioning the formula so that everyone is upgraded to this?

[MikeMcQuaid]

@BrewTestBot test this please

[MikeMcQuaid]

Can you provide more details as to when this is needed? Is it needed when using Terminal.app?

[joshuarubin]

@mistydemeo:

If this is recommended for all users, do you think it's worth revisioning the formula so that everyone is upgraded to this?

Probably. I updated the PR.

@MikeMcQuaid:

Can you provide more details as to when this is needed? Is it needed when using Terminal.app?

Changes introduced in tmux 2.2 removed much of tmux's built in handling for unicode in favor of relying on system libraries. This cleaned up the codebase and, on many operating systems, improved tmux's support of unicode. However, the Mac's wcwidth function (man 3 wcwidth, "number of column positions of a wide-character code") has been broken for a very long time, often returning -1 (which the homebrew patch to tmux 2.2 "fixed" to mean 1 column) and sometimes returning the wrong column width entirely for a character.

This is what caused people to have lots of problems with tmux on mac for 2.2, especially if they had unusual characters for terminal prompts or tmux status lines.

I worked with the tmux author to integrate optional support for utf8proc which was developed for the julia language and is widely seen as an excellently designed and maintained, cross-platform unicode support library.

When utf8proc is enabled, all unicode character widths are calculated using it, instead of the built in wcwidth.

I should note that there are a few edge cases that I believe should be updated within utf8proc (detailed here: JuliaStrings/utf8proc#83). However, by and large, it is a massive improvement over the mac's built in libraries.

To answer your question, yes, it greatly improves support for wide characters in Terminal.app, iTerm2.app, any terminal in which tmux is run.

[MikeMcQuaid]

Ok, thanks for the comprehensive response @joshuarubin!

[MikeMcQuaid]

Thanks again for your contribution to Homebrew! Without people like you submitting PRs we couldn't run this project. You rock!

[joshuarubin]

Thanks a bunch!

[theckman]

@joshuarubin I think this change has caused tmux to have some redraw issues on my MBP. Specifically, emacs is rendering things all garbled. Here's a short video showing the issue, the 2.3_1 usage starts at about 0:40:

• https://www.youtube.com/watch?v=ZbKyhZRwhtw

[joshuarubin]

I was able to reproduce this, only after installing spacemacs though (I'm not much of an emacs guy). It seems that some of the characters used in the mode line are getting their width calculated differently (from before, and from how the terminal calculates character width) by utf8proc which is causing a line overflow. Disabling the mode line should fix the rendering (or at least prove that it is causing the issue).

This is a dicey issue. The problem is that wide characters (like emoji) should usually be rendered with 2 columns, but that Terminal.app, iTerm2.app and likely others actually render it as 1 column (try running echo "😀z" in a terminal, you'll see that the z gets covered up). This is also font dependent as some fonts may incorrectly implement normal and wide characters widths.

Figuring out the width a character should be is one thing (which utf8proc does very well). Figuring out the width that the terminal actually displays it as is another issue. It probably isn't realistic to try to reconcile all of these differences (where the terminal will almost always be the offender if utf8proc is used).

iTerm2.app (nightly) has an option to enable Unicode 9 character widths to address this issue, but that breaks a whole lot of other things and really isn't safe to enable yet.

I already override some of utf8proc's calculations here to get better compatibility with how terminals actually render the characters.

If we can identify the characters causing the issue, we may be able to patch it in homebrew, though I am not sure if it will be accepted upstream in tmux, particularly if we have many individual cases to check and can't just use a unicode category.

[MikeMcQuaid]

If we can identify the characters causing the issue, we may be able to patch it in homebrew, though I am not sure if it will be accepted upstream in tmux, particularly if we have many individual cases to check and can't just use a unicode category.

Just a note to avoid people wasting time: if it's not accepted upstream we won't accept it in Homebrew either.

[wincent]

This causes some horrible rendering glitches for me (reported in tmux/tmux#632), so I've been forced to uninstall the Homebrew-supplied tmux and build my own without --enable-utf8proc from master.

Is there a way to make this optional or overridable? Feels like there may be a long tail of issues in utf8proc that may take a while to address upstream.

[theckman]

I am starting to think the same. I worked around it by reverting my local Homebrew repo to the commit before this change was added, installed tmux and used brew switch to stay pinned to the working version.

This was absolutely a breaking change for me. I've yet to discover the exact cause of my rendering glitch, but I'd like to be able to continue using the latest tmux without these undesirable behaviors.

It may be worth mentioning that while yet I was aware of the issues this was trying to solve, the bugs it introduces for me are worse than the original problem.

[MikeMcQuaid]

I'd like to get @joshuarubin's thoughts on this.

@wincent @theckman Are you using iTerm or Terminal.app?

[wincent]

@MikeMcQuaid: iTerm.

[theckman]

@MikeMcQuaid I've replicated my issues in both iTerm and Terminal.app, but I primarily use iTerm.

[joshuarubin]

Can we get some examples of the problems? Are you using uncommon fonts or symbols? The best thing I can think would be to make the utf8proc change configurable in the brew file itself so people can opt out of it if it's not working for them.

[MikeMcQuaid]

@joshuarubin An option is the easy way out. We need to decide what the default should be. I'm thinking we should defer to tmux's defaults. Do you know why they weren't changed?

[theckman]

@joshuarubin The only place I've hit it was using Spacemacs (Emacs). It's probably worth mentioning that I reverted back when I commented above / provided the YouTube link, which was probably within a few hours of finding it. So I didn't have much time to find other issues.

[wincent]

Can we get some examples of the problems? Are you using uncommon fonts or symbols?

I doubt that the font is relevant. See these issues in the tmux repo for examples:

• Tmux couldn't render some unicode characters (➊, ➋, ➌, ...) tmux/tmux#631
• Unicode character in status bar confuses tmux tmux/tmux#632
• Ambiguous console cursor position tmux/tmux#637
• issues displaying 'ⓔ' tmux/tmux#639

Seems like lots of people are running into this.

I'm thinking we should defer to tmux's defaults.

I don't know why the default was changed, but utf8proc was added in tmux/tmux@6c94774 and disabled by default a couple weeks after in tmux/tmux@6e8f400

[joshuarubin]

There is a change in tmux not yet in the latest release (I think) that removes some character overrides that were put in place when utf8proc was used. Because of this, it's unclear in each of these cases, 1) is utf8proc used and 2) were the overrides in place if it was?

[wincent]

All of my testing has been against the current HEAD of master in the tmux repo.

[MikeMcQuaid]

I think it's worth reverting back to the tmux default and letting them decide what to set this value to.

[joshuarubin]

So that means the overrides are no longer in place. The overrides were for all things in the unicode "symbol" category. They protected people using things like powerline on a unicode8 terminal. If you use iterm2, switch to the nightly and enable unicode9 widths, see if that helps.

[joshuarubin]

Can we keep utf8proc as a brew level option in any case?

[MikeMcQuaid]

Sure.

[theckman]

@joshuarubin @MikeMcQuaid thank you for your continued discussion around this. I had some spare time today so I whipped-up a quick Homebrew PR to remove utf8proc as being included by default, but allowed it as an option if you want to build against it.

This is #6838.

[theckman]

Just to add a final update to this issue. A change was made in #6838 to disable utf8proc by default. This has been merged in to master approximately 23 hours ago. The rendering issues in this ticket should be resolved (for now).

@joshuarubin I'll give your suggestion a go in the coming days and do some playing around with iTerm2 nightlies. Hopefully that will make things work 💯.