Interfaces to expose internal APIs for SIXEL

[diamondburned]

This pull request adds several interfaces to optionally expose screen implementations for SIXEL graphics and such. It does not break any existing API or behavior. The Windows implementation of the screen is untouched and therefore will not support any of these new interfaces.

Author: diamondburned <[email protected]>
Date:   Fri Feb 26 23:24:24 2021 -0800

    Add interceptor and pixel APIs for raw terminal support
    
    This commit adds several interfaces to allow raw terminal support, such
    as for SIXEL graphics.
    
    DrawInterceptor was added for extensions to intercept each draw event
    to synchronize its state with the screen's and redraw things it needs to
    while occupying the screen lock to prevent data race with the user.
    
    PixelSizer was added to expose the screen's size in pixels from
    TIOCGWINSZ. This is needed to properly resize and position SIXEL images
    on the terminal.
    
    CellBufferViewer was added for low-level efficient damage tracking for
    draw interceptors to synchronize the screen state.

Author: diamondburned <[email protected]>
Date:   Thu Mar 4 17:19:36 2021 -0800

    Dispatch ResizeEvent after drawing
    
    This commit changes the Unix screen to dispatch events only after
    drawing. This helps draw interceptors update its states in time before
    the user gets the event.

Proof-of-concept

As of the time this pull request was made, a tcell SIXEL library was written as a proof-of-concept that such an API could be used for not just static SIXEL images but also animations:

[codecov]

Codecov Report

Merging #436 (435b35e) into master (c94849e) will decrease coverage by 0.52%.
The diff coverage is 5.42%.

@@            Coverage Diff             @@
##           master     #436      +/-   ##
==========================================
- Coverage   16.91%   16.39%   -0.53%     
==========================================
  Files          20       21       +1     
  Lines        1383     1452      +69     
==========================================
+ Hits          234      238       +4     
- Misses       1135     1200      +65     
  Partials       14       14              

Impacted Files	Coverage Δ
interceptors.go	0.00% <0.00%> (ø)
screen.go	0.00% <ø> (ø)
tscreen.go	0.00% <0.00%> (ø)
tscreen_unix.go	0.00% <0.00%> (ø)
cell.go	68.57% <33.33%> (-7.24%)	⬇️
color.go	86.20% <0.00%> (+4.72%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c94849e...435b35e. Read the comment docs.

[gdamore]

My biggest concern with this is understanding how we handle understanding where the graphical content, and what the interaction with text is or should be. We don't know the size of the text (although perhaps we can take the screen dimensions in pixels and divide by the screen dimensions in characters?)

What are the interactions if the terminal is resized to a partial character width? (Is that possible?)

Are sixel images supposed to occupy an integral number of character cells? What happens if they don't? Are there interactions with text placed nearby?

How will this all work with the redraw model, which keeps a model of the screen in case the screen needs to be resized?

Ultimately, I need to better understand how this graphical content is going to work within our larger model. This represents a substantial enough departure from everything else we do, that while it's cool, I don't want to rush into it only to be faced with a ton of problems later.

[diamondburned]

Before answering those questions, I'd like to add that the added changes actually have nothing to do with SIXEL directly. They're changes that merely expose the underlying APIs so the caller have more control over the terminal, and since tcell (draw-wise) does not handle automatic (and/or relative) layouting, the current design doesn't conflict with anything already in place.

---

We don't know the size of the text (although perhaps we can take the screen dimensions in pixels and divide by the screen dimensions in characters?)

This is what the PixelSizer interface is supposed to expose: TIOCGWINSZ by default tries to get the terminal's pixel sizes, and on terminals capable of drawing SIXEL images, the pixel sizes can be obtained. The cell dimension can then be calculated from it. Below is a snippet from tsixel's implementation which allows getting the cell dimension in pixels:

// ScreenState stores the screen size in two units: cells and pixels.
type ScreenState struct {
	Cells  image.Point
	Pixels image.Point
}

func (sz *ScreenState) update(screen tcell.Screen) {
	sz.Cells.X, sz.Cells.Y = screen.Size()

	pxsz, _ := screen.(tcell.PixelSizer)
	sz.Pixels.X, sz.Pixels.Y = pxsz.PixelSize()
}

// CellSize returns the size of each cell in pixels.
func (sz ScreenState) CellSize() image.Point {
	return image.Point{
		X: sz.Pixels.X / sz.Cells.X,
		Y: sz.Pixels.Y / sz.Cells.Y,
	}
}

---

What are the interactions if the terminal is resized to a partial character width? (Is that possible?)

This should depend on how the geometry is implemented, but from what I've seen of my implementation, the geometry just gets calculated to 0:

I don't even think this is possible, and the terminal is likely smart enough to report the appropriate pixel sizes. For example, the terminal I'm using (foot) reports a geometry of (2, 1) for the size in cells and (16, 16) for the size in pixels.

---

Are sixel images supposed to occupy an integral number of character cells? What happens if they don't? Are there interactions with text placed nearby?

They're not supposed to. The only size limit for SIXEL is that the height must be in multiple of 6, because each SIXEL strip is 6 pixels tall and 1 pixel wide. To properly layout SIXEL images onto the terminal in a cell-friendly way, certain edge cases must be accounted for. Below is tsixel's implementation of a function that converts a pair of coordinate from cells to pixels:

// PtInPixelsRounded converts a point which unit is in cells to pixels and
// rounds it to be within the SIXEL multiples.
func (sz ScreenState) PtInPixelsRounded(pt image.Point) image.Point {
	cell := sz.CellSize()

	pt.X *= cell.X
	pt.Y *= cell.Y

	// Whatever.
	if pt.X == 0 || pt.Y == 0 {
		return pt
	}

	// Round the image down to the proper SIXEL heights.
	excessY := pt.Y % SIXELHeight

	// Account for this loss in the width.
	pt.X -= excessY * pt.X / pt.Y // floor division
	pt.Y -= excessY

	// Round the image down to the cell size after we changed the size to no
	// longer round.
	if excessY > 0 {
		excessX := pt.X % cell.X

		// Account for the loss of the loss.
		pt.X -= excessX
		pt.Y -= ceilDiv(excessX*pt.Y, pt.X) // ceiling division
	}

	return pt
}

---

How will this all work with the redraw model, which keeps a model of the screen in case the screen needs to be resized?

CellBufferViewer was exposed for this reason. Its job is to expose the underlying cell buffer for damage tracking using the new DirtyRegion method, which behaves similarly to Dirty with a slightly different out-of-bound behavior.

SIXEL implementations may or may not utilize this API to minimize redraws: if the region is not damaged and the terminal was not resized, then the SIXEL image would still be on the screen, therefore a redraw of that is not needed. Right now, tsixel does utilize this API.

[diamondburned]

Unrelated to this PR (at all), it seems like foot supports terminal synchronized updates. I think it would be another nice thing I could work on for a pull request.

[gdamore]

Unrelated to this PR (at all), it seems like foot supports terminal synchronized updates. I think it would be another nice thing I could work on for a pull request.

There is probably very little benefit for a tcell application from that - at least traditionally - because Tcell already issues the updates intelligently, so that we're not drawing, and redrawing the same areas of the screen. The only thing it would help is "tearing" in the face of a change made while e.g. a screen resize is done. Basically it would let us get some kind of double-buffering.

And again, the problem with these as is usual is figuring out how to determine it's presence.

[gdamore]

This API cannot work with Windows console (legacy) terminals, because they don't express the drawing as a byte stream with embedded characters.

Likewise for web based terminals.

[gdamore]

The master branch has evolved in ways that may be incompatible with this -- there are a number of conflicts, and some of what I think you wanted is already done.

However, for me as I look at this there are also some challenging questions. For example, what about terminal implementations that don't offer pixels at all?

I'm thinking that a better way to do this, is to extend the Tty interface, since sixel graphics can only work on terminfo based terminals. Your extension could add the PixelSizer API there. (I would also be willing to contemplate just changing WindowSize() to return the WindowSize struct from x/or/sys/unix I suppose -- although perhaps we should put a type alias in there for that.

The other thing I'd do is then add at that level the interceptor there. So you have direct access to the file handle (fd) we use to talk to the terminal.

I would be willing to add a DrawHook() that gives you the ability to draw before or after (maybe you only need one of these - it looks like your code already does this?) the rest of the terminal update is done. I think that API needs to be done in tcell directly, and it might be useful to use it for testing.

So instead of intercepteradder as a new interface, I'd just extend Screen().

I'd probably also do that for CellBufferViewer. But maybe the CellBuffer could just be passed as an argument to your hooks. That would probably be a simpler API, and avoid asynchronous concerns.

[rockorager]

The other thing I'd do is then add at that level the interceptor there. So you have direct access to the file handle (fd) we use to talk to the terminal.

I like this approach. I have done a few different ways to implement sixels with tcell, and the limitations all come down to:

1. Assume os.Stdout is the same as the tty tcell is drawing to (fair assumption most of the time, but I'd rather have Write access to the Tty interface)
2. Mark the region I've drawn a sixel in as dirty. I can hack around this by setting a "unique" tcell.Style for the cells I want to be considered dirty.

To me, exposing the Tty and exposing a way to manually set a region of the screen as dirty should be sufficient. My implementations haven't needed a draw hook to work, but there are certainly other ways to do it that would require it I imagine.

type Screen interface{
        ...
        // Tty returns the underling tty, if the screen is a terminal 
        Tty() (Tty, bool)
        
        // SetDirty sets the underlying cells as dirty
        SetDirty(x, y, width, height int)
}

[rockorager]

@diamondburned Do you mind if I take a stab at a new PR addressing this?

[diamondburned]

@diamondburned Do you mind if I take a stab at a new PR addressing this?

Please do; I also really want to see a better API for SIXEL support in tcell.

I'll close this PR once you can make a new one.

[gdamore]

I think the TTY being exposed now enables this, and we can close this PR. I'm closing it. If there is additional functionality needed, please feel free to open a new one based against current master.