image, image/draw: add interfaces for using RGBA64 directly

[zephyrtronium]

I have an application that involves calculating image data at extremely high resolutions, sometimes on the scale of gigapixels, then resampling the results to smaller sizes. Using package x/image/draw to perform the resampling gives excellent results, but rather slowly. One significant reason for this is that every call to At allocates, because the result type is an interface. A quick benchmark shows that this is responsible for 80% of the allocations in my application.

Package x/image/draw already contains specialized fast paths for many of the raw image types in package image. However, because my images are calculated by histogramming sometimes trillions of iterations of a chaotic process, I cannot use those types to accumulate the image. I also cannot copy to such a type prior to rescaling, because the images I'm working with already occupy most of the memory I have available.

In order to resolve this, I propose to create a standardized interface in package image that allows image processing routines to define fast paths that can avoid unnecessary allocations. The interface would be as follows:

package image

// RGBA64Image is an image with pixels that can be transformed directly to
// color.RGBA64.
type RGBA64Image interface {
	// RGBA64At returns the RGBA64 color of the pixel at (x, y). It is
	// equivalent to calling At(x, y).RGBA() and converting the resulting
	// 32-bit values to color.RGBA64, but it may avoid allocations from
	// converting concrete color types to the color.Color interface type.
	RGBA64At(x, y int) color.RGBA64
	Image
}

Additionally, I propose that a corresponding interface be added to package image/draw (with an alias in x/image/draw):

package draw

// RGBA64Image is an Image with a SetRGBA64 method to set the color of a
// single pixel. It is like Image, but using it may mitigate allocations from
// converting concrete color types to the color.Color interface.
type RGBA64Image interface {
	SetRGBA64(x, y int, c color.RGBA64)
	Image
}

I propose that the image types Alpha, Alpha16, CMYK, Gray, Gray16, NRGBA, NRGBA64, Paletted, and RGBA be extended to implement both of these interfaces (noting that RGBA64 already does), and that NYCbCrA, Uniform, and YCbCr be extended to implement the former.

With these interfaces, image processing algorithms in image/draw, x/image/draw, and elsewhere can implement generic fast paths, rather than only having fallbacks for image.Image that force at least an allocation per pixel. An experiment using a modified version of x/image/draw to implement such a fast path for Kernel.Scale shows that resampling an image from 1280x1280 to 128x128 goes from 3296775 to 32775 allocations per operation, a 100x improvement, with approximately a 3x speed improvement. In particular, the primary culprit method in my application disappears completely from the memory profile.

---

I originally proposed to add an interface with RGBA64At to x/image/draw specifically, rather than to the standard library, because the interface allocations are most impactful when processing extremely large images, and resampling seems like the most likely operation for them. However, I've since talked with others who have had serious performance issues using image/draw for color transformations on many images, and @nigeltao suggested in a comment that these interfaces be added to the standard library.

[ianlancetaylor]

CC @nigeltao

[nigeltao]

Note that the RGBAAt(x, y int) (r, g, b, a uint32) proposal conflicts with the existing func (p *RGBA) RGBAAt(x, y int) color.RGBA in the standard library's package image.

Perhaps add some interfaces to the standard library (and make its concrete image types implement them):

---

package image

type ImageV2 interface {
    Image
    // AtDotRGBA(x, y) is equivalent to At(x, y).RGBA() but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    AtDotRGBA(x, y int) (r, g, b, a uint32)
}

// or perhaps

type ImageV2 interface {
    Image
    // RGBA64At(x, y) is equivalent to At(x, y).RGBA(), after packing RGBA's
    // four uint32 return values into one color.RGBA64, but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    RGBA64At(x, y int) color.RGBA64
}

---

package draw

type ImageV2 interface {
    Image
    // SetRGBA64(x, y, c) is equivalent to Set(x, y, c) but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    SetRGBA64(x, y int, c color.RGBA64)
}

---

golang.org/x/image/draw obviously gets:

type ImageV2 = draw.ImageV2

Yeah, the names are horrible and inconsistent, but we're constrained by Go 1 backwards compatibility...

[nigeltao]

CC @dmitshur

[zephyrtronium]

@nigeltao
I had forgotten about image.RGBA.RGBAAt. Considering this, the RGBA64At(x, y int) color.RGBA64 interface does seem more appropriate.

Reading through it again, package image also has somewhat of a convention that would suggest a better name. The concrete image types are RGBA, Gray16, &c. whereas the interface types are Image and PalettedImage. So, it would be natural to add RGBA64Image.

Unless there are objections, tomorrow I will retitle and change this proposal to adding

// RGBA64Image is an image with pixels that can be transformed directly to
// RGBA64.
type RGBA64Image interface {
	// RGBA64At returns the RGBA64 color of the pixel at (x, y). It is
	// equivalent to calling At(x, y).RGBA() and converting the resulting
	// 32-bit channels to color.RGBA64, but it may avoid allocations from
	// converting concrete color types to the color.Color interface type.
	RGBA64At(x, y int) color.RGBA64
	Image
}

to package image, updating the concrete image types to implement it, adding a corresponding RGBA64Image interface to image/draw and x/image/draw, updating the concrete image types in package image to implement these interfaces, and implementing fast paths in image/draw and x/image/draw using these types.

[nigeltao]

Sounds good to me.

With the new fast paths, I'd only bother with:

• "image.RGBA64Image on draw.RGBA64Image"

and not e.g.

• "image.Image on draw.RGBA64Image" or
• "image.RGBA64Image on draw.Image"

[naisuuuu]

I was about to submit a separate proposal to add additional fast paths to x/image/draw for more resampling destination types. This proposal would solve that and many more similar problems in a much more elegant way. Really hoping to see it come to fruition!

[nigeltao]

@ianlancetaylor as above (March 12), this proposal (affecting the std image package) Sounds Good To Me.

What's the formal process from here? Do I upgrade this issue from Proposals/Incoming to Proposals/Active?

[gopherbot]

Change https://golang.org/cl/311129 mentions this issue: image: add RGBA64Image interface

[ianlancetaylor]

@nigeltao No, that move will be done when the proposal review committee gets to this issue.

I'll add it to the list to consider in the next meeting.

[rsc]

Seems OK. Adding to minutes.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[changkun]

The CL was sent before the freeze. The proposal was also accepted roughly on the border of dev and freeze. Although the beta1 is out for few days, is there still a chance to get this in in the 1.17 release? So that we won't have to wait another 6 months?

[ianlancetaylor]

Someone will have to review the CL.

Once that is done, you'll have to ask for a freeze exception. See https://groups.google.com/g/golang-dev/c/VNJFUxHWLHo/m/PBmGdYqoAAAJ .

[nigeltao]

The reviewers have since responded. I filed #46811 to discuss the freeze exception.

[dmitshur]

2 questions via #46688.

In the original proposal, it was suggested that:

I propose that the image types Alpha, Alpha16, CMYK, Gray, Gray16, NRGBA, NRGBA64, Paletted, and RGBA be extended to implement both of these interfaces (noting that RGBA64 already does), and that NYCbCrA, Uniform, and YCbCr be extended to implement the former.

CL 311129 implemented all of that, with the exception of image.Uniform.

@nigeltao Can you confirm leaving out Uniform is intentional and okay? (Asking this question for #46688.)

Also, the image/draw.RGBA64Image interface suggested in the proposal was:

type RGBA64Image interface {
	SetRGBA64(x, y int, c color.RGBA64)
	Image
}

CL 311129 implemented a slightly different one, which includes the RGBA64At(x, y int) color.RGBA64 method:

type RGBA64Image interface {
	image.RGBA64Image
	Set(x, y int, c color.Color)
	SetRGBA64(x, y int, c color.RGBA64)
}

I just want to point it out because it's a difference, and ask @nigeltao to confirm this is intentional and okay for purposes of #46688. Thanks.

(Everything else matches.)

[gopherbot]

Change https://golang.org/cl/331570 mentions this issue: image: add Uniform.RGBA64At and Rectangle.RGBA64At

[nigeltao]

@nigeltao Can you confirm leaving out Uniform is intentional and okay?

Leaving out image.Uniform in CL 311129 was unintentional.

On further thought, leaving out image.Rectangle (which implements image.Image) in both the CL and this proposal issue was an additional oversight.

My guiding principle is that, if a concrete image.Foo type previously implemented the image.Image interface, it should now also implement the image.RGBA64Image interface. Similarly, if it previously implemented the draw.Image interface, it should now also implement the draw.RGBA64Image interface.

I have sent out CL 331570 to address image.Uniform and image.Rectangle.

I just want to point it out [the interface definitions] because it's a difference, and ask @nigeltao to confirm this is intentional and okay

Yes, it's intentional that each draw.Foo interface is a superset of image.Foo (and for bar.QuxImage to be a superset of bar.Image). Sorry for the oversight in the proposal. Thanks for catching it. No further action required.

[gopherbot]

Change https://golang.org/cl/340049 mentions this issue: image/draw: add RGBA64Image fast path

[gopherbot]

Change https://golang.org/cl/351852 mentions this issue: image/draw: add RGBA64Image fast path for RGBA dst