Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WebGL support #756

Merged
merged 100 commits into from
Feb 3, 2022
Merged

Add WebGL support #756

merged 100 commits into from
Feb 3, 2022

Conversation

Juice10
Copy link
Contributor

@Juice10 Juice10 commented Nov 9, 2021

This PR adds support for webgl recording and replaying and adds tests to ensure its stability.

This PR covers the following:

  • Tweak naming of vscode workspace to be a little more explicit
  • Basic webgl recording & replay support
  • Support for recording & replay in webgl2 context
  • Testing visual snapshots with jest-image-snapshot
  • Capture, serialize and deserialize WebGL variables
  • Serialize ImageData
  • Serialize HTMLImageElement
  • Record Canvas before it gets added to the dom (webgl and webgl2).
  • Preload HTMLImageElement used in webgl events
  • Fix bug: documents that have a canvas object that hasn't been accessed by any context yet, automatically get their '2d' context invoked which disables any further attempt at accessing them via webgl
  • Speed up and slow down webgl replay to align with the frame-rate of the replaying browser
  • Sort deserializes webGL variables by context
  • Buffer up webgl events and emit them as one larger event (possibly with throttling)
  • Handle speeding up rrweb playback (newFrame: true) slows this down.
  • Implement Canvas Manager and respect record.freezePage()
  • And much more...

Not implemented yet, out of scope from this PR:

@Juice10 Juice10 marked this pull request as ready for review December 24, 2021 10:40
@eoghanmurray
Copy link
Contributor

Idea: We might want to introduce a maximum framerate here for recording of webgl.
i.e. decoupling the raw webgl events from the emission of an rrweb, which might contain multiple webgl api calls.
(a maximum frame rate could be applicable to mutations also)

@eoghanmurray
Copy link
Contributor

eoghanmurray commented Jan 7, 2022

if that decoupling was in place, we'd also likely want to respect the freezePage aspect of the public API, whereby we want to pause recording of background activity, i.e. buffer up the activity so that it can all be emitted later if the recording is unfrozen:

record.freezePage = () => {
mutationBuffers.forEach((buf) => buf.freeze());
};

Instead of fixing it on the replay side we buffer up webgl canvas mutations and wait for a new RAF to flush them. This allows us to remove `newFrame` from the events and simplify things a little
@Juice10
Copy link
Contributor Author

Juice10 commented Jan 10, 2022

Idea: We might want to introduce a maximum framerate here for recording of webgl.
i.e. decoupling the raw webgl events from the emission of an rrweb, which might contain multiple webgl api calls.
(a maximum frame rate could be applicable to mutations also)

WebGL is a bit of a black box so unfortunately its hard to tell which of the 200+ webgl2 invoked commands are necessary for playback of future frames. This makes it very hard to tell which commands we can leave out without breaking the (future) playback.
If we where to introduce a maximum frame rate for webgl recording and we'd effectively have to keep every invoked command, then you'd have less events but the events would be much bigger, negating most of your potential savings.

There also isn't really a full snapshot like capability with webgl, whereby we could snapshot a point in time and animate with webgl commands from a full-snapshot.
The closest we could get to that is calling canvas.toDataURL every Nth frame but that can be pretty heavy both in terms of performance and file size.

@Juice10
Copy link
Contributor Author

Juice10 commented Feb 1, 2022

if that decoupling was in place, we'd also likely want to respect the freezePage aspect of the public API, whereby we want to pause recording of background activity, i.e. buffer up the activity so that it can all be emitted later if the recording is unfrozen:

record.freezePage = () => {
mutationBuffers.forEach((buf) => buf.freeze());
};

I've added a CanvasManager (35da32d) which respects record.freezePage and won't send any events as long as the MutationBuffer is frozen or locked.

@Yuyz0112 Yuyz0112 merged commit ab9fed0 into master Feb 3, 2022
Yuyz0112 pushed a commit that referenced this pull request Feb 3, 2022
* Add very basic webgl support

* document the default

* only capture rr_dataURL in 2d canvas contexts

* rr_dataURL no longer part of webgl snapshot

* ignore __diff_output__ from jest-image-snapshot

* Rename generic "Monorepo" to "RRWeb Monorepo"

* Serialize WebGL variables

* Move rrweb test port number to unique port

rrweb-snapshot uses 3030, rrweb uses 3031

* Prepare for WebGL2

* Split up canvas replay and record webgl vars

* fix typo

* fix typo part 2

* fix typo

* Handle non-variables too

* provide correct context for warning

* (De)Serialize a lot of different objects

* monorepo root should be the first in the list

* Upgrade puppeteer to 11.x

* Correctly de-serialize webgl variables

* Encode arrayBuffers contents to base64

* rename contents to base64

* add webgl2 support and serialize HTMLImageElements

* Support serializing ImageData

* Correctly classify WebGL2 events

* Serialize format changed

* check if canvas has contents before we save the dataURL

* Remove blank dataURL

* reference original file not type defintion file

* update types

* rename code worspace

* update dependencies

* add spector to inspect webgl

* remove live server settings from code workspace

* Save canvas context in the node

Prevents from saving webgl canvases as 2d dataUrls

* remove extra braces

* add ICanvas type

* use ICanvas from rrweb-snapshot in rrweb instead of OgmentedCanvas

* add snapshots and webgl 2 tests

* Upgrade to puppeteer 12.0.1

* Revert back to puppeteer 9.1.1

* Keep index order consistent between replay and record

* keep correct index order in webgl2

* fixed forgotten import

* buffer up pending canvas mutations

* unify the way webgl and webgl2 get patched

* fix parsing error

* Add types for serialize-args

* Add debugging for webgl replay

* Move start-server to utils

* turn off debug mode by default

* Move pendingCanvasMutations to local object and fix if/else statement

* Always save pending mutations

* only use assert snapshot as it's clearer whats going on

* Ugly fix for now

* Making the tests more DRY

* flush at the end of each request animation frame

* Looks like the promise made this test more predictable

* add waitForRAF

* Make nested iframe recording robust no matter the test speed

* mute noisy error in test

* force a requestAnimationFrame

* Bundle events within one frame together as much as possible

WebGL events need to be bundled together as much as possible so they don't accidentally get split over multiple animation frames.

 `newFrame: true` is used to indicate the start of an new animation frame in the recording, and that the event shouldn't be bundled with the previous events.

* Rename RafStamps

* Override event.delay

* cleanup

* Add tests for addDelay

* Add webgl e2e test

* Remove settimeout

* DRY-up test

* Preload images in webgl

* Add e2e test for webgl image preloading

* don't turn on devtools by default!

* Remove spector

* close server after use

* Add imageMap parameter

* Make e2e image test more robust

* document debug mode

* cleanup

* WebGL recording in iframes & Safari 14 support

* fix tests

* don't save null objects as WebGLVar

* group (de)serialized webgl variables by context

* Fix test

* fix tests

* bundle webgl mutations on request animation frame

Instead of fixing it on the replay side we buffer up webgl canvas mutations and wait for a new RAF to flush them. This allows us to remove `newFrame` from the events and simplify things a little

* Add canvas element to mutation observer file

* Add Canvas (Mutation) Manager

Allows you to do `record.freezePage()` and canvas events will get paused.

Based on #756 (comment)

* cleanup

* Make sure the correct </body> gets replaced

* Perf: Speed up check to see if canvas is blank

* Access unpatched getImageData

* Use is2DCanvasBlank only for 2d context

* Only record canvas when recordCanvas is true
@LIFM0623
Copy link

LIFM0623 commented Jun 6, 2024

您好,我有个需求是记录地图变化,地图是mapbox_gl的,是否因为这个webgl渲染的原因导致无法记录

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants