WebGL Backend: Centralize async reading of data, postpone disposal if a pending read

[dsmilkov]

All the work involves only the WebGL backend.

• Centralize async reading of data in a single setTimeout loop
• Postpone disposal if there is a pending read
• Use LRU cache to page textures out of the GPU memory and into CPU (Avoids crashing browser)
• Some optimizations:
  • Cache the ext.GPU_DISJOINT_EXT parameter since reading it is expensive
  • Do binary search for queryIsDone since checking is expensive
  • Reuse download buffers for readPixels to minimize js memory trashing

Did a stress test using demos/mnist_eager where I made tensor.dispose() a no-op. There is significant slowdown (~10x) due to constant downloads and js GC trashing memory, but the browser doesn't crash. The optimizations I did were motivated by the profiling from this stress test.

Fixes tensorflow/tfjs#394

This change is 

[nsthorat]

Nice work.

Can you add a unit tests where you mock window screen width and allocate a bunch of tensors and make sure nothing blows up?

---

Review status: 0 of 10 files reviewed at latest revision, all discussions resolved, some commit checks failed.

---

src/kernels/backend_webgl.ts, line 90 at r3 (raw file):

}

// Empirically determined constant used to decide the number of bytes on GPU

this makes it seem like this is the number of bytes. Maybe we can specify the bytes are this constant * screen area * dpi

---

Comments from Reviewable

[dsmilkov]

Added a unit test with 1x1 screen and allocation of much larger textures to make sure everything gets paged out of the GPU after the shader program finishes.

---

Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.

---

src/kernels/backend_webgl.ts, line 90 at r3 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

this makes it seem like this is the number of bytes. Maybe we can specify the bytes are this constant * screen area * dpi

Done.

---

Comments from Reviewable