[Bug] Poor multi-frame cine loop playback

[jmannau]

Describe the Bug

Currently rendering multi-frame cine images, such as multi frame ultrasounds, isn't great. Slower machines can drop frames, particularly when multiple multi-frame cine images are rendered at the same time.

Steps to Reproduce

1. Goto https://viewer.ohif.org/viewer?StudyInstanceUIDs=1.2.840.113663.1500.1.248223208.1.1.20110323.105903.687
2. Change the layout to 2x2
3. Add an instance to each frame
4. Press the cine-play tool play button

The current behavior

The following tests were run in OHIF demo viewer:

Version number: 3.9.2
Commit hash: 610faa5a2738da5eabc40e57e338c359f481e852

The image below if a performance snapshot, recorded on a Mac M1 Pro, running Mac OX 15.1.1 and Chromium 131.

From this, there appear to be 2 opportunities to improve performance. Other than the time taken to actually render each frame, there are 2 calls that combined take approx 50% of the execution time.

1. combineFrameInstance is being called a lot (possible more than once per render). It is called in StackViewport, getFrameOfReferenceUID (

   cornerstone3D/packages/core/src/RenderingEngine/StackViewport.ts

   Line 534 in 160d80f

   this.getImagePlaneReferenceData(sliceIndex)?.FrameOfReferenceUID;

   ) which fetches ImagePlaneModule - is it possible to cache the ImagePlaneModule either in StackViewport or in the metadataProvider to reduce the time taken to calculate this for every frame?

2. fastComputeRange which is a vtk.js method is the other opportunity. vtk.js calculates the range of every

   I have spent some time investigating this.

   1. On each render, StackViewport creates an instance of vtkDataArray and then calls render.
   2. On render, vtk.js eventually calls ImageMapper.buildBufferObjects (https://github.com/Kitware/vtk-js/blob/35a933890f491d419ccfd376efabe6b19c252982/Sources/Rendering/OpenGL/ImageMapper/index.js#L932). This retrieves a reference to the vtkDataArray instance created in step 1.
   3. vtk.js then creates a new vtkDataArray on line 1320 (https://github.com/Kitware/vtk-js/blob/35a933890f491d419ccfd376efabe6b19c252982/Sources/Rendering/OpenGL/ImageMapper/index.js#L1320) ready for displaying. It is this new instance that has getRange and subsequently fastComputeRange called on every render.

   In order to improve performance, we need to find a way to reduce/remove the need to calculate the range of this new vtkDataArray on every render.

   It is possible to pass pre-calculated range in StackViewport when creating the image vtkDataArray (see https://kitware.github.io/vtk-js/docs/structures_DataArray.html#Scalar-array-in-memory). ie:

   const scalarArray = vtkDataArray.newInstance({
     name: 'Pixels',
     numberOfComponents: numberOfComponents,
     values: values,
     ranges: [
       {
         min: ...,
         max: ...,
         component: 0
       },
       {
         min: ...,
         max: ...,
         component: 1
       },
       {
         min: ...,
         max: ...,
         component: 2
       },
     ]
   });

   which could be calculated by the imageLoader. However this won't help as vtk.js doesn't use the range of imgScalars when building the texture to render.

   So the 2 options appear to be:

   1. find a way to cache pre-rendered vtk frames removing the need for vtk.js to recalculate the frame on every render.

   or

   2. update vtk.js ImageMapper to use the range of imgScalars to reduce the need to calculate the range. For instance, in the cases I examined, the ijkMode === SlicingMode.NONE (https://github.com/Kitware/vtk-js/blob/35a933890f491d419ccfd376efabe6b19c252982/Sources/Rendering/OpenGL/ImageMapper/index.js#L1296) which then takes a direct copy of imgScalars. In this case, is it possible to copy the imgScalars range and apply them to the constructed dataArray in lines https://github.com/Kitware/vtk-js/blob/35a933890f491d419ccfd376efabe6b19c252982/Sources/Rendering/OpenGL/ImageMapper/index.js#L1320-L1327

The expected behavior

Playback of multi-frame ultrasound images should be smooth on as many devices as possible.

OS

MacOS 15.2

Node version

N/A

Browser

Chromium 131.0 (Brave)

[sedghi]

we recentely did some performance optimizations for multiframe in OHIF via OHIF/Viewers#4693 and OHIF/Viewers#4560

I will include these in upcoming 3.9.3

can you try with latest and see if it is improved before i go deep into vtk side?

[jmannau]

Hi, I think I mislead you in the issue a bit.

The core issue is related to using cornerstone3d in our app. For instance, we use cornerstone v2 to display a series of cine-loop ultrasound. See the screen shot below.

In our app, we get the same performance issue as documented in OHIF above. A screenshot of the Chrome performance analysis is below, showing the same high activity of fastComputeRange

We have a custom metadata provider which explains why combineFrameInstance isn't shown in this performance snapshot.

I referred to OHIF in the original issue as it is the easiest way to replicate the performance trace and shows what I think is the biggest issue with cine-loop rendering performance. This is why I did a deep dive int vtk.js

[sedghi]

find a way to cache pre-rendered vtk frames removing the need for vtk.js to recalculate the frame on every render.

This doesn't work. The camera (pan, zoom etc), windowlevel etc can be changed during the cineloop which would require a need to re-render. Then we need to have image comparisons etc, it simply gets out of hand.

I feel like caching the range, and just change vtk.js to accept those ranges and bypass its range calculation is the easiest way forward

[sedghi]

Initially, I also supported caching the multi-instance metadata instead of recreating it each time, but it seems @wayfarer3130 has some concerns. I think it would be fine, honestly.