VolumeViewport Orientation when ImageOrientation metadata is unusual

[LittleURF]

TLDR:
How to set proper orientation for a VolumeViewport when ImageOrientationPatient metadata doesn't match the provided constant orientations?
---

Hi,
I'm having issues with finding out the proper Orientation values to set, when I'm rendering a volume with ImageOrientation metadata values different than the usual Axial/Sagital/Coronal values.

I believe that the ImageOrientation values map to orientations like this:

    coronalImageOrientationPattern = [1, 0, 0, 0, 0, -1];
    // sliceNormal: <Point3>[0, 1, 0],
    // viewUp: <Point3>[0, 0, 1],

    sagittalImageOrientationPattern = [0, 1, 0, 0, 0, -1];
    // sliceNormal: <Point3>[1, 0, 0],
    // viewUp: <Point3>[0, 0, 1],

    axialImageOrientationPattern = [1, 0, 0, 0, 1, 0];
    // sliceNormal: <Point3>[0, 0, -1],
    // viewUp: <Point3>[0, -1, 0],

But when the ImageOrientation value is different(like below), the standard orientations don't result in the expected Axial/Coronal/Sagital views.
[0.85005025920695,-0.5267015823236,2.1648092e-8,0.00883004239504,0.01425087456269,-0.9998594615872].

I think cornerstone doesn't have any instructions about what to do in such case.

So I attempted to create a custom orientation based on the ImageOrientation, I found out that the sliceNormal part is created by crossing the ImageOrientation like this: sliceNormal = cross(imageOrientation.slice(0, 3), imageOrientation.slice(3, 6)) , though I'm not sure if it's correct. And for the ViewUp part I have no idea how I should set it properly...
When using the calculated SliceNormal and some viewUp from the constant orientations, the view seems like it's correct, except for the camera. It's often positioned wrongly. For example, for these views to be correct, we'd need to: flip the camera horizontally / move the camera to the back of the image instead of the front etc. Cornerstone doesn't give us functionalites like that tho, and even if it did, it'd be hard to dynamically decide what camera operations need to be done to make the specific view correct.

So my question is, how to set the orientation for volumes with custom ImageOrientationPatient metadata?

Volume rendered with Orientation Coronal

Volume rendered with Orientation Axial

Volume rendered with custom orientation - sliceNormal = cross(imageOrientation.slice(0, 3), viewUp = [0, 0, 1]
(in this case the problem is left=right)

Used DICOM files - series-000001.zip

[sedghi]

Hmm, I looked into your data and as it appears it is oblique in plane, I don't think there is anything wrong with Cornerstone Axial, Sagittal, Coronal planes, and since in your last video you cross producted the vectors you can see the reformated view

[LittleURF]

In that case, what should one do to achieve the proper oblique orientation?
Meaning, the one that's shown when the example data is opened in fe. the ohif viewer https://v3-demo.ohif.org/local.
(the difference compared to my last video in the initial comment is: left = right)

[sedghi]

OHIF is using the same Cornerstone3D engine, you can try drag and dropping here. So is the problem you are having the camera orientation or the left and right? I'm confused

[LittleURF]

Well in the end my problem for this example is the left and right, but I believe it's caused by the orientation. With different Dicoms I'm having different camera problems(other than left=right), so I'm not looking only for a 'left=right' solution, but generally for a 'how should the volume/viewport be prepared so that I can view the Dicoms how other viewers do it'.

I've earlier tried the DICOMs with the cornerstone uploader you linked, but it renders a stack instead of a volume, and in that case the problem doesn't appear. I suppose that could be a thing to investigate

[sedghi]

Hmmm, so our volume camera is different than our stack camera, which is interesting to know. Just before I look deeper into this, can you check other dataset you have (not just this Knee MRI), to see if that is persistent bug?

'how should the volume/viewport be prepared so that I can view the Dicoms how other viewers do it'.

I guess if there is a bug, then we need to fix it, but generally you need to get the camera from the viewport viewport.getActiveCamera (I guess) and manipulate its parameters there. (You can flip there)

[LittleURF]

Sure, I've previously tested quite a few Dicom series, and for now every serie with unusual ImageOrientation metadata is wrong.
Examples:
1
2

[LittleURF]

Are there any updates on this matter?

[sedghi]

Hey, I was thinking have you tried setting the orthogonal viewup and slice normal similar to this? The volumeViewport support any arbitrary orientation

[jjustinm4]

@LittleURF @sedghi im struggling to get a series of dicomp10 files rendered as a volume . im very new to cornerstonejs and i couldnt get much help .

this is the code ive written so far . im able to locally load files and get 1 slice rendered in 3d but cant load whol image . A little help will be higly appreacited thanks

import {
CONSTANTS,
Enums,
getRenderingEngine,
RenderingEngine,
setVolumesForViewports,
Types,
volumeLoader,
} from '@cornerstonejs/core';
import * as cornerstoneDICOMImageLoader from '@cornerstonejs/dicom-image-loader';
import * as cornerstoneTools from '@cornerstonejs/tools';
import {
addButtonToToolbar,
addDropdownToToolbar,
addManipulationBindings,
createImageIdsAndCacheMetaData,
initDemo,
setTitleAndDescription,
} from '../../../../utils/demo/helpers';
import dicomParser from 'dicom-parser';
import {
convertMultiframeImageIds,
prefetchMetadataInformation,
} from '../../../../utils/demo/helpers/convertMultiframeImageIds';

cornerstoneDICOMImageLoader.external.dicomParser = dicomParser;
const { ToolGroupManager, Enums: csToolsEnums } = cornerstoneTools;

const { ViewportType } = Enums;

// Define a unique id for the volume
let renderingEngine;
const volumeName = 'CT_VOLUME_ID';
const volumeLoaderScheme = 'cornerstoneStreamingImageVolume';
const volumeId = ${volumeLoaderScheme}:${volumeName};
const renderingEngineId = 'myRenderingEngine';
const viewportId = '3D_VIEWPORT';

// ======== Set up page ======== //
setTitleAndDescription(
'3D Volume Rendering',
'Here we demonstrate how to 3D render a volume.'
);

let change = false;
const size = '512px';
const content = document.getElementById('content');
const viewportGrid = document.createElement('div');

viewportGrid.style.display = 'flex';
viewportGrid.style.flexDirection = 'row';

const element1 = document.createElement('div');
element1.oncontextmenu = () => false;

element1.style.width = size;
element1.style.height = size;

viewportGrid.appendChild(element1);
content.appendChild(viewportGrid);

const instructions = document.createElement('p');
instructions.innerText = 'Click the image to rotate it.';
content.append(instructions);

addButtonToToolbar({
title: 'Apply random rotation',
onClick: () => {
const renderingEngine = getRenderingEngine(renderingEngineId);
const viewport = renderingEngine.getViewport(viewportId);
viewport.setProperties({ rotation: Math.random() * 360 });
viewport.render();
},
});

addDropdownToToolbar({
options: {
values: CONSTANTS.VIEWPORT_PRESETS.map((preset) => preset.name),
defaultValue: 'CT-Bone',
},
onSelectedValueChange: (presetName) => {
const renderingEngine = getRenderingEngine(renderingEngineId);
const viewport = renderingEngine.getViewport(viewportId);
viewport.setProperties({ preset: presetName });
viewport.render();
},
});
let imageIds = [];
addButtonToToolbar({
title: 'Load images',
onClick: async () => {
change = true;
const fileInput = document.createElement('input');
fileInput.type = 'file';
fileInput.multiple = true;
fileInput.accept = '';

fileInput.addEventListener('change', async () => {
  const files = fileInput.files;

  for (let i = 0; i < files.length; i++) {
    const imageId = cornerstoneDICOMImageLoader.wadouri.fileManager.add(
      files[i]
    );
    imageIds.push(imageId);
  }
  if (imageIds.length !== 0) {
    imageIds = await convertMultiframeImageIds(imageIds);
    await prefetchMetadataInformation(imageIds);
    console.log('Generated Image IDs:', imageIds);
    console.log('after getting ids ', imageIds.length);
    if (imageIds.length !== 0) {
      console.log('calling run');
      run();
    }
  }
});

fileInput.click();

},
});

let viewport;

/**

• Runs the demo
  */
  async function run() {
  await initDemo();

const toolGroupId = 'TOOL_GROUP_ID';
const toolGroup = ToolGroupManager.createToolGroup(toolGroupId);

addManipulationBindings(toolGroup, {
is3DViewport: true,
});

if (!change) {
imageIds = await createImageIdsAndCacheMetaData({
StudyInstanceUID:
'1.3.6.1.4.1.14519.5.2.1.7009.2403.871108593056125491804754960339',
SeriesInstanceUID:
'1.3.6.1.4.1.14519.5.2.1.7009.2403.367700692008930469189923116409',
wadoRsRoot: 'https://domvja9iplmyu.cloudfront.net/dicomweb',
});
}

renderingEngine = new RenderingEngine(renderingEngineId);

const viewportInputArray = [
{
viewportId: viewportId,
type: ViewportType.VOLUME_3D,
element: element1,
defaultOptions: {
orientation: Enums.OrientationAxis.CORONAL,
background: CONSTANTS.BACKGROUND_COLORS.slicer3D,
},
},
];

renderingEngine.setViewports(viewportInputArray);
console.log('Image IDs length in run:', imageIds.length);
if (imageIds.length !== 0) {
toolGroup.addViewport(viewportId, renderingEngineId);

try {
  const volume = await volumeLoader.createAndCacheVolume(volumeId, {
    imageIds,
  });
  await volume.load();
  viewport = renderingEngine.getViewport(viewportId);

  await setVolumesForViewports(
    renderingEngine,
    [{ volumeId }],
    [viewportId]
  );
  viewport.setProperties({ preset: 'CT-Bone' });
  console.log('rendering successful');
  viewport.render();
} catch (error) {
  console.error('Error:', error);
}

}
}