Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(measurements): The image stack sync tool fails to work on non-FOR instances and hangs the browser #642

Merged
merged 12 commits into from
Sep 7, 2023
Merged
Original file line number Diff line number Diff line change
@@ -1,9 +1,15 @@
import { vec3, mat4 } from 'gl-matrix';
import { IStackViewport } from '../types';
import { StackViewport } from '../RenderingEngine';
import spatialRegistrationMetadataProvider from './spatialRegistrationMetadataProvider';
import { metaData } from '..';
import isEqual from './isEqual';

/**
* Defines the allowed difference as a percent between the unit normals before
* two planes are considered not coplanar. Since this value is small compared
* to the unit lenght, this value is approximately the angular difference, measured
* in radians. That is, allow about a 3 degrees variation.
*/
const ALLOWED_DELTA = 0.05;

/**
* It calculates the registration matrix between two viewports (currently only
Expand All @@ -21,40 +27,28 @@ function calculateViewportsSpatialRegistration(
viewport1: IStackViewport,
viewport2: IStackViewport
): void {
if (
!(viewport1 instanceof StackViewport) ||
!(viewport2 instanceof StackViewport)
) {
throw new Error(
'calculateViewportsSpatialRegistration: Both viewports must be StackViewports, volume viewports are not supported yet'
);
}

const isSameFrameOfReference =
viewport1.getFrameOfReferenceUID() === viewport2.getFrameOfReferenceUID();

if (isSameFrameOfReference) {
return;
}

const imageId1 = viewport1.getCurrentImageId();
const imageId2 = viewport2.getCurrentImageId();

const imagePlaneModule1 = metaData.get('imagePlaneModule', imageId1);
const imagePlaneModule2 = metaData.get('imagePlaneModule', imageId2);

const isSameImagePlane =
imagePlaneModule1 &&
imagePlaneModule2 &&
isEqual(
imagePlaneModule1.imageOrientationPatient,
imagePlaneModule2.imageOrientationPatient
);
if (!imagePlaneModule1 || !imagePlaneModule2) {
console.log('Viewport spatial registration requires image plane module');
return;
}
const { imageOrientationPatient: iop2 } = imagePlaneModule2;
const isSameImagePlane = imagePlaneModule1.imageOrientationPatient.every(
(v, i) => Math.abs(v - iop2[i]) < ALLOWED_DELTA
);

if (!isSameImagePlane) {
throw new Error(
'Viewport spatial registration only supported for same orientation (hence translation only) for now'
console.log(
'Viewport spatial registration only supported for same orientation (hence translation only) for now',
imagePlaneModule1?.imageOrientationPatient,
imagePlaneModule2?.imageOrientationPatient
);
return;
}

const imagePositionPatient1 = imagePlaneModule1.imagePositionPatient;
Expand All @@ -67,7 +61,6 @@ function calculateViewportsSpatialRegistration(
);

const mat = mat4.fromTranslation(mat4.create(), translation);

spatialRegistrationMetadataProvider.add([viewport1.id, viewport2.id], mat);
}

Expand Down
24 changes: 16 additions & 8 deletions packages/tools/src/store/SynchronizerManager/Synchronizer.ts
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file fix the browser hanging issue.

Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,7 @@ class Synchronizer {
}

this._ignoreFiredEvents = true;
const promises = [];
try {
for (let i = 0; i < this._targetViewports.length; i++) {
const targetViewport = this._targetViewports[i];
Expand All @@ -214,19 +215,26 @@ class Synchronizer {
if (targetIsSource) {
continue;
}

this._eventHandler(
this,
sourceViewport,
targetViewport,
sourceEvent,
this._options
promises.push(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how to avoid this synchronization - the basic synchronizer design relies on being able to turn off secondary notifications so that only a single change applies, and if this doesn't use async, then you get an event from the target viewport, which changs the source viewport, which sends an event etc.

this._eventHandler(
this,
sourceViewport,
targetViewport,
sourceEvent,
this._options
)
);
}
} catch (ex) {
console.warn(`Synchronizer, for: ${this._eventName}`, ex);
} finally {
this._ignoreFiredEvents = false;
if (promises.length) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wayfarer3130 and I reviewed this on a call; this is causing issues in the TVM mode. This is also an important fix that was making the browser hang so we want to keep it.
Mentioning here just so we know we need to fix it before merging

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the viewport status instead of making this async

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be ok in the TVM mode now, at least I can't reproduce any hangs. I believe the issue was caused by multiple initial renders causing a loop, and the existing OHIF changes to prevent extra renders resolved the base issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mateusfreira can you try again, It seems to work with tmtv now

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mateusfreira - you have to try it with the merge update branch of FlexView, as it was changes outside of CS3D that fixed the hang issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mateusfreira can you try and let me know?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sedghi and @wayfarer3130 I confirmed it is working as expected
Screenshot 2023-09-07 at 12 27 46 PM

Promise.allSettled(promises).then(() => {
this._ignoreFiredEvents = false;
});
} else {
this._ignoreFiredEvents = false;
}
}
}

Expand Down
131 changes: 63 additions & 68 deletions packages/tools/src/synchronizers/callbacks/stackImageSyncCallback.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { vec3 } from 'gl-matrix';
import { vec3, mat4 } from 'gl-matrix';
import {
getRenderingEngine,
Types,
Expand All @@ -8,28 +8,27 @@ import {
import { Synchronizer } from '../../store';
import { jumpToSlice } from '../../utilities';
import areViewportsCoplanar from './areViewportsCoplanar ';

const getSpatialRegistration = (targetId, sourceId) =>
utilities.spatialRegistrationMetadataProvider.get(
'spatialRegistrationModule',
[targetId, sourceId]
);

/**
* Synchronizer callback to synchronize the source viewport image to the
* target viewports closest image in its stack. There are two scenarios
* target viewports closest image in its stack.
*
* 1) viewports are in the same frameOfReferenceUID then we can use the
* absolute imagePositionPatient for the source viewport's current image
* and set the target viewport's image to the closest image in its stack
* (which might have different slice thickness so cannot use slice number)
* This synchronizer does a setup (which can already be predefined as required)
* to register the target and soruce viewports. The registration will default
* to the identity registration if the same FOR is present in both viewports,
* unless the option `useInitialPosition` is set in the target viewport.
*
* 2) viewports have different frameOfReferenceUIDs then we look inside the
* registrationMetadataProvider to check if there is a corresponding matrix
* for mapping between the source and target viewport if so it is used to
* and is applied to the imagePositionPatient of the source viewport's to
* get the imagePositionPatient of the target viewport's closest image in
* its stack.
* Note for 2) The consuming apps using Cornerstone3D (OHIF, etc) are responsible
* to provide such data in the registrationMetadataProvider. This can be done
* The consuming apps using Cornerstone3D (OHIF, etc) MAY provide such data in
* the registrationMetadataProvider to override the data here. This can be done
* by various methods 1) Using spatialRegistrationModule inside dicom 2) assuming
* the user has actually manually scrolled the target viewport to the correct
* slice before initiating the synchronization 3) using some other method
* But overall, the consuming app is responsible for providing the data.
*
*
* @param synchronizerInstance - The Instance of the Synchronizer
* @param sourceViewport - The list of IDs defining the source viewport.
Expand All @@ -53,13 +52,16 @@ export default async function stackImageSyncCallback(
sourceViewport.viewportId
) as Types.IStackViewport;

const options = synchronizerInstance.getOptions(targetViewport.viewportId);

if (options?.disabled) {
return;
}

const tViewport = renderingEngine.getViewport(
targetViewport.viewportId
) as Types.IStackViewport;

const frameOfReferenceUID1 = sViewport.getFrameOfReferenceUID();
const frameOfReferenceUID2 = tViewport.getFrameOfReferenceUID();

const imageId1 = sViewport.getCurrentImageId();
const imagePlaneModule1 = metaData.get('imagePlaneModule', imageId1);
const sourceImagePositionPatient = imagePlaneModule1.imagePositionPatient;
Expand All @@ -70,69 +72,62 @@ export default async function stackImageSyncCallback(
return;
}

if (frameOfReferenceUID1 === frameOfReferenceUID2) {
// if frames of references are the same we can use the absolute
// imagePositionPatient to find the closest image in the target viewport's stack
const closestImageIdIndex = _getClosestImageIdIndex(
sourceImagePositionPatient,
targetImageIds
);
// if the frame of reference is different we need to use the registrationMetadataProvider
// and add that to the imagePositionPatient of the source viewport to get the
// imagePositionPatient of the target viewport's closest image in its stack
let registrationMatrixMat4 = getSpatialRegistration(
targetViewport.viewportId,
sourceViewport.viewportId
);

if (!registrationMatrixMat4) {
const frameOfReferenceUID1 = sViewport.getFrameOfReferenceUID();
const frameOfReferenceUID2 = tViewport.getFrameOfReferenceUID();
if (
closestImageIdIndex.index !== -1 &&
tViewport.getCurrentImageIdIndex() !== closestImageIdIndex.index
frameOfReferenceUID1 === frameOfReferenceUID2 &&
options.useInitialPosition !== false
) {
// await tViewport.setImageIdIndex(closestImageIdIndex.index);
await jumpToSlice(tViewport.element, {
imageIndex: closestImageIdIndex.index,
});

return;
}
} else {
// if the frame of reference is different we need to use the registrationMetadataProvider
// and add that to the imagePositionPatient of the source viewport to get the
// imagePositionPatient of the target viewport's closest image in its stack
const registrationMatrixMat4 =
utilities.spatialRegistrationMetadataProvider.get(
'spatialRegistrationModule',
[targetViewport.viewportId, sourceViewport.viewportId]
registrationMatrixMat4 = mat4.identity(mat4.create());
} else {
utilities.calculateViewportsSpatialRegistration(sViewport, tViewport);
registrationMatrixMat4 = getSpatialRegistration(
targetViewport.viewportId,
sourceViewport.viewportId
);

}
if (!registrationMatrixMat4) {
throw new Error(
`No registration matrix found for sourceViewport: ${sourceViewport.viewportId} and targetViewport: ${targetViewport.viewportId}, viewports with different frameOfReferenceUIDs must have a registration matrix in the registrationMetadataProvider. Use calculateViewportsRegistrationMatrix to calculate the matrix.`
);
return;
}
}

// apply the registration matrix to the source viewport's imagePositionPatient
// to get the target viewport's imagePositionPatient
const targetImagePositionPatientWithRegistrationMatrix = vec3.transformMat4(
vec3.create(),
sourceImagePositionPatient,
registrationMatrixMat4
);
// apply the registration matrix to the source viewport's imagePositionPatient
// to get the target viewport's imagePositionPatient
const targetImagePositionPatientWithRegistrationMatrix = vec3.transformMat4(
vec3.create(),
sourceImagePositionPatient,
registrationMatrixMat4
);

// find the closest image in the target viewport's stack to the
// targetImagePositionPatientWithRegistrationMatrix
const closestImageIdIndex2 = _getClosestImageIdIndex(
targetImagePositionPatientWithRegistrationMatrix,
targetImageIds
);
// find the closest image in the target viewport's stack to the
// targetImagePositionPatientWithRegistrationMatrix
const closestImageIdIndex2 = _getClosestImageIdIndex(
targetImagePositionPatientWithRegistrationMatrix,
targetImageIds
);

if (
closestImageIdIndex2.index !== -1 &&
tViewport.getCurrentImageIdIndex() !== closestImageIdIndex2.index
) {
await jumpToSlice(tViewport.element, {
imageIndex: closestImageIdIndex2.index,
});
}
if (
closestImageIdIndex2.index !== -1 &&
tViewport.getCurrentImageIdIndex() !== closestImageIdIndex2.index
) {
await jumpToSlice(tViewport.element, {
imageIndex: closestImageIdIndex2.index,
});
}
}

function _getClosestImageIdIndex(targetPoint, imageIds) {
// todo: this does not assume orientation yet, but that can be added later
// todo: handle multiframe images
return imageIds.reduce(
(closestImageIdIndex, imageId, index) => {
const { imagePositionPatient } = metaData.get(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ export default function voiSyncCallback(
voiRange: range,
};

if (options.syncInvertState && invertStateChanged) {
if (options?.syncInvertState && invertStateChanged) {
tProperties.invert = invert;
}

Expand Down