divide by zero, perhaps multiplication overflow

[setharnold]

openjpeg/src/lib/openmj2/mj2_convert.c

Line 53 in da5e897

frame_size = (long)(tk->w * tk->h * (1.0 + (double) 2 / (double)(

Hello,

This code mixes doubles and integers in an odd way; I do not know if the tk->w * tk->h multiplication can overflow the integer values or not. The tk->CbCr_subsampling_dx * tk->CbCr_subsampling_dy step is probably carried out as integers, and thus a 0 result here is possible.

Floating point is difficult enough already without using integers in the same expressions. I'd feel much more confident about the safety of this code if it were re-written to avoid mixing types and was prepared to handle the potential integer overflows in the multiplications.

   frame_size = (long)(tk->w * tk->h * (1.0 + (double) 2 / (double)(
            tk->CbCr_subsampling_dx *
            tk->CbCr_subsampling_dy)));  /* Calculate frame size */
frame_size *= prec_size;

Thanks