CVE-2016-9112 FPE(Floating Point Exception) in lib/openjp2/pi.c:523

[YangY-Xiao]

Vulnerability
openjpeg FPE in pi.c:523

Version
openjpeg-2.1.2

Address Sanitizer Output
==11185==ERROR: AddressSanitizer: FPE on unknown address 0xb769d99f (pc 0xb769d99f bp 0xbfcb50c8 sp 0xbfcb5050 T0)
#0 0xb769d99e (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x7599e)
#1 0xb76ab215 (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x83215)
#2 0xb76bbd71 (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x93d71)
#3 0xb76bbb63 (/home//openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x93b63)
#4 0xb764e2bf (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x262bf)
#5 0xb766a2d6 (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x422d6)
#6 0xb764853a (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x2053a)
#7 0xb765496d (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x2c96d)
#8 0xb768ab5a (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x62b5a)
#9 0x81368f7 (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/opj_decompress+0x81368f7)
#10 0xb73d8636 (/lib/i386-linux-gnu/libc.so.6+0x18636)
#11 0x805f327 (/home/y/openjpeg/openjpeg-2.1.2/build-clang/bin/opj_decompress+0x805f327)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: FPE (/home/yang/openjpeg/openjpeg-2.1.2/build-clang/bin/libopenjp2.so.7+0x7599e)

GDB information
Program received signal SIGFPE, Arithmetic exception.
0xb7fb62ed in opj_pi_next_cprl (pi=0x807dfb8) at /home/yang/openjpeg/openjpeg-2.1.2/src/lib/openjp2/pi.c:523
523 if (!((pi->y % (OPJ_INT32)(comp->dy << rpy) == 0) || ((pi->y == pi->ty0) && ((try0 << levelno) % (1 << rpy))))){
(gdb) p comp->dy
$1 = 8
(gdb) p rpy
$2 = 31
(gdb) p comp->dy << rpy
$3 = 0

Analysis
In pi.c (OPJ_INT32)(comp->dy << rpy) = 0, and program does not check the value of (OPJ_INT32)(comp->dy << rpy).

Poc
Contact me if you need Poc file at [email protected]

[YangY-Xiao]

The same bug can occur in openjp2\pi.c(line 363, line 366, line 444, line 447, line 526); openjp3d\pi.c(line 218, line 221, line 224, line 312, line 315, line 318, line 403, line 406, line 409); openmj2\pi.c(line 218, line 221, line 299, line 302, line 378, line 381).

[1ucian0]

Please, refer to this issue as CVE-2016-9112

[rouault]

Likely fixed per d27ccf0 and later commits in pi.c