Assertion failure in S_is_utf8_invariant_string_loc

[p5pRT]

Migrated from rt.perl.org#133882 (status was 'resolved')

Searchable as RT133882$

[p5pRT]

From @dur-randir

Created by @dur-randir

While fuzzing perl v5.29.8-21-gde59f38ed9 built with afl and run
under libdislocator, I found the following program (text also attached
to this message)

00000000 73 2f 2f 30 30 30 30 30 30 30 30 30 30 30 66 6f |s//00000000000fo|
00000010 72 6d 61 74 20 20 20 20 20 20 20 20 20 20 20 20 |rmat |
00000020 5c 30 20 20 20 20 20 20 20 20 20 20 27 30 30 30 |\0 '000|
00000030 30 30 30 30 5c 78 7b 38 30 30 7d 2f 3b 65 76 61 |0000\x{800}/;eva|
00000040 6c |l|

to cause an assertion failure​:

perl​: ./inline.h​:470​: _Bool S_is_utf8_invariant_string_loc(const U8
*const, STRLEN, const U8 **)​: Assertion `*ep >= s && *ep < send'
failed.

GDB stack trace is following​:

#0 __GI_raise (sig=sig@​entry=6) at ../sysdeps/unix/sysv/linux/raise.c​:50
#1 0x00007ffff7c25535 in __GI_abort () at abort.c​:79
#2 0x00007ffff7c2540f in __assert_fail_base (fmt=0x7ffff7d87ee0
"%s%s%s​:%u​: %s%sAssertion `%s' failed.\n%n",
  assertion=0x5555559212a5 "*ep >= s && *ep < send",
file=0x555555920d20 "inline.h", line=470, function=<optimized out>) at
assert.c​:92
#3 0x00007ffff7c330f2 in __GI___assert_fail (assertion=0x5555559212a5
"*ep >= s && *ep < send", file=0x555555920d20 "inline.h", line=470,
  function=0x555555946e60 <__PRETTY_FUNCTION__.17334>
"S_is_utf8_invariant_string_loc") at assert.c​:101
#4 0x000055555560b800 in S_is_utf8_invariant_string_loc
(s=0x555555b50990 "ࠀ\n;", len=18446744073709551608, ep=0x7fffffffcdc0)
at inline.h​:470
#5 0x000055555560ba11 in Perl_is_utf8_string_loclen (s=0x555555b50990
"ࠀ\n;", len=18446744073709551608, ep=0x0, el=0x0) at inline.h​:978
#6 0x000055555560b980 in S_is_utf8_non_invariant_string
(s=0x555555b50988 "'0000000ࠀ\n;", len=0) at inline.h​:770
#7 0x0000555555613f68 in S_newSV_maybe_utf8 (start=0x555555b50988
"'0000000ࠀ\n;", len=0) at toke.c​:2068
#8 0x0000555555641cc0 in Perl_yylex () at toke.c​:8698
#9 0x00005555556590dc in Perl_yyparse (gramtype=258) at perly.c​:340
#10 0x00005555558244cd in S_doeval_compile (gimme=1 '\001',
outside=0x555555b268d8, seq=4294967246, hh=0x0) at pp_ctl.c​:3501
#11 0x000055555582c268 in Perl_pp_entereval () at pp_ctl.c​:4477
#12 0x00005555556f692d in Perl_runops_debug () at dump.c​:2537
#13 0x00005555555da124 in S_run_body (oldscope=1) at perl.c​:2692
#14 0x00005555555d96a2 in perl_run (my_perl=0x555555b24260) at perl.c​:2615
#15 0x000055555558e12e in main (argc=2, argv=0x7fffffffe1e8,
env=0x7fffffffe200) at perlmain.c​:127

This is a regression between 5.26 and 5.28, bisect points to

commit 1d2af57
Author​: Karl Williamson <khw@​cpan.org>
Date​: Sat Jan 13 15​:40​:34 2018 -0700

  Avoid some branches

  This replaces some looping with branchless code in two places​: looking
  for the first UTF-8 variant byte in a string (which is used under
  several circumstances), and looking for an ASCII or non-ASCII character
  during pattern matching.

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl 5.29.9:

Configured by dur-randir at Wed Feb 27 14:51:01 MSK 2019.

Summary of my perl5 (revision 5 version 29 subversion 9) configuration:
  Commit id: c1e47bad34ce1d9c84ed57c9b8978bcbd5a02e98
  Platform:
    osname=darwin
    osvers=13.4.0
    archname=darwin-thread-multi-2level
    uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0:
mon jan 11 18:17:34 pst 2016; root:xnu-2422.115.15~1release_x86_64
x86_64 '
    config_args='-de -Dusedevel -DDEBUGGING -Dusethreads'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -DPERL_USE_SAFE_PUTENV'
    optimize='-O3 -g'
    cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=
    so=dylib
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined
dynamic_lookup -L/usr/local/lib -fstack-protector'



@INC for perl 5.29.9:
    lib
    /usr/local/lib/perl5/site_perl/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/site_perl/5.29.9
    /usr/local/lib/perl5/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/5.29.9


Environment for perl 5.29.9:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/dur-randir
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin
    PERLBREW_HOME=/Users/dur-randir/.perlbrew
    PERLBREW_MANPATH=/Users/dur-randir/perlbrew/perls/perl-5.22.1/man
    PERLBREW_PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin
    PERLBREW_PERL=perl-5.22.1
    PERLBREW_ROOT=/Users/dur-randir/perlbrew
    PERLBREW_SHELLRC_VERSION=0.84
    PERLBREW_VERSION=0.84
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/zsh

[p5pRT]

From @dur-randir

0055

[p5pRT]

@khwilliamson - Status changed from 'new' to 'open'

[p5pRT]

From @khwilliamson

Fixed by
commit 5346364
Author​: Karl Williamson <khw@​cpan.org>
Date​: Thu Mar 7 13​:44​:34 2019 -0700

  PATCH​: [perl #133882] Assertion failure
 
  The asserts in this routine were doing there job. It was called
  inappropriately, with len set to 0, which means for it that it's
  supposed to calculate the length by using strlen(). But, len being 0
  here meant that the input was empty. When run under valgrind, errors
  would also show up.
 
  This function was being called to see if the string had any characters
  that varied depending on if it is UTF-8 or not. Since we know that the
  answer is no if the length is 0, we simply don't call this function
  then.

--
Karl Williamson

[p5pRT]

@khwilliamson - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0, this and 160 other issues have been
resolved.

Perl 5.30.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists, feel free to reopen this ticket.

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'