New perls #128

khwilliamson · 2018-02-07T19:58:37Z

No description provided.

This correctly indents things in blocks, and removes trailing space

This documents process_utf8(), and adds another helpful comment

On modern perls, there is a definition for the REPLACEMENT CHARACTER UTF-8 string. Use this if available, as it is portable to EBCDIC, and this one isn't.

The value for this condition is known before the loop, so move it outside the loop.

Perl 5.26 introduced infrastructure in the core that can be used by Encode to check UTF-8 stream validity much faster than before. This commit replaces the current scheme for checking UTF-8 validity if the infrastructure is availabe, by one in which normal processing doesn't require having to decode the UTF-8 into code points. The copying of characters individually from the input to the output is changed to be a single operation for each entire span of valid input at once. Thus in the normal case, what ends up happening is a tight loop to check the validity, and then a memmove of the entire input to the output, then return. If an error is found, it copies all the valid input before the error, then handles the character in error, then positions to the next input position, and repeats the whole process starting from there. Thus, this does not need to know about the intricacies of UTF-8 malformations, relying on the core to handle this. There are currently some problems with Encode on EBCDIC platforms. The infrastructure is known to correctly work there, so I'm hopeful this will solve these portability issues. This also fixes a TODO test when Encode is run on modern perls. Below are some benchmarking results for decode(), when called on a long string of valid bytes which are Unicode characters consisting of 1, 2, 3, and 4 bytes. Key: Ir Instruction read Dr Data read Dw Data write COND conditional branches The numbers represent raw counts per loop iteration. 'Inf' doesn't mean infinity actually, but the number stored was considered too big to print by the program that created this data. The Ratio is the improvement of 'this' change vs. blead, and is valid even if the raw counts are shown as 'Inf'. The higher the ratio, the better. The most important number is the COND one, as each one of these interrupt the CPU instruction pipeline. The best result below is for 2-byte characters. It indicates that this patch works in less than half the time it currently takes, i.e. the speed of this routine more than doubles with this patch on 2-byte characters, and is at least 30% faster in all cases. decoding 1-byte blead this Ratio % -------- -------- ------- Ir 918975.0 319752.0 287.4 Dr 305933.0 218330.0 140.1 Dw 203137.0 203138.0 100.0 COND 302803.5 227987.0 132.8 decoding 2-byte blead this Ratio % -------- -------- ------- Ir Inf Inf 413.0 Dr Inf 805823.0 224.1 Dw Inf 403139.0 248.8 COND Inf 902996.0 266.1 decoding 3-byte blead this Ratio % -------- -------- ------- Ir Inf Inf 246.5 Dr Inf Inf 181.4 Dw Inf 603139.0 182.9 COND Inf Inf 139.9 decoding 4-byte blead this Ratio % -------- -------- ------- Ir Inf Inf 258.7 Dr Inf Inf 156.9 Dw Inf 803139.0 149.8 COND Inf Inf 142.8

This makes it much more legible

This should be using the 'uvchr' functions, instead of the 'uvuni' ones. These are actually identical on non-EBCDIC machines, so there will be no change of behavior except on EBCDIC, where the 'uvchr' is smart enough to handle the differences.

Early perl versions didn't have this defined. Also the default changed as to what is acceptable vs. not, so if this isn't defined, define a symbol to deal with that default change.

dankogai · 2018-02-08T00:31:34Z

Thank you. VERSION++ follows immediately.

karenetheridge · 2018-02-13T19:46:55Z

These changes cause breakages in YAML::Tiny (and CPAN::Meta::YAML, a dual-life module) on perl 5.26.1 and earlier: https://rt.cpan.org/Ticket/Display.html?id=124399

Is Encode using a feature that is only available in blead? Should ppport.h be used to provide equivalent functionality in earlier perls?

dankogai · 2018-02-14T01:42:02Z

I doubt it. travis w/ following config is happy.

language: perl
perl:
  - "5.26"
  - "5.24"
  - "5.22"
  - "5.20"
  - "5.18"
  - "5.16"
  - "5.14"
  - "5.12"
  - "5.10"
  - "5.8"

https://travis-ci.org/dankogai/p5-encode

Also I could not reproduce it on any of my environments.

karenetheridge · 2018-02-14T01:49:47Z

Encode's tests work fine for me too -- apparently they are not testing the change that is tickling YAML::Tiny's tests.

karenetheridge · 2018-02-19T21:17:58Z

PR #130 fixes this (not yet released).

$Revision: 2.97 $ $Date: 2018/02/21 12:14:33 $ ! Encode.xs Pulled: New perls that fixes dankogai/p5-encode#129 https://rt.cpan.org/Ticket/Display.html?id=124399 dankogai/p5-encode#130 2.96 2018/02/11 05:35:26 ! Encode.pm encoding.pm Unicode/Unicode.pm VERSION++ to make bleadperl happy <CADED=K4v5WQ3R7+aTu1xV4q2RcZFT=jriZubqfrHe7PZrgRmDA@mail.gmail.com> 2.95 2018/02/08 00:26:15 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.pm Unicode/Unicode.xs encengine.c Pulled: new perls dankogai/p5-encode#128

* Clean-up whitespace * Add ptest dependencies License-Update: Only use license lines from META.json Upstream release notes: 3.01 $ $Date: 2019/03/13 00:26:18 $ ! Encode.xs patched: Warning: Use of uninitialized value in subroutine entry dankogai/p5-encode#139 (comment) ! Encode/encode.h Pulled: Fix compile error and warning dankogai/p5-encode#138 3.00 2019/01/31 ! Encode.pm VERSION bumped to 3.00 to make PAUSE happy 2.100 2019/01/31 04:26:40 ! Encode.xs MANIFEST + t/xml.t Pulled: Do not access SV* buffer if we have not called SvPV_force() dankogai/p5-encode#137 ! MANIFEST remove utf8messages.t which is already deleted from the repository. 2.99 2019/01/21 03:13:35 ! Unicode/Unicode.xs Unicode/Unicode.pm VERSION++'ed as Perl core needed. dankogai/p5-encode#136 ! encengine.c Pulled: protect do_encode from NULL dst dankogai/p5-encode#135 2.98 2018/04/22 09:02:00 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.xs encengine.c t/decode.t t/enc_eucjp.t t/utf8messages.t t/utf8warnings.t - t/utf8messages.t + t/utf32warnings.t Pulled: Introduce new Encode check flag Encode::ONLY_PRAGMA_WARNINGS dankogai/p5-encode#134 2.98 2018/04/22 09:02:00 ! t/truncated_utf8.t Resolved: RT125131: truncated_utf8.t TODO test pass in blead https://rt.cpan.org/Ticket/Display.html?id=125131 ! Encode.xs Pulled: Remove XS functions _bytes_to_utf8() and _utf8_to_bytes() dankogai/p5-encode#133 ! Unicode/Unicode.xs Pulled: Automatically compute length in attr() macro dankogai/p5-encode#132 ! Encode.xs Pulled: Fix compile warnings on 64bit MS VS2017 dankogai/p5-encode#131 2.97 2018/02/21 12:14:33 ! Encode.xs Pulled: New perls that fixes dankogai/p5-encode#129 https://rt.cpan.org/Ticket/Display.html?id=124399 dankogai/p5-encode#130 2.96 2018/02/11 05:35:26 ! Encode.pm encoding.pm Unicode/Unicode.pm VERSION++ to make bleadperl happy <CADED=K4v5WQ3R7+aTu1xV4q2RcZFT=jriZubqfrHe7PZrgRmDA@mail.gmail.com> 2.95 2018/02/08 00:26:15 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.pm Unicode/Unicode.xs encengine.c Pulled: new perls dankogai/p5-encode#128 Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Khem Raj <[email protected]>

Source: meta-openembedded MR: 00000 Type: Integration Disposition: Merged from meta-openembedded ChangeID: 1636d7d Description: * Clean-up whitespace * Add ptest dependencies License-Update: Only use license lines from META.json Upstream release notes: 3.01 $ $Date: 2019/03/13 00:26:18 $ ! Encode.xs patched: Warning: Use of uninitialized value in subroutine entry dankogai/p5-encode#139 (comment) ! Encode/encode.h Pulled: Fix compile error and warning dankogai/p5-encode#138 3.00 2019/01/31 ! Encode.pm VERSION bumped to 3.00 to make PAUSE happy 2.100 2019/01/31 04:26:40 ! Encode.xs MANIFEST + t/xml.t Pulled: Do not access SV* buffer if we have not called SvPV_force() dankogai/p5-encode#137 ! MANIFEST remove utf8messages.t which is already deleted from the repository. 2.99 2019/01/21 03:13:35 ! Unicode/Unicode.xs Unicode/Unicode.pm VERSION++'ed as Perl core needed. dankogai/p5-encode#136 ! encengine.c Pulled: protect do_encode from NULL dst dankogai/p5-encode#135 2.98 2018/04/22 09:02:00 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.xs encengine.c t/decode.t t/enc_eucjp.t t/utf8messages.t t/utf8warnings.t - t/utf8messages.t + t/utf32warnings.t Pulled: Introduce new Encode check flag Encode::ONLY_PRAGMA_WARNINGS dankogai/p5-encode#134 2.98 2018/04/22 09:02:00 ! t/truncated_utf8.t Resolved: RT125131: truncated_utf8.t TODO test pass in blead https://rt.cpan.org/Ticket/Display.html?id=125131 ! Encode.xs Pulled: Remove XS functions _bytes_to_utf8() and _utf8_to_bytes() dankogai/p5-encode#133 ! Unicode/Unicode.xs Pulled: Automatically compute length in attr() macro dankogai/p5-encode#132 ! Encode.xs Pulled: Fix compile warnings on 64bit MS VS2017 dankogai/p5-encode#131 2.97 2018/02/21 12:14:33 ! Encode.xs Pulled: New perls that fixes dankogai/p5-encode#129 https://rt.cpan.org/Ticket/Display.html?id=124399 dankogai/p5-encode#130 2.96 2018/02/11 05:35:26 ! Encode.pm encoding.pm Unicode/Unicode.pm VERSION++ to make bleadperl happy <CADED=K4v5WQ3R7+aTu1xV4q2RcZFT=jriZubqfrHe7PZrgRmDA@mail.gmail.com> 2.95 2018/02/08 00:26:15 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.pm Unicode/Unicode.xs encengine.c Pulled: new perls dankogai/p5-encode#128 Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Khem Raj <[email protected]> Signed-off-by: Jeremy Puhlman <[email protected]>

* Clean-up whitespace * Add ptest dependencies License-Update: Only use license lines from META.json Upstream release notes: 3.01 $ $Date: 2019/03/13 00:26:18 $ ! Encode.xs patched: Warning: Use of uninitialized value in subroutine entry dankogai/p5-encode#139 (comment) ! Encode/encode.h Pulled: Fix compile error and warning dankogai/p5-encode#138 3.00 2019/01/31 ! Encode.pm VERSION bumped to 3.00 to make PAUSE happy 2.100 2019/01/31 04:26:40 ! Encode.xs MANIFEST + t/xml.t Pulled: Do not access SV* buffer if we have not called SvPV_force() dankogai/p5-encode#137 ! MANIFEST remove utf8messages.t which is already deleted from the repository. 2.99 2019/01/21 03:13:35 ! Unicode/Unicode.xs Unicode/Unicode.pm VERSION++'ed as Perl core needed. dankogai/p5-encode#136 ! encengine.c Pulled: protect do_encode from NULL dst dankogai/p5-encode#135 2.98 2018/04/22 09:02:00 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.xs encengine.c t/decode.t t/enc_eucjp.t t/utf8messages.t t/utf8warnings.t - t/utf8messages.t + t/utf32warnings.t Pulled: Introduce new Encode check flag Encode::ONLY_PRAGMA_WARNINGS dankogai/p5-encode#134 2.98 2018/04/22 09:02:00 ! t/truncated_utf8.t Resolved: RT125131: truncated_utf8.t TODO test pass in blead https://rt.cpan.org/Ticket/Display.html?id=125131 ! Encode.xs Pulled: Remove XS functions _bytes_to_utf8() and _utf8_to_bytes() dankogai/p5-encode#133 ! Unicode/Unicode.xs Pulled: Automatically compute length in attr() macro dankogai/p5-encode#132 ! Encode.xs Pulled: Fix compile warnings on 64bit MS VS2017 dankogai/p5-encode#131 2.97 2018/02/21 12:14:33 ! Encode.xs Pulled: New perls that fixes dankogai/p5-encode#129 https://rt.cpan.org/Ticket/Display.html?id=124399 dankogai/p5-encode#130 2.96 2018/02/11 05:35:26 ! Encode.pm encoding.pm Unicode/Unicode.pm VERSION++ to make bleadperl happy <CADED=K4v5WQ3R7+aTu1xV4q2RcZFT=jriZubqfrHe7PZrgRmDA@mail.gmail.com> 2.95 2018/02/08 00:26:15 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.pm Unicode/Unicode.xs encengine.c Pulled: new perls dankogai/p5-encode#128 Signed-off-by: Tim Orling <[email protected]>

* Clean-up whitespace * Add ptest dependencies License-Update: Only use license lines from META.json Upstream release notes: 3.01 $ $Date: 2019/03/13 00:26:18 $ ! Encode.xs patched: Warning: Use of uninitialized value in subroutine entry dankogai/p5-encode#139 (comment) ! Encode/encode.h Pulled: Fix compile error and warning dankogai/p5-encode#138 3.00 2019/01/31 ! Encode.pm VERSION bumped to 3.00 to make PAUSE happy 2.100 2019/01/31 04:26:40 ! Encode.xs MANIFEST + t/xml.t Pulled: Do not access SV* buffer if we have not called SvPV_force() dankogai/p5-encode#137 ! MANIFEST remove utf8messages.t which is already deleted from the repository. 2.99 2019/01/21 03:13:35 ! Unicode/Unicode.xs Unicode/Unicode.pm VERSION++'ed as Perl core needed. dankogai/p5-encode#136 ! encengine.c Pulled: protect do_encode from NULL dst dankogai/p5-encode#135 2.98 2018/04/22 09:02:00 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.xs encengine.c t/decode.t t/enc_eucjp.t t/utf8messages.t t/utf8warnings.t - t/utf8messages.t + t/utf32warnings.t Pulled: Introduce new Encode check flag Encode::ONLY_PRAGMA_WARNINGS dankogai/p5-encode#134 2.98 2018/04/22 09:02:00 ! t/truncated_utf8.t Resolved: RT125131: truncated_utf8.t TODO test pass in blead https://rt.cpan.org/Ticket/Display.html?id=125131 ! Encode.xs Pulled: Remove XS functions _bytes_to_utf8() and _utf8_to_bytes() dankogai/p5-encode#133 ! Unicode/Unicode.xs Pulled: Automatically compute length in attr() macro dankogai/p5-encode#132 ! Encode.xs Pulled: Fix compile warnings on 64bit MS VS2017 dankogai/p5-encode#131 2.97 2018/02/21 12:14:33 ! Encode.xs Pulled: New perls that fixes dankogai/p5-encode#129 https://rt.cpan.org/Ticket/Display.html?id=124399 dankogai/p5-encode#130 2.96 2018/02/11 05:35:26 ! Encode.pm encoding.pm Unicode/Unicode.pm VERSION++ to make bleadperl happy <CADED=K4v5WQ3R7+aTu1xV4q2RcZFT=jriZubqfrHe7PZrgRmDA@mail.gmail.com> 2.95 2018/02/08 00:26:15 ! Encode.pm Encode.xs Encode/encode.h Unicode/Unicode.pm Unicode/Unicode.xs encengine.c Pulled: new perls dankogai/p5-encode#128 Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Khem Raj <[email protected]>

khwilliamson added 8 commits February 7, 2018 12:55

Encode: White space only

f68fc65

This correctly indents things in blocks, and removes trailing space

Encode: Add comments

e638b6c

This documents process_utf8(), and adds another helpful comment

Encode/encode.h: Use system REPLACEMENT char if available

daf506d

On modern perls, there is a definition for the REPLACEMENT CHARACTER UTF-8 string. Use this if available, as it is portable to EBCDIC, and this one isn't.

Encode/Encode.xs: Pull condition out of loop

656b360

The value for this condition is known before the loop, so move it outside the loop.

encengine.c: Properly indent code within blocks

4dff655

This makes it much more legible

Unicode.xs: Fix EBCDIC issues

5e1acbc

This should be using the 'uvchr' functions, instead of the 'uvuni' ones. These are actually identical on non-EBCDIC machines, so there will be no change of behavior except on EBCDIC, where the 'uvchr' is smart enough to handle the differences.

Make sure UTF8_DISALLOW_ILLEGAL_INTERCHANGE is defined

1d86b6a

Early perl versions didn't have this defined. Also the default changed as to what is acceptable vs. not, so if this isn't defined, define a symbol to deal with that default change.

dankogai merged commit fc742bc into dankogai:master Feb 8, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New perls #128

New perls #128

khwilliamson commented Feb 7, 2018

dankogai commented Feb 8, 2018

karenetheridge commented Feb 13, 2018 •

edited

Loading

dankogai commented Feb 14, 2018

karenetheridge commented Feb 14, 2018

karenetheridge commented Feb 19, 2018

New perls #128

New perls #128

Conversation

khwilliamson commented Feb 7, 2018

dankogai commented Feb 8, 2018

karenetheridge commented Feb 13, 2018 • edited Loading

dankogai commented Feb 14, 2018

karenetheridge commented Feb 14, 2018

karenetheridge commented Feb 19, 2018

karenetheridge commented Feb 13, 2018 •

edited

Loading