Switch between safe and unsafe calls to isValidUtf8 depending on size

[Bodigrim]

See https://www.reddit.com/r/haskell/comments/rc0us9/comment/hnrx0nr/?utm_source=share&utm_medium=web2x&context=3

CC @noughtmare @kozross

[kozross]

This is an interesting observation. I can make that change easily enough - what do you think we should set the 'breakpoint' at which we separate safe and unsafe calls?

[noughtmare]

I think around 1MB is reasonable. I've made a branch with benchmarks here. The results on my machine are:

All
  isValidUtf8
    1000
      unsafe: OK (0.61s)
        35.2 ns ± 1.2 ns
      safe:   OK (0.42s)
        98.4 ns ± 6.0 ns
      mix:    OK (0.63s)
        36.4 ns ± 1.1 ns
    10000
      unsafe: OK (0.37s)
        177  ns ± 6.9 ns
      safe:   OK (0.25s)
        231  ns ±  22 ns
      mix:    OK (0.19s)
        173  ns ±  17 ns
    100000
      unsafe: OK (0.29s)
        2.14 μs ± 198 ns
      safe:   OK (0.14s)
        2.22 μs ± 189 ns
      mix:    OK (0.28s)
        2.14 μs ± 149 ns
    1000000
      unsafe: OK (0.19s)
        24.3 μs ± 1.8 μs
      safe:   OK (0.20s)
        24.4 μs ± 1.7 μs
      mix:    OK (0.40s)
        24.5 μs ± 1.9 μs
    10000000
      unsafe: OK (0.13s)
        506  μs ±  47 μs
      safe:   OK (0.13s)
        507  μs ±  43 μs
      mix:    OK (0.13s)
        510  μs ±  39 μs
    100000000
      unsafe: OK (0.18s)
        5.97 ms ± 401 μs
      safe:   OK (0.19s)
        5.99 ms ± 342 μs
      mix:    OK (0.19s)
        6.01 ms ± 364 μs
    1000000000
      unsafe: OK (0.78s)
        59.5 ms ± 2.0 ms
      safe:   OK (0.18s)
        59.5 ms ± 4.0 ms
      mix:    OK (0.18s)
        59.1 ms ± 3.7 ms

All 21 tests passed (6.27s)

The input is just S.replicate (10 ^ i) 65 where i is between 3 and 9.

[Bodigrim]

Yeah, 1M sounds reasonable to me. Please patch an instance for ShortByteString as well:

bytestring/Data/ByteString/Short/Internal.hs

Lines 613 to 614 in c7cba39

	foreign import ccall unsafe "bytestring_is_valid_utf8" cIsValidUtf8
	:: ByteArray# -> CSize -> IO CInt

[Bodigrim]

@noughtmare @kozross could one of you contribute a patch please?

[noughtmare]

Does this require a benchmark? I can't really imagine that this might regress in the future.