improve index, rindex and indices against string to count the index by utf8 characters

[itchyny]

This is a fix for #1430 and #1624 to resolve confusing byte index.

[coveralls]

Coverage decreased (-0.4%) to 84.628% when pulling 23301d9 on itchyny:index-rindex-indices-utf8 into ad9fc9f on stedolan:master.

[nicowilliams]

@itchyny why did you close this?

[itchyny]

Because I think this will not be merged in the near future due to this comment #1624 (comment) (I still think it's worth changing the behavior though).

[nicowilliams]

Ah, yes, of course. Thanks.

[emanuele6]

This is a fix for #1430.

Do we want to merge this, or close that issue?

[nicowilliams]

This is a fix for #1430.

Do we want to merge this, or close that issue?

Great question. String indexing is fraught; maybe we shouldn't have it at all! But if we'll have it we should definitely have:

• index by nth glyph
• index by nth codepoint

and maybe:

• index by nth byte (because it's all UTF-8, and indexing by byte in Unicode-aware apps may make sense)

That we have indexing by nth byte is essentially a design bug born of not having enough Unicode functionality.

[emanuele6]

@nicowilliams

I think we should change the behaviour of index to make .[index("a"):] work as expected, since string slice indexing is by nth unicode codepoint, otherwise index/rindex/indices are pretty much useless for strings. That is also how index functions work in gojq/fq.