Array slicing for GDScript

[curly-brace]

would be nice if we can use pythonish slice feature like

array[2:4]
array[2:]
array[:-1]

or even may be extended slicing with step

array[::2]
array[::-1]

[bojidar-bg]

Note that this is easily done with range and comprehensions from #4716:

var result = array[from:to:step]
# Is the same as:
var result = (array[i] for i in range(from,to,step))

[bojidar-bg]

Also, to throw in CoffeeScript syntax here as well:

x = array[from...to]

We might adapt this to support step size (since they don't):

x = array[from..step..to]
# or
x = array[from...to:step]

[curly-brace]

imho python syntax with : is more convenient - less characters and more compact

[akien-mga]

imho python syntax with : is more convenient - less characters and more compact

+1. And less confusing. We are python-like, so let's not import behaviours from every other languages in the world or GDScript will become really inconsistent.

[neikeq]

D syntax ($ is the size of the array):

array[2..4]
array[2..$]
array[$-1..$]

Just for reference. I agree Python's one fits better with GDScript :P

[brakhane]

I'm willing to work on this one.

As a first step I'm preparing a PR that adds negative indexing. (If we allow array[:-1] we should allow array[-1] as well.)

[brakhane]

What about assigning to array slices? Are there use cases?

I'm confident that I can get the code to support it, but if there are no use cases, I might not bother with it.

I think one restriction we have to make is that the replacement values must have the same size (which Python also enforces on array-like structures with a fixed size, like bytes)

In other words, the following should work:

var a = range(10)
a[2:5] = [10, 11, 12]
print(a) #[0, 1, 10, 11, 12, 3, 4, 5, 6, 7, 8, 9]

The following might:

a = range(10)
a[1::2] = [10, 11, 12, 13, 14]
print(a) # [0, 10, 2, 11, 4, 12, 6, 13, 8, 14]

But this most probably wont:

a = range(10)
a[2:5] = [10]
print(a) # [0, 1, 10, 5, 6, 7, 8, 9]

Opinions?

[brakhane]

Just for the record, I'm still working on it. I might have a PR ready in the next week, but it can take longer.

[bojidar-bg]

@brakhane Nobody is in a great hurry -- take your time :)

[gau-veldt]

see #5701 and #5765
I added a simple 2-argument slice operation to DVector and ByteArray binds which with some future binds can apply to any of the GDSript lowlevel array types (IntArray, StringArray, etc). Code is not yet there to handle -1. Just realized I left that convenience out just now.

[akien-mga]

I think this should be implemented for 3.0, currently we just have the subarray(from, to) method implemented for PoolByteArray, but not for other builtin types, so it's very inconsistent. If we don't implement something consistent for 3.0, I'd propose to revert #5879 (which was not yet shipped in a stable release) to avoid introduction an API that might be replaced later on.

[brakhane]

My implementation is 98% done (it works, but there are some memory leaks and random crashes which I didnt have the time to debug), but due to real life time constraints I won't be able to finish it.

I'm more than willing to put my WIP into a branch so somebody can (help me) finish it. I can help with questions regarding the implementation. It would be a shame if my effort is going to waste (I spend probably around 40h so far on it)

The implementation is pretty feature complete (and I'm actually quite proud of how complete it turned out), for example slicing works like in Python for most array types, including assignment (a[1:4] = [1,2,3] for example does work, so does a[1:3] = b[10:13:2] and vice versa).

There are a few caveats though:

• the implementation is kinda complex, and you must have some understanding about Variant.cpp and variant_op as well as the bytecode interpreter to make sense of it.
• To keep my sanity and help debugging, I had to replace some of the macros in variant_op with equivalent templates, IMO they make the code more readable (and with optimizations enabled as fast as before), but since the code is based on 2.1, there probably need to be some refactoring to compile with 3.0; also, since the code is WIP, I didn't have time to split it into seperate commits, so sometimes a macro is replaced with a template while at the same time the variant logic is added as well; it makes it a little bit harder (but not too difficult) to distinguish the two
• To avoid making the byte code interpreter incompatibel with old versions, I had to introduce a new data type slice, which needs to be exposed into the editor. This work is done, but there are a few rough edges.
• As already said, the "only" issue left is some memory corruption which I couldn't figure out; whoever wants to debug this should be familiar on how the Godot bytecode interpreter works

Let me know If i should put the code somewhere. I think someone familiar with the godot script internals will get the kinks ironed out in 10h or less.

[vnen]

@brakhane since you already have work done, it would be great if you put the code in your fork and open a Pull Request. This way other people can review and comment on your code.

[groscalin]

How's it going?
Looking forward this. :)

[reduz]

@bojidar-bg is our new gdscript dev, care to give it a try? :P

[reduz]

well, will push for 3.1

[aidanhs]

Just wanted to note that the subarray method uses inclusive ranges, whereas this will (hopefully) use python-style slicing ranges.

I say hopefully because the subarray method is a bit painful to use, requiring you to write special casing to make sure you never try and create a zero width slice (e.g. if you're trying to consume part of a buffer by slicing it twice, the 'remaining' slice can be zero length), and to add -1 to all lengths when requesting a slice.

Edit: in fact, I can't think of a situation off the top of my head where you wouldn't need to add a -1 to the end slice, unless you're slicing constant number bytes and so can do the -1 when writing it.

[testman42]

Any progress with this?
I just found out that I could really use this functionality in GDscript

[akien-mga]

Moving to the next milestone as 3.1 is now feature frozen (and 3.2 has various GDScript improvements planned).

[jupiterjosh]

Hi I just came across this item and wanted to throw in my two cents.

If smart slicing in any way impacts the performance of individual array indexing then we should probably avoid it. Iterating over objects in GDScript is too common of a use-case to suffer performance penalties for the sake of a little syntactic sugar.

While I'm all for syntactic sugar, I think the average game developer has reasonable expectations related to atomicity and speed when anything related to brackets, curly braces, parentheses etc. is implemented in GDScript. If that contract is broken then the question of when I should push something to GDNative becomes much more hazy. I think a good "sub_array()" or "slice()" function would be the best way to go because then (I assume) you could keep existing array indexing code as-is.

[and3rson]

Is there any progress on this? I could submit a PR that adds such methods.

[Henelik]

Still want this to be implemented. Any news?

[SolarLune]

Same, I want to grab a subset from an array, and it seems like both list comprehensions and a function to slice a generic array don't exist. One of these should probably be added, as the alternative is a bit verbose.

[SnailBones]

@bojidar-bg

Note that this is easily done with range and comprehensions from #4716:

var result = array[from:to:step]
# Is the same as:
var result = (array[i] for i in range(from,to,step))

That's not currently correct, as comprehensions have yet to be included. The current syntax is:

var result = []
for i in range(from, to, step):
	result.append(array[i])

Bizarrely, the devs rejected a PR for this feature (#15222) for reasons of readability.

We've discussed this on IRC with many core developers, and the general feeling is that the syntax of list comprehensions makes code quite hard to read, and goes against the design principles of GDScript to have a straightfoward and easy to read syntax.

Personally, I find the one line no harder to read than the three. Anybody else think this conversation should be reopened?

[Calinou]

@SnailBones I think having a built-in Array.slice() method makes the most sense here. Languages like JavaScript have a similar function, and it seems to work well there 🙂

[levilindsey]

I think list comprehensions would be nice, but having a separate Array.slice() would be useful regardless.

[creikey]

I'm working on an Array.slice method right now

[creikey]

Would it be more preferred to create a new Variant slice type like how python does it where that slice object is fed into the array as an operator, or for the call to the slice function in the array to be hard-coded in the compiler? I have the slice method done and I'm looking into adding the start:end:delta syntax to gdscript indexing.

[Henelik]

The slice object would be more "Pythonic" ;) but the hard-coded method would almost certainly be faster. Either way, using the start:end:delta syntax would be very helpful.

…

On Sat, Jul 20, 2019 at 6:05 PM Cameron Reikes ***@***.***> wrote: Would it be more preferred to create a new Variant slice type like how python does it where that slice object is fed into the array as an operator, or for the call to the slice function in the array to be hard-coded in the parser? I have the slice method done and I'm looking into adding the start:end:delta syntax to gdscript indexing. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4715?email_source=notifications&email_token=ACJUE6P4A4CDQWUS4PSQQH3QAOK2JA5CNFSM4CEI4JPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2NXQ4Q#issuecomment-513505394>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJUE6LOTSF243FOYXXR2SDQAOK2JANCNFSM4CEI4JPA> .

[creikey]

I only think to use a new slice kind of variant because then the slicing could be an operator in the parser and it would work out nicer, however I'm not super experienced in that area so it would be great to have one of the gdscript devs weigh in.

[Henelik]

You're right about that. It would also be much easier to re-use the syntax structure for other things later. Sounds like the best option to me.

…

On Sat, Jul 20, 2019 at 11:28 PM Cameron Reikes ***@***.***> wrote: I only think to use a new slice kind of variant because then the slicing could be an operator in the parser and it would work out nicer — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4715?email_source=notifications&email_token=ACJUE6IMHLNTYTVL7DITY2TQAPQX3A5CNFSM4CEI4JPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2N3D4Q#issuecomment-513520114>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJUE6P5Q55EXAWD6IB4ARDQAPQX3ANCNFSM4CEI4JPA> .

[brakhane]

I will see if I can find my old implementation that's still lying around somewhere. It was 90% finished and worked kinda like Python does, in that there is a slice object that you can use to index into arrays. You could even do Stuff like arr[2:4] = [1,2,3].

I stopped working on it because of time constraints

[creikey]

@brakhane That would be very helpful. Right now I'm in the process of adding a new slice object variant type so that the act of slicing can be considered an operation, making modifying the parser much easier.

[brakhane]

@creikey That sounds pretty similar to my approach. I've deleted my godot GitHub repo, but I might have the code still lying around elsewhere. I'll post an update

[brakhane]

@creikey I've found a not completely recent WIP (there might be some features missing in this version). You can find it here: https://github.com/brakhane/godot/compare/4c4ab14..8a258c0

There are still some bugs, for example, I just noticed here https://github.com/brakhane/godot/compare/4c4ab14..8a258c0#diff-7d521a4f767fb1ae3c908a20616084a4R1446 it should say ".end" instead of ".start"

If you have questions, I'm happy to answer them. I will also try to find a more recent version.

[creikey]

I am of the opinion now that adding pythonic array slicing rather than just a function with an inclusive upper bound is adding needless complexity, as it would entail creating a new variant, with little benefit in comparison to just adding a method.