Code

Deep __getitem__ in C++.#8

Code

Merged

jpivarski merged 128 commits intomasterscikit-hep/awkward:masterfrom feature/deep-getitem-2scikit-hep/awkward:feature/deep-getitem-2Copy head branch name to clipboardSep 21, 2019

+2,471-169

Commits

Commits on Sep 2, 2019

• Start on PR7, deep __getitem__, for real this time.

  jpivarski

  committed
  943fe30

  Copy full SHA for 943fe30

Commits on Sep 3, 2019

• Added Content::minmax_depth, which will be used by Ellipsis.

  jpivarski

  committed
  d2ab7dc

  Copy full SHA for d2ab7dc

• [skip ci] writing.

  jpivarski

  committed
  ed0b57e

  Copy full SHA for ed0b57e

• Slices and RawArray compile.

  jpivarski

  committed
  f5eb87d

  Copy full SHA for f5eb87d

• [skip ci] Update the README.

  jpivarski

  committed
  2b872a0

  Copy full SHA for 2b872a0

• [skip ci] What about this link?

  jpivarski

  committed
  032a72f

  Copy full SHA for 032a72f

• [skip ci] What about this link?

  jpivarski

  committed
  42521a0

  Copy full SHA for 42521a0

• [skip ci] Finalized README.

  jpivarski

  committed
  638585a

  Copy full SHA for 638585a

• Add a C++ test of RawArray so we can test this without the full cycle.

  jpivarski

  committed
  1f9b88b

  Copy full SHA for 1f9b88b

• [skip ci] Have to re-think Slices.

  jpivarski

  committed
  ff7d801

  Copy full SHA for ff7d801

• Defined slices.

  jpivarski

  committed
  ea4fd46

  Copy full SHA for ea4fd46

• Poking around, I decided that I should learn how to do slicing on NumpyArray, not RawArray (because NumpyArray supports multiple dimensions).

  jpivarski

  committed
  a097a16

  Copy full SHA for a097a16

• To do the testing in Python, we'll need to convert all of the slice types to C++.

  jpivarski

  committed
  91ea23b

  Copy full SHA for 91ea23b

• Don't actually raise exceptions while testing.

  jpivarski

  committed
  3b6869a

  Copy full SHA for 3b6869a

• [skip ci] working on it.

  jpivarski

  committed
  804dc16

  Copy full SHA for 804dc16

• All Slice types should be readable now.

  jpivarski

  committed
  0942429

  Copy full SHA for 0942429

• Make syntax of test okay for Python 2.7.

  jpivarski

  committed
  6bf5006

  Copy full SHA for 6bf5006

• Make syntax of test okay for Python 2.7 (one more instance).

  jpivarski

  committed
  b834073

  Copy full SHA for b834073

• Figure out what Windows uses as integer format strings.

  jpivarski

  committed
  e9b4b7c

  Copy full SHA for e9b4b7c

Commits on Sep 4, 2019

• Intentionally fail so that we can see the format string explicitly on Windows.

  jpivarski

  committed
  1610532

  Copy full SHA for 1610532

• Should be fixed for Windows.

  jpivarski

  committed
  0a73ed8

  Copy full SHA for 0a73ed8

• Another test for Windows.

  jpivarski

  committed
  de74fa9

  Copy full SHA for de74fa9

• It's safest to let pybind11 decide if it needs to cast.

  jpivarski

  committed
  478a6fd

  Copy full SHA for 478a6fd

• Give NumpyArray a nice representation for common numerical types.

  jpivarski

  committed
  407e5e7

  Copy full SHA for 407e5e7

• Started on the true getitem.

  jpivarski

  committed
  50f6a29

  Copy full SHA for 50f6a29

• Working on chaining getitems.

  jpivarski

  committed
  d8d18f7

  Copy full SHA for d8d18f7

• [skip ci] Interaction between SliceAt and SliceStartStop is starting to work.

  jpivarski

  committed
  2f6ea5b

  Copy full SHA for 2f6ea5b

• Most of getitem has been implemented for SliceAt and SliceStartStop.

  jpivarski

  committed
  6cd7c81

  Copy full SHA for 6cd7c81

• Consolidated and cleaned up.

  jpivarski

  committed
  42ba7e2

  Copy full SHA for 42ba7e2

• But don't intentionally raise an Exception.

  jpivarski

  committed
  b7f2ec1

  Copy full SHA for b7f2ec1

• Fix compilation issues on 32-bit Windows.

  jpivarski

  committed
  81c6ba9

  Copy full SHA for 81c6ba9

Commits on Sep 5, 2019

• [skip ci] carry is composable.

  jpivarski

  committed
  a3534a6

  Copy full SHA for a3534a6

• SliceAt and SliceStartStop for no-carry and yes-carry, but there's a memory error to deal with.

  jpivarski

  committed
  544e4e1

  Copy full SHA for 544e4e1

• Fixed memory error (running off end of array).

  jpivarski

  committed
  aed8c6b

  Copy full SHA for aed8c6b

• Fix Windows 32-bit compilation issues.

  jpivarski

  committed
  104ad28

  Copy full SHA for 104ad28

• All cases are working except two or more slices.

  jpivarski

  committed
  829b452

  Copy full SHA for 829b452

• [skip ci] working on it.

  jpivarski

  committed
  7dac973

  Copy full SHA for 7dac973

• [skip ci] Got this one case, but I should think more about it.

  jpivarski

  committed
  c815dca

  Copy full SHA for c815dca

Commits on Sep 6, 2019

• [skip ci] Working out the logic of carrying in Python; good for depth <= 2 so far.

  jpivarski

  committed
  75c8fca

  Copy full SHA for 75c8fca

• [skip ci] more cases working.

  jpivarski

  committed
  038c755

  Copy full SHA for 038c755

• [skip ci] I need to rethink what 'carry' means.

  jpivarski

  committed
  c090185

  Copy full SHA for c090185

• [skip ci] Got it for all levels of slice2.

  jpivarski

  committed
  6d98130

  Copy full SHA for 6d98130

• [skip ci] Working for some int indexes.

  jpivarski

  committed
  e488781

  Copy full SHA for e488781

• [skip ci] All int and slice2 cases are working.

  jpivarski

  committed
  3a2b6d9

  Copy full SHA for 3a2b6d9

• [skip ci] Integer array indexing works, apart from iterating 'as one'.

  jpivarski

  committed
  b9ae80c

  Copy full SHA for b9ae80c

• [skip ci] Maybe the 'as one' thing is working...

  jpivarski

  committed
  2298b19

  Copy full SHA for 2298b19

• [skip ci] Walking 'as one' through a slice.

  jpivarski

  committed
  8fa1989

  Copy full SHA for 8fa1989

• [skip ci] Have to think harder about how walking 'as one' interacts with slices.

  jpivarski

  committed
  58ef80e

  Copy full SHA for 58ef80e

Commits on Sep 7, 2019

• [skip ci] Stride-based getitem(slice2) works, and it seems to be simpler than the functional version.

  jpivarski

  committed
  323f9d0

  Copy full SHA for 323f9d0

• [skip ci] Intarray composes with strided slice2.

  jpivarski

  committed
  d086f92

  Copy full SHA for d086f92

• [skip ci] Very nearly have intslice composing with slice2.

  jpivarski

  committed
  7c44c83

  Copy full SHA for 7c44c83

• [skip ci] Trouble is, we reshape a strided array differently from a compacted array.

  jpivarski

  committed
  27c19b4

  Copy full SHA for 27c19b4

Commits on Sep 16, 2019

• Start a new technique for __getitem__ and also check that Azure still works.

  jpivarski

  committed
  12ade50

  Copy full SHA for 12ade50

• But don't run deliberately broken code.

  jpivarski

  committed
  15f58be

  Copy full SHA for 15f58be

• [skip ci] set up to do array-first getitem

  jpivarski

  committed
  2e781bb

  Copy full SHA for 2e781bb

• [skip ci] seems to be a good machine for array indexing

  jpivarski

  committed
  884ac36

  Copy full SHA for 884ac36

• [skip ci] works for 0, 1, 2 slices

  jpivarski

  committed
  235965c

  Copy full SHA for 235965c

• [skip ci] works for 0, 1, 2, 3 slices

  jpivarski

  committed
  bf59a2d

  Copy full SHA for bf59a2d

• [skip ci] and arrays work again

  jpivarski

  committed
  2cc162f

  Copy full SHA for 2cc162f

Commits on Sep 17, 2019

• [skip ci] working for some non-trivial strides (haven't checked negative strides yet)

  jpivarski

  committed
  a745274

  Copy full SHA for a745274

• [skip ci] can't do uneven or negative strides with a carry; have to compact in these cases

  jpivarski

  committed
  914c554

  Copy full SHA for 914c554

• [skip ci] working on compaction

  jpivarski

  committed
  caebe50

  Copy full SHA for caebe50

• [skip ci] correctly ingesting arrays and slicing still works; need to check compaction again

  jpivarski

  committed
  dc19aaf

  Copy full SHA for dc19aaf

• [skip ci] compaction works with the new ctypes-based __init__

  jpivarski

  committed
  8bfc74c

  Copy full SHA for 8bfc74c

• [skip ci] integers were very easy to add: good sign!

  jpivarski

  committed
  e8b580e

  Copy full SHA for e8b580e

• [skip ci] slice3 works

  jpivarski

  committed
  4b6e772

  Copy full SHA for 4b6e772

• [skip ci] passing down advanced

  jpivarski

  committed
  da13613

  Copy full SHA for da13613

• [skip ci] working on int vs array policies

  jpivarski

  committed
  1fb88e5

  Copy full SHA for 1fb88e5

• [skip ci] those policies are very complicated

  jpivarski

  committed
  4076b98

  Copy full SHA for 4076b98

Commits on Sep 18, 2019

• [skip ci] Numpy has a strange rule for split advanced indexing that presupposes rectilinear structure and I won't support it.

  jpivarski

  committed
  36482ca

  Copy full SHA for 36482ca

• [skip ci] working on making integers and arrays by broadcasting

  jpivarski

  committed
  f57c985

  Copy full SHA for f57c985

• [skip ci] Our scope will not include basic indexes between advanced indexes (Numpy's implementation relies on rectilinear structure).

  jpivarski

  committed
  2d129ff

  Copy full SHA for 2d129ff

• [skip ci] simplify code before introducing multidimensional integer-array index

  jpivarski

  committed
  0935aa6

  Copy full SHA for 0935aa6

• [skip ci] multidimensional integer-array indexes work

  jpivarski

  committed
  e1930ea

  Copy full SHA for e1930ea

• [skip ci] we get boolean-array indexing by preprocessing

  jpivarski

  committed
  83a2de1

  Copy full SHA for 83a2de1

• [skip ci] All of the getitem_next_array cases are working.

  jpivarski

  committed
  edfa084

  Copy full SHA for edfa084

• [skip ci] Earlier work on selecting by strides converted over and it works (negative strides, non-trivial input strides, etc.).

  jpivarski

  committed
  16dc780

  Copy full SHA for 16dc780

• [skip ci] same for integer

  jpivarski

  committed
  e76928f

  Copy full SHA for e76928f

• [skip ci] same for newaxis and Ellipsis

  jpivarski

  committed
  8a82c8b

  Copy full SHA for 8a82c8b

• The getitem study is done; now to implement it!

  jpivarski

  committed
  83382b7

  Copy full SHA for 83382b7

• Removed all getitem and SliceItem from C++.

  jpivarski

  committed
  081b3d2

  Copy full SHA for 081b3d2

• Fix visibility warnings in MacOS (Cling is getting the wrong -fvisibility default?).

  jpivarski

  committed
  ad0c79d

  Copy full SHA for ad0c79d

• And choose the new CMP0063 policy for that.

  jpivarski

  committed
  c19ba89

  Copy full SHA for c19ba89

• [skip ci] save work

  jpivarski

  committed
  84aaa7b

  Copy full SHA for 84aaa7b

Commits on Sep 19, 2019

• [skip ci] writing up theory notes

  jpivarski

  committed
  055fccf

  Copy full SHA for 055fccf

• [skip ci] writing up theory notes

  jpivarski

  committed
  51d693c

  Copy full SHA for 51d693c

• [skip ci] writing up theory notes

  jpivarski

  committed
  12afb68

  Copy full SHA for 12afb68

• [skip ci] writing up theory notes

  jpivarski

  committed
  adef282

  Copy full SHA for adef282

• [skip ci] writing up theory notes

  jpivarski

  committed
  8d02f97

  Copy full SHA for 8d02f97

• [skip ci] writing up theory notes

  jpivarski

  committed
  8b8ae68

  Copy full SHA for 8b8ae68

• [skip ci] writing up theory notes

  jpivarski

  committed
  75d3b5d

  Copy full SHA for 75d3b5d

• [skip ci] theory notes are done

  jpivarski

  committed
  25cf336

  Copy full SHA for 25cf336

• [skip ci] theory notes are done

  jpivarski

  committed
  6507357

  Copy full SHA for 6507357

• Rewriting all the Slices; compiles again.

  jpivarski

  committed
  262f759

  Copy full SHA for 262f759

• Rewriting all the Slices; toslice_part has to modify the Slice in place (because it needs to flatten boolarray -> intarrays).

  jpivarski

  committed
  17b54ad

  Copy full SHA for 17b54ad

Commits on Sep 20, 2019

• toslice and toslice_part compile

  jpivarski

  committed
  76a5d48

  Copy full SHA for 76a5d48

• toslice and toslice_part are successfully linked and can be called

  jpivarski

  committed
  7f21b6b

  Copy full SHA for 7f21b6b

• Reinstated Slice tests (in a way that can be permanent).

  jpivarski

  committed
  88b91a2

  Copy full SHA for 88b91a2

• SliceArray64 uses '...' for large arrays (shows no more than 6 elements at each level).

  jpivarski

  committed
  2c2cf4a

  Copy full SHA for 2c2cf4a

• Broadcasting works.

  jpivarski

  committed
  92bd4de

  Copy full SHA for 92bd4de

• Includes test against the case we're giving up on.

  jpivarski

  committed
  ad1133c

  Copy full SHA for ad1133c

• Numpy::getitem is beginning to work: SliceAt by strides is done.

  jpivarski

  committed
  1382488

  Copy full SHA for 1382488

• Fix 32-bit compilation.

  jpivarski

  committed
  d59fcb9

  Copy full SHA for d59fcb9

• [skip ci] working on slice bystrides

  jpivarski

  committed
  b22ecf4

  Copy full SHA for b22ecf4

• [skip ci] slices might be working

  jpivarski

  committed
  f5ec8fe

  Copy full SHA for f5ec8fe

• The new getitem is tested for integer and slice.

  jpivarski

  committed
  76ba396

  Copy full SHA for 76ba396

• Short-circuited NumpyArray::get and NumpyArray::slice through NumpyArray::getitem, but IDs will need to be propagated through it.

  jpivarski

  committed
  e31b34a

  Copy full SHA for e31b34a

• Fix 32-bit Windows.

  jpivarski

  committed
  703c037

  Copy full SHA for 703c037

• The experimental getitem became standard __getitem__.

  jpivarski

  committed
  6df2940

  Copy full SHA for 6df2940

• NumpyArray::getitem_bystrides.

  jpivarski

  committed
  52aff06

  Copy full SHA for 52aff06

• Fix Python 2.7.

  jpivarski

  committed
  d774273

  Copy full SHA for d774273

• Starting onto NumpyArray::contiguous.

  jpivarski

  committed
  590c16e

  Copy full SHA for 590c16e

• [skip ci] working on contiguous

  jpivarski

  committed
  fae177e

  Copy full SHA for fae177e

• Contiguous is working.

  jpivarski

  committed
  1f4d331

  Copy full SHA for 1f4d331

• Stub of NumpyArray::getitem_next compiles.

  jpivarski

  committed
  dbd8175

  Copy full SHA for dbd8175

• getitem_next null in progress

  jpivarski

  committed
  711da56

  Copy full SHA for 711da56

Commits on Sep 21, 2019

• One level of getitem_next is working.

  jpivarski

  committed
  1063d7a

  Copy full SHA for 1063d7a

• Two levels of getitem_next are working.

  jpivarski

  committed
  a125fcf

  Copy full SHA for a125fcf

• Add boolean mask tests and fix integer size/signed warnings.

  jpivarski

  committed
  74c0e2b

  Copy full SHA for 74c0e2b

• Empty index array is not a special case.

  jpivarski

  committed
  92fc9ab

  Copy full SHA for 92fc9ab

• Slices in NumpyArray::getitem_next work.

  jpivarski

  committed
  957d279

  Copy full SHA for 957d279

• All cases in NumpyArray::getitem are covered.

  jpivarski

  committed
  95dd022

  Copy full SHA for 95dd022

• Start moving functions to cpu-kernels and fix Python 2.7.

  jpivarski

  committed
  b5c50a2

  Copy full SHA for b5c50a2

• Moving more functions into cpu-kernels.

  jpivarski

  committed
  e1743cb

  Copy full SHA for e1743cb

• Fixed a bug in contiguous.

  jpivarski

  committed
  f471578

  Copy full SHA for f471578

• Moving more functions into cpu-kernels.

  jpivarski

  committed
  71b5f5a

  Copy full SHA for 71b5f5a

• Finished moving functions into cpu-kernels.

  jpivarski

  committed
  c2a7e22

  Copy full SHA for c2a7e22

• Minimize '#include' scopes.

  jpivarski

  committed
  cd4ec30

  Copy full SHA for cd4ec30