support creating array of array #2299
Conversation
| @@ -481,6 +481,20 @@ public static DType getNonNestedRapidsType(DataType type) { | |||
| return result; | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
Java docs?
We shouldn't need to add this method, if you follow my suggestion
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeCreator.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeCreator.scala
Outdated
Show resolved
Hide resolved
wbo4958
changed the title
sql-plugin/src/main/scala/com/nvidia/spark/rapids/literals.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeCreator.scala
Outdated
Show resolved
Hide resolved
|
build |
1 similar comment
|
build |
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeCreator.scala
Outdated
Show resolved
Hide resolved
|
build |
wbo4958
force-pushed
the
array-of-array
branch
2 times, most recently
from
bc8c2a5 to
cbdbc3d
Compare
wbo4958
changed the title
|
Can we please not rebase. It can really confuse github with outstanding commits. Just merge. |
|
|
||
| override def tagExprForGpu(): Unit = { | ||
| wrapped.dataType match { | ||
| case ArrayType(ArrayType(ArrayType(_, _), _), _) => |
There was a problem hiding this comment.
Where did this limitation come from? Does cudf only support so much nesting?
There was a problem hiding this comment.
I tested like array(array(array(a))) and libcudf will throw the exception, I read the PR for making list and found there is a comment in Currently, only lists columns of one depth level are supported. rapidsai/cudf#8046
There was a problem hiding this comment.
Could you file a follow on issue for more deeply nested? It is not something we have to work on. I just want to be sure that it is documented.
There was a problem hiding this comment.
sure, filed the following issue #2453
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGenerateExec.scala
Outdated
Show resolved
Hide resolved
|
build |
Signed-off-by: Bobby Wang <[email protected]>
|
build |
|
Please do not rebase your patch. Just merge in the new branch. It is much less likely to confuse github. |
Signed-off-by: Bobby Wang <[email protected]>
Signed-off-by: Bobby Wang <[email protected]>
This PR is for #2011
Currently, Cudf only supports creating 1D or 2D array, See rapidsai/cudf#8046