[VL] can not write hive table on HDFS

[RaoZhiRou-Z]

Description

when add the config "spark.gluten.sql.native.writer.enabled true"，and an error occurred："The file path is not local when writing data with parquet format in velox runtime!"
Does velox not support writing hive table on HDFS now？

void VeloxParquetDatasource::init(const std::unordered_map<std::string, std::string>& sparkConfs) { if (strncmp(filePath_.c_str(), "file:", 5) == 0) { sink_ = dwio::common::FileSink::create(filePath_, {.pool = pool_.get()}); } else { throw std::runtime_error("The file path is not local when writing data with parquet format in velox runtime!"); } ......... }

[PHILO-HE]

@RaoZhiRou-Z, did you do the test with gluten main branch?

[RaoZhiRou-Z]

@RaoZhiRou-Z, did you do the test with gluten main branch?

yes，the main branch

[PHILO-HE]

@RaoZhiRou-Z, it seems a bug. I created one PR to fix it: #5881. Please take a review.

[RaoZhiRou-Z]

@RaoZhiRou-Z, it seems a bug. I created one PR to fix it: #5881. Please take a review.

thanks，I will have a try

[RaoZhiRou-Z]

@PHILO-HE I have tried the bug-fix，and there is another core error，the core stack shows that：

#0 __GI_raise (sig=sig@entry=6) at /root/work/deck/devel/toolchain/glibc-2.33/signal/raise.c:49
#1 0x00007f6143cfa527 in __GI_abort () at abort.c:79
#2 0x00007f61435ec089 in os::abort(bool) () from /home/work/zjh/env/jdk/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so
#3 0x00007f61437b5363 in VMError::report_and_die() () from /home/work/zjh/env/jdk/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so
#4 0x00007f61435f64e5 in JVM_handle_linux_signal () from /home/work/zjh/env/jdk/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so
#5 0x00007f61435e8f48 in signalHandler(int, siginfo, void) () from /home/work/zjh/env/jdk/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so
#6
#7 0x00007f5ffc54e52a in facebook::velox::dwio::common::FileSink::writeWithLogging(std::vector<facebook::velox::dwio::common::DataBuffer<char, void>, std::allocator<facebook::velox::dwio::common::DataBuffer<char, void> > >&) ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libvelox.so
#8 0x00007f5ffc54f102 in facebook::velox::dwio::common::FileSink::write(facebook::velox::dwio::common::DataBuffer<char, void>) ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libvelox.so
#9 0x00007f5ffc3c9306 in facebook::velox::parquet::ArrowDataBufferSink::Close() ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libvelox.so
#10 0x00007f5ffc3c525a in facebook::velox::parquet::Writer::close() ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libvelox.so
#11 0x00007f5ffe1ce775 in Java_org_apache_gluten_datasource_DatasourceJniWrapper_close ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libgluten.so
#12 0x00007f612d018767 in ?? ()
#13 0x00007f5ff178fca8 in ?? ()
#14 0x00007f612d0184c7 in ?? ()
#15 0x00007f612d018482 in ?? ()
#16 0x00007f5ff178fc68 in ?? ()
#17 0x00007f5fe36470e8 in ?? ()
#18 0x00007f5ff178fcd8 in ?? ()
#19 0x00007f5fe3647298 in ?? ()
#20 0x0000000000000000 in ?? ()**

[PHILO-HE]

@RaoZhiRou-Z, it's strange. Could you share some details about your test for reproducing?