Re-enable google-bigquery and add a work around

[zhfeng]

Exclude org.apache.arrow:arrow-memory-netty because it is not compatible with the lastest netty 4.1.94.Final.

[jamesnetherton]

We should add the same doc notes for the workaround like we did for pubsub & storage.

Also, not sure how safe the Arrow exclusion is. It is referenced in a few places:

https://github.com/search?q=repo%3Agoogleapis%2Fjava-bigquery+arrow+language%3AJava&type=code&l=Java

I guess the fact that the native compilation + tests passed means that maybe we don't hit those code paths in Camel...

[zhfeng]

Yeah, it is exactly what I was concerned. It seems that google-bigquery default uses DataFormat.ARROW when read/write a large data table result.

We don't hit those codes because we lack of a test for the large table. So how can write a sentence for this limitation?

[jamesnetherton]

Thinking a bit more, I'm not sure I like excluding Arrow because it potentially impacts negatively on JVM mode, and the issue is purely a native mode one.

[jamesnetherton]

@zhfeng is it just the shaded Netty of Arrow that is causing the problem? If Arrow does not modify any of the Netty bits, maybe we could try something similar to what we did for Kudu and shaded Netty? See here: #2008 & #1907.

[zhfeng]

@jamesnetherton I think apache-arrow does not shade Netty. The problem is Quarkus 3.2.0.Final update Netty to 4.1.94.Final and there is an un-compatible change in PooledByteBufAllocator. So I think it could also impact the JVM mode if we test with a large table.

The fix in apache-arrow is apache/arrow@ea4f03a#diff-d967dd684919ce097b81e439fafc96217f1263985f5864faa4b78eb96e7a4f16L164

So is it possible to have this file in our camel-quarkus-google-bigquery?

[jamesnetherton]

Ah, yeah. Sorry for the confusion!

So is it possible to have this file in our camel-quarkus-google-bigquery?

We could temporarily have it in our extension. Quarkus has RemovedResourceBuildItem, not sure if that could also be used to remove the bad impl from Arrow?

If it becomes too much work. Maybe we just stick with the original plan of removing Arrow until we get the fix.

[zhfeng]

Sine org.apache.arrow:arrow-memory-netty is a runtime dependency, RemovedResourceBuildItem seems not working. I just try to copy 4 files from apache-arrow

• LargeBuffer.java
• MutableWrappedByteBuf.java
• PooledByteBufAllocatorL.java
• UnsafeDirectLittleEndian.java

and apply the fix for PooledByteBufAllocatorL.java. All of the google-bigquery tests look good both in JVM and native mode.

Also I think we need a test for a large table and enable the result with arrow data format. Can it be supported by wiremock?

[jamesnetherton]

Can it be supported by wiremock?

Yes, probably. Not sure if it'd mean storing huge files for the stubbed data in the repo though.....

[ppalaga]

storing huge files for the stubbed data

We could perhaps generate those at build time?

[zhfeng]

I raise

• Improve the camel-quarkus-google-bigquery to test with a large table #5048