fix(java): replace org.json with gson to resolve the jar conflict with spark 3.5.1

[SaintBacchus]

Now use spark 3.5.1 binary package will cause this Exception

The spark package contain a json jar named json-1.8.jar which also has JSONArray

[chenkovsky]

just another possible solution. I refactored fragment in #3307, removed dependency of json.

[SaintBacchus]

#3307 was a big change of java package. @chenkovsky can you explain the nesseary to refactor the fragment?

[chenkovsky]

#3307 was a big change of java package. @chenkovsky can you explain the nesseary to refactor the fragment?

because lance for python has changed. I think java should be also changed sooner or later. the earlier, the less debt.

[eddyxu]

Curious: does this impact the minimal version of spark we can support

[SaintBacchus]

Yes，now it's only fully tested in spark 3.5.1

[SaintBacchus]

I plan to support multi versions of spark later but I think now the 3.5.x was our minimal version of spark.
Otherwise some users want a old version, I will integrate as them want

[eddyxu]

As long as it can run with the latest databricks runtime, it is fine to me.

https://docs.databricks.com/en/release-notes/runtime/index.html

We don't have too many spark users yet, it is fine to be aggressive rn.