[C++][Python] Support parsing a StringArray full of JSON to a Table

[ryan-johnson-databricks]

Describe the enhancement requested

Working with pyarrow, there doesn't seem to be any way to parse a StringArray full of json into an Array or Table of nested data. This surprises me, becuase pyarrow.json.read_json does exactly the right thing... but only for line-delimited json files. At least, I didn't see anything e.g. in pyarrow.compute and a google search came up empty.

Looking deeper there doesn't seem to be anything on the C++ side, either.

Skimming the C++ sources, I think there's a core logic that could be wrapped up into a proper compute function. If I read correctly:

• TableReaderImpl::Read
• calls TableReaderImpl::ParseAndInsert
• calls ParseBlock
• which uses a HandlerBase instance that implements BlockParser
• whose doParse method uses a rapidjson Reader to do the heavy lifting.

I think to parse a StringArray (instead of a file), we'd just need an ArrayParser variant, similar to BlockParser, that presents rapidjson a different buffer for each string to be parsed --- maybe a std::spanstream --- so that EOF becomes the "delimiter" instead of newline?

Component(s)

C++, Python