KSQL should support nested AVRO records

[getachew]

Hello folks,
I get this error while creating a stream for an Avro schema that has a nested type declaration. Not sure if this is an issue on my side but appreciate any input.

Could not fetch the AVRO schema from schema registry. Cannot find correct type for avro type: AddressUSRecord for a schema that looks like

{
    "name": "person",
    "type": "record",
    "fields": [
        {"name": "firstname", "type": "string"},
        {"name": "lastname", "type": "string"},
        {
            "name": "address",
            "type": {
                        "type" : "record",
                        "name" : "AddressUSRecord",
                        "fields" : [
                            {"name": "streetaddress", "type": "string"},
                            {"name": "city", "type": "string"}
                        ]
                    },
        }
    ]
}

[hjafarpour]

@getachew KSQL does not support nested column types yet. You need to flatten your records first. We plan to add support to the nested types.

[getachew]

Thanks @hjafarpour
I will start looking into the source to see if I can add support and contribute.

[peoplemerge]

+1

[miguno]

For the record, support for nested data structures is something that we are planning to add. It's just not available right now.

[KyleJamesWalker]

With the title change and labels, is this a good place to track progress on KSQL supporting nesting? @apurvam if not is there anything tracking this?

[apurvam]

@KyleJamesWalker this is the place to track progress on this feature. We are going to standardize on using these issues to gauge interest in features, and that will help with prioritization.

As things go, this is pretty high up on the feature requests.

[conorfennell]

Just to add for priorities, this is a blocker on my team for using KSQL

[julienmmm]

+1.

[tommyshiou]

+1

[peoplemerge]

+1'd already but a blocker for me too.

[VladMl]

+1

[zamirarif]

I need this desperately :)

I hope it will be accommodated soon.

[jdlindu]

+1

[dongbin]

+1

[betocols]

+1

[jay999123]

I'm too facing this issue. Hope this will be fixed soon

[piotr-yuxuan]

+1

[Igosuki]

• 1

[Igosuki]

In order to implement this, you simply have to allow columns to have a delimiter and use that to recursively go through structs, nothing more.

[jay999123]

@Igosuki , Could you please provide more details on your comment? like some examples?
You mean to say that, while querying a topic using KSQL, the topic schema should have
"have to allow columns to have a delimiter" ?

Thanks.

[Igosuki]

Over the top of my head :

select user_agent.os from tracking;

'.' being the delimiter here and the avro schema of records in the tracking stream being :

record Event {
  user_agent UserAgent
}

record UserAgent {
  os string
}

[Igosuki]

When sequencing the query to match fields you can remove the leftmost member of a column name if it corresponds to a table alias of the from clause so delimiters don't clash

[jay999123]

Has this issue been addressed in the latest release of confluent 4.1?

[apurvam]

@JanardhanVarikuti no, it isn't in the 4.1 release, but it will be in the next release, sometime in July. Hopefully we will get it in a monthly preview release before that though.

[jay999123]

@apurvam Thanks for the update.

[lhayhurst]

Hello! Is this still on track for a July release with a monthly preview release prior to that?
Thank you for prioritizing this :-)

[miguno]

Yes, this is still on track @lhayhurst.

[gunjanarora]

Hi,

Is there a developer preview version available to support this?

[apurvam]

@gunjanarora https://www.confluent.io/preview-release

[apurvam]

This is now available in KSQL 5.0.0.

[dmi3zkm]

@apurvam @miguno
Sorry to bother.
I know there is a note in docs saying that it's not possible to create nested STRUCT as the result of a query. Just curious are there any plans to implement this feature?
Is it possible to accomplish by means of UDF ?

[apurvam]

It's on the roadmap, but not likely to be tackled any time soon, @dmi3zkm .

[suresh1784]

@apurvam any update on this?

[agavra]

Hello @suresh1784 - I'm not sure which issue you are referring to, but both the issue in the original description and the one referred to by dmi3zkm are both resolved. To see how to create a struct, please see https://docs.ksqldb.io/en/latest/how-to-guides/query-structured-data/#structs

[suresh1784]

@agavra , Thanks.

I have one other query, can I get the offset and partition of the topic on which I am creating the KSQL table.