out of range schema id in avroProducer for IBM Event Streams

[jesusmah]

Description

When working with IBM Event Streams and the confluent_kafka.avro library I get the following error:

Traceback (most recent call last):
  File "ContainerAvroProducer.py", line 75, in <module>
    kp.publishEvent(TOPIC_NAME,container_event,"containerID")
  File "/home/kafka/KcAvroProducer.py", line 43, in publishEvent
    self.producer.produce(topic=topicName,value=json.loads(value),key=json.loads(value)[key], callback=self.delivery_report)
  File "/root/.local/lib/python3.7/site-packages/confluent_kafka/avro/__init__.py", line 80, in produce
    value = self._serializer.encode_record_with_schema(topic, value_schema, value)
  File "/root/.local/lib/python3.7/site-packages/confluent_kafka/avro/serializer/message_serializer.py", line 113, in encode_record_with_schema
    return self.encode_record_with_schema_id(schema_id, record, is_key=is_key)
  File "/root/.local/lib/python3.7/site-packages/confluent_kafka/avro/serializer/message_serializer.py", line 144, in encode_record_with_schema_id
    outf.write(struct.pack('>bI', MAGIC_BYTE, schema_id))
struct.error: argument out of range

the schema id that IBM Event Streams produces is avrotest-value-6cqbo6

Apparently, the Java client does not have such problem.

How to reproduce

Send a message to a topic in IBM Event Streams whose name is of some length (not much as you can see above).

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system:
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[rnpridgeon]

Do you know if IBM Event Streams Schema Registry uses the same wire format as Confluent's Schema Registry? The error message seems to indicate that the schema ID may not be an unsigned int.

Does this happen to be a problem when using the 2.7 interpreter as well?

Thanks,
Ryan

[jesusmah]

I don't know about the wire format or if it works with the 2.7 interpreter... pretty new here. Sorry.
What one of the IBM Event Streams developer said about this problem is:

"it’s due to Event Streams returning a negative ID (which fits in a Java unsigned integer, but Pythons struct.pack call does not like)"

[rnpridgeon]

Yeah it looks like technically the schema registry wire format uses a signed int. In practice no schema will ever be assigned a negative id as its indexed from 1. That said this is technically a bug even if the Confluent schema registry will never handout a negative id.

https://github.com/confluentinc/schema-registry/blob/0e55aaa0baf26fa26803fa473d16f9df383739a0/core/src/main/java/io/confluent/kafka/schemaregistry/id/ZookeeperIdGenerator.java