[QUESTION] send() vs send_and_wait()

[hakan458]

What is the real difference between two? Wait does "and wait for the result" mean for send_and_wait() exactly?

In one part of my application, using send_and_wait() is very fast, but in in another part of the app, it is verrry slow. Both producers are setup the same way (although they are separate producers). Not sure why that would be the case. I tried using send() just now in place of send_and_wait and had no issues, blazing fast.

Is using send() riskier in anyway? Will the producer still retry if there is an error sending the message, or would I have to handle that manually?

For more context, on the producer side I don't need to wait for any reason for the message to be consumed / processed. I just need to make sure it is sent to the topic.

[vmaurin]

Usually for a Kafka producer, you try to batch your writes :

• less roundtrip to the server
• more efficient compression (what is compressed is the batch, not each individual messages)
• less processing needed for the server (usually the broker just need to take the bytes from the socket and append them to the log file)

In aiokafka, send method objective could be translate as "enqueue a message to be sent". It returns a future that will be set when the message is actually sent (part of a batch or not).
send_and_wait is then a shortcut that:

• call send
• await for the returned future to be completed

Then after that, there is a lot of factor that could go on, especially with option like the linger time, the compression used, the actual throughput on this consumer, ...
The retries and ack are also managed by the producer itself, with options you can control. If you just await on send, it will still do it as configured, but the result of the operation will be "lost" as nobody is looking into the future (I think in this case, there is a warning from asyncio on this regard). It is still a good practice to at least add a callback to the future returned by send to log/monitor the result of the send operation