Maximum Message Size

[petersilva]

None of the MQP message protocols define a maximum message size, but message queuing brokers are optimized for small messages. Brokers of all MQP achieve their highest performance forwarding many small messages, rather than few large ones. Even rare Large messages can have an outsized effect on performance by requiring buffers on brokers to expand for those rare cases, so keeping messages are uniformly "small" is necessary for best performance.

The renewal of MQP protocol usage in GTS aims to have only announcements of products, not to transport the data itself, which is a particularly easy way to get smaller messages. However, many people with long experience with other messaging protocols expect the old methods, where products themselves were included in the files ( examples #36 ) and the Aviation people seem to want data to be included in the body as well. This is unlikely to scale well.

The aviation people have mentioned 8MB as a maximum product size for aviation. We believe that, whatever method is adopted by other groups, they will eventually find themselves wanting to send products larger than the maximum, and will eventually be faced with two choices: either they add support for links to data, as we have already done, or they introduce segmentation. Having that choice happen at a large message size is likely worse for performance.

The old GTS 386 limit was 500,000 bytes, which already sounds relatively large. That's already a high number in my view, but I think the committee was interested in doing some performance studies before settling on a recommendation. So this is a placeholder for now.

(transferred from wmo-im/GTStoWIS2#44 )

[golfvert]

I believe that the (only?) valid argument is for urgent data (such as CAP warnings). For "normal" earth system data, the default method, get the message, fetch the data through URL is good enough for our operations.
So, if we consider CAP being the (only?) use case and considering that CAP standard doesn't specify a maximum size (that would have been too easy!!), the idea could be to look at the maximum size a weather warning in CAP may have and use this as a threshold. IMHO, it should as small as possible (a few kB) and putting data in the message itself should be strongly discouraged, whatever its size.

[josusky]

This issue seems to be a duplicate of #6, but let me repeate my last comment from that issue:

In TT-Protocols we have converged on 4kB limit but the exact value can be re-considered after the pilot phase. Perhaps we could even allow a higher limit for high-priority things that are sent rarely, e.g. warnings. For sure the brokers shall be configured to handle notifications that are by an order of magnitude (or two) bigger - for resilience.
And we shall not forget that we have the possibility to gzip the embedded content. CAP and other XML-based messages are easily packed to 10% of their original size.

[antje-s]

1. a maximum message size should be set and messages larger than this should be rejected by the Global Broker (or in pilot phase at least a warning should be thrown)
2. the level is more difficult...it should be possible to send all warnings and have some size buffer, but it should be low enough so that there are no problems with pubsub (suggestion is to first take the 4kB and generate warnings at the Global Broker for larger messages, so that you can see if warnings were affected)

[petersilva]

Truncation is not going to work, especially for CAP, the main format of concern. It is XML and important information could be scatterred anywhere in the content. it is not organized to present the most important information first.

The purpose of the messages is to let people decide whether they want the full message.
To make that decision, consumers need to know:

• what area is effected
• what type of event. ? (users dealing with snow cleaning will not find an amber alert relevant.)
• temporal extent

That's all. The essential information needed needs to be in the message itself. The format includes geometry and "start_datetime" and "end_datetime", and the type of event should fall naturally out of the topic hierarchy. If the subtopic and geometry information is combined, that is probably good enough, and including any of the content is redundant (I mean... if the whole warning is < 4kb, then it is fine to include it, but if truncation enters into the story, then do not include any content, since it is just a guess whether the first 4Kb are useful/relevent/crucial.)

sample Canadian warnings:

https://hpfx.collab.science.gc.ca/20230113/WXO-DD/alerts/cap/20230113/CWUL/10/
listing:

 T_WWCN10_C_CWUL_202301131056_1016228456.cap 2023-01-13 10:58  310K  
 T_WWCN10_C_CWUL_202301131056_2449165209.cap 2023-01-13 10:57  310K  
 T_WWCN10_C_CWUL_202301131056_3633968603.cap 2023-01-13 10:57  148K  

The first 4K of a 310K CAP is unlikely to be helpful. It will contain information on a subset of regions affected by the alert... the rest of the regions won't be included.

[josusky]

I agree with @petersilva, neither Global Broker nor any other system shall truncate the notifications. The CAP messages can be compressed to less than 20% of their size which is sometimes more than the magic 4kB but I would not be too worried about it now, these messages (or notifications about such messages) are not so frequent.

[golfvert]

There is no truncation envisaged.
A message with a payload included whose size is above the agreed limit will be (eventually) discarded.
At the moment, a simple warning in the message is added.

[antje-s]

Shall we close the issue with the note: agreed that first 4kB will be set as maximum (with adjustment if necessary due to more experience), during pilot phase with added warning comment from GB - later GB will discard greater messages?

[tomkralidis]

TT-WISMD 2023-09-11:

• maximum message size != inline content
• inline content is set to 4KB in the specification
• issues also arise RE: compaction, minification
• maximum WNM size (whether it contains inline content or not) SHALL be no larger than 8KB
  • implication is that GB could then DISCARD messages > 8KB
• we can revisit over time if applicable (for example from metrics)
• we also need to consider in the context of the initial recommendations made in clarify broker protocol elements wis2-guide#5

Relevant doc updates:

• WNM: update WNM with REQUIREMENT
• Guide: update that GBs MAY discard WNMs > 8KB

[tomkralidis]

PR in #63

[tomkralidis]

Note: this will NOT be updated in the Guide since we already have a provision that covers:

A Global Broker will validate notification messages against the standard format (see 4.3 Notification message format and structure), discarding non-compliant messages and raising an alert.