Add batching based on payload size

[alexanderdean]

No description provided.

[jbeemster]

What kind of threshold are we looking at keeping payloads below for this ticket? I also assume this would only be relating to POST requests yes?

[alexanderdean]

Hmm, I think we should make it user-configurable? We will keep increasing the payload size supported by our collectors so this should be tunable I guess depending on the collector version a user is using...

[jbeemster]

@alexanderdean what do we in the case of a GET request being too big? Do we bin it or let the collector deal with it?

[alexanderdean]

Good Q. @fblundun - what does the JS Tracker do?

[jbeemster]

Also if the collectors are going to check the size of the payload regardless does this not create a bit of double work? Do all the collectors check payload size?

[alexanderdean]

Collectors don't formally check payload size. Collectors can reject events which exceed their tolerated payload size.

I know we had an issue at one stage where a collector was rejecting an oversized GET and that was backing up the events being sent from the browser.

[alexanderdean]

@jbeemster - the JS tickets:

• Make single events exceeding the maximum request size attempt to fire exactly once snowplow-javascript-tracker#359
• Investigate having a maximum size for GET requests snowplow-javascript-tracker#362

[fblundun]

The JS Tracker currently doesn't check the size of GETs.

For POSTs: this function is used to determine the size of the request body. maxPostBytes is the configurable limit and it defaults to 40000. If a single event is too big, the tracker attempts to fire it once and then forgets about it. Otherwise it gets added to the localStorage queue, along with metadata about its size in bytes. When events are sent in batches, each batch is less than maxPostBytes in size.