[BUG]perdayvolume for my bundle is not correct

[YifengMao]

Describe the bug
perdayvolume/throughput/indexed are not correct.

To Reproduce
longevity_syslog.zip
my bundle is very simple. Here's a table of result
egx cluster 3 node to send data via hec to a standalone splunk host

httpwaitresponse	payloadmaxsize	perdayvolum_total	Throughput	actual
false	256K	200	1477K/s 126G	130G
true	256K	200	1474K/s 126G	130G
true	256K	600	4054K/s 350G	355G
true	256K	900	5863K/s 504G	498G
true	256K	3000(hit bottleneck)	10907K/s 942G	1.01T

Additional context
version: 6.3.1

[YifengMao]

2 problems here:

• throughput is less than half of actual throughput.
• actual throughput is greater than perdayvolume, not in a acceptable range (+/- 10%)

[arctan5x]

Hi Yifeng, would you verify this is the latest version? Also, would you provide environment details such as which cluster, container, etc.

[li-wu]

Some testing result:

1. Setting persistentQueueSize and queueSize with 500MB to 1GB does not take effect;
2. Using this pr fix #166 #192 does not work;

instead:
making the timer to sleep half the self.time improves the throughput. And I will try to use more exact sleep time.

[arctan5x]

@YifengMao would this be still valid? If not, may I close it?

[li-wu]

@arctan5x I think this is still valid and not solved. This is also related to how we are going to measure the performance of Eventgen.

[GordonWang]

The issue is resolved. I verified in the test env.
splunk cluster: 3 indexers and 1 SH.
set the perday volume size as 1000GB.

Then, use 2 spl to check the indexed data size. The result is near 1000GB in either way. The volume size deviation is less than 1%.

Here are the screenshots.