Bugfix/197 multiprocess not working #218
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
li-wu
requested review from
arctan5x,
GordonWang,
jmeixensperger and
lephino
as code owners
li-wu
commented
May 22, 2019
| @@ -32,7 +32,7 @@ def __init__(self, time, sample=None, config=None, genqueue=None, outputqueue=No | |||
| self.profiler = config.profiler | |||
| self.config = config | |||
| self.sample = sample | |||
| self.end = getattr(self.sample, "end") if getattr(self.sample, "end") is not None else -1 | |||
There was a problem hiding this comment.
This is introduced in this pr: https://github.com/splunk/eventgen/pull/189/files. getattr will throw exception when end is not defined in self.sample.
li-wu
commented
May 22, 2019
| @@ -187,12 +187,11 @@ def real_run(self): | |||
| for worker_id in range(self.config.generatorWorkers): | |||
| # self.generatorPlugin is only an instance, now we need a real plugin. Make a copy of | |||
| # of the sample in case another generator corrupts it. | |||
| copy_sample = copy.copy(self.sample) | |||
There was a problem hiding this comment.
I am not sure why we need to make a copy of the example here since for each sample we put a generator object into queue and a work will pick it up and process it. Previous fix (https://github.com/splunk/eventgen/pull/162/files) has great performance downgrade for multiprocess mode.
There was a problem hiding this comment.
Think this is more from legacy. I'm guessing this comes from the old eg 5 days. Think I had to do this when sample's weren't fully loaded in mem, or you'd have 1 sample save a file into a new sample.
li-wu
commented
May 22, 2019
| @@ -171,15 +171,15 @@ def _sendHTTPEvents(self, payload): | |||
| numberevents = len(payload) | |||
| self.logger.debug("Sending %s events to splunk" % numberevents) | |||
| for line in payload: | |||
| self.logger.debug("line: %s " % line) | |||
There was a problem hiding this comment.
I am not sure why changing it back to debugv here works for historical reason? But changing this back to debugv has the right data volume generated for multiprocess mode.
lephino
approved these changes
May 22, 2019
| @@ -187,12 +187,11 @@ def real_run(self): | |||
| for worker_id in range(self.config.generatorWorkers): | |||
| # self.generatorPlugin is only an instance, now we need a real plugin. Make a copy of | |||
| # of the sample in case another generator corrupts it. | |||
| copy_sample = copy.copy(self.sample) | |||
There was a problem hiding this comment.
Think this is more from legacy. I'm guessing this comes from the old eg 5 days. Think I had to do this when sample's weren't fully loaded in mem, or you'd have 1 sample save a file into a new sample.
jmeixensperger
approved these changes
May 22, 2019
arctan5x
approved these changes
May 23, 2019
li-wu
added a commit
that referenced
this pull request
Jun 5, 2019
* Fix jinja template bug under SA-Eventgen app * Feature timeMultiple (#141) * Adjusting interval with timeMultiple * Update issue templates add the bug report and feature request issue templates * changing the stanzas to produce data * Windbag generator/count + end=0 edge cases (#145) * Generator handling for count = -1, end = 0 working properly * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * [Docs] update contribute docs (#148) * [Docs] update contribute docs * [Docs] update the contribute * Fix make docs bug and summary anchor link error * init unittest * [Build] add ut for timeparser functions * Add more UT for config module * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * Update docs * Post-PEP8 Fixes (#157) * skip sort, gitignore 3rd party libs * fixed yapf ignore, ran format across repo * Issue 160 (#163) * Fixed timer and token * Added a conditional for end=-1 * Update eventgentimer.py * Fixed timer and token (#162) * add extendIndexes feature (#154) * add extendIndexes feature * set extendIndexes as a list value * correct log level * upate doc, change num to weight * Test fix (#168) * Add sample functional test for replay mode * Add token replacement functional tests * skip failed case * Added a timeout * created a results dir * Update version.json * Fix previous pep8 format regression bug (#171) * Fix merge conflict bug * Delete tool orca related part (#178) * add test for jinja template test (#177) * clean up the logic about sampleDir and jinja_template_dir setting Add functional test for jinja template generator Fixes #174 Fixes #167 * revert the change about token, sample mode test cases * merge the change from develop branch * merge test conf from develop branch * use urllib 1.24.2 * format with pep8 * fix bug in template dir resolution * update test case to address multiline event * fix a issue that when setup eventgen with 200+ indexers, any exceptions will terminate setup and re-writing default configure files * Fix bug #179 urlparams is not set (#181) * fix issue-183 * add modinput ft (#185) * Issue 159 (#186) * Fixed timer and token * Added a conditional for end=-1 * Added timeMultiple logic * Time Multiple Fix * Add default pull request reviewers (#190) * Default to -1 (#189) * Changed verbose -> verbosity (#191) * Update README.md (#195) Like other Splunk products - Splunk Enterprise Security, Splunk Business Workflow ... - Splunk Event Generator does not need a definitive article "The" before the product name. * update doc for friendly reading and add backward capability section. (#193) * Update index.md (#194) Update index.md * change the sampleDir setting in test cases (#196) * return status=2 when all tasks finished. (#182) * return status=2 when all tasks finished. * Change release verion to 6.3.6 (#200) * Change release verion to 6.3.6 * Fix flaky tests (#204) * Fix flaky tests * Rename modinput to mod_input to avoid functional test fail * when hit an exception, make sure egx logs it to files * Fix circleci status badges (#208) * Clean up and interval error (#211) * Fix generatorWorks not work issue (#207) * Fix navigation error with installed with splunk stream (#214) * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * Add 3rd lib in app (#210) * Add 3rd lib in app * Bugfix/197 multiprocess not working (#218) * Fix issue 197 multiprocess not working * fix issue 219 (#220) * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * Add license credits (#222) * Feature/multi indexes (#224) * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * add extendIndexes feature * set extendIndexes as a list value * upate doc, change num to weight * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * add python code style lint and format * Add more unit tests for config module * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * fix no len dict out of range * clean duplicate code * Revert "Metrics output plugin" (#226) * [issue 217]disable logging queue in multiprocess mode (#223) * disable logging queue in multiprocess mode * Fixed fileName (#229) * Issue 201 (#221) * Removing unavailable servers * Removed nonresponding httpserver * Added a counter * fixed a bug * Added docs and spec * Addressing unused var * fixing url reference * fixed currentreadsize * Httpevent str formatting * fix #166 (#192)
li-wu
added a commit
that referenced
this pull request
Jun 5, 2019
* Fix jinja template bug under SA-Eventgen app * Feature timeMultiple (#141) * Adjusting interval with timeMultiple * Update issue templates add the bug report and feature request issue templates * changing the stanzas to produce data * Windbag generator/count + end=0 edge cases (#145) * Generator handling for count = -1, end = 0 working properly * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * [Docs] update contribute docs (#148) * [Docs] update contribute docs * [Docs] update the contribute * Fix make docs bug and summary anchor link error * init unittest * [Build] add ut for timeparser functions * Add more UT for config module * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * Update docs * Post-PEP8 Fixes (#157) * skip sort, gitignore 3rd party libs * fixed yapf ignore, ran format across repo * Issue 160 (#163) * Fixed timer and token * Added a conditional for end=-1 * Update eventgentimer.py * Fixed timer and token (#162) * add extendIndexes feature (#154) * add extendIndexes feature * set extendIndexes as a list value * correct log level * upate doc, change num to weight * Test fix (#168) * Add sample functional test for replay mode * Add token replacement functional tests * skip failed case * Added a timeout * created a results dir * Update version.json * Fix previous pep8 format regression bug (#171) * Fix merge conflict bug * Delete tool orca related part (#178) * add test for jinja template test (#177) * clean up the logic about sampleDir and jinja_template_dir setting Add functional test for jinja template generator Fixes #174 Fixes #167 * revert the change about token, sample mode test cases * merge the change from develop branch * merge test conf from develop branch * use urllib 1.24.2 * format with pep8 * fix bug in template dir resolution * update test case to address multiline event * fix a issue that when setup eventgen with 200+ indexers, any exceptions will terminate setup and re-writing default configure files * Fix bug #179 urlparams is not set (#181) * fix issue-183 * add modinput ft (#185) * Issue 159 (#186) * Fixed timer and token * Added a conditional for end=-1 * Added timeMultiple logic * Time Multiple Fix * Add default pull request reviewers (#190) * Default to -1 (#189) * Changed verbose -> verbosity (#191) * Update README.md (#195) Like other Splunk products - Splunk Enterprise Security, Splunk Business Workflow ... - Splunk Event Generator does not need a definitive article "The" before the product name. * update doc for friendly reading and add backward capability section. (#193) * Update index.md (#194) Update index.md * change the sampleDir setting in test cases (#196) * return status=2 when all tasks finished. (#182) * return status=2 when all tasks finished. * Change release verion to 6.3.6 (#200) * Change release verion to 6.3.6 * Fix flaky tests (#204) * Fix flaky tests * Rename modinput to mod_input to avoid functional test fail * when hit an exception, make sure egx logs it to files * Fix circleci status badges (#208) * Clean up and interval error (#211) * Fix generatorWorks not work issue (#207) * Fix navigation error with installed with splunk stream (#214) * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * Add 3rd lib in app (#210) * Add 3rd lib in app * Bugfix/197 multiprocess not working (#218) * Fix issue 197 multiprocess not working * fix issue 219 (#220) * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * Add license credits (#222) * Feature/multi indexes (#224) * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * add extendIndexes feature * set extendIndexes as a list value * upate doc, change num to weight * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * add python code style lint and format * Add more unit tests for config module * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * fix no len dict out of range * clean duplicate code * Revert "Metrics output plugin" (#226) * [issue 217]disable logging queue in multiprocess mode (#223) * disable logging queue in multiprocess mode * Fixed fileName (#229) * Issue 201 (#221) * Removing unavailable servers * Removed nonresponding httpserver * Added a counter * fixed a bug * Added docs and spec * Addressing unused var * fixing url reference * fixed currentreadsize * Httpevent str formatting * fix #166 (#192) * Change release version to 6.4.0 and add release notes (#232)
li-wu
added a commit
that referenced
this pull request
Jun 5, 2019
* Fix jinja template bug under SA-Eventgen app * Feature timeMultiple (#141) * Adjusting interval with timeMultiple * Update issue templates add the bug report and feature request issue templates * changing the stanzas to produce data * Windbag generator/count + end=0 edge cases (#145) * Generator handling for count = -1, end = 0 working properly * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * [Docs] update contribute docs (#148) * [Docs] update contribute docs * [Docs] update the contribute * Fix make docs bug and summary anchor link error * init unittest * [Build] add ut for timeparser functions * Add more UT for config module * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * Update docs * Post-PEP8 Fixes (#157) * skip sort, gitignore 3rd party libs * fixed yapf ignore, ran format across repo * Issue 160 (#163) * Fixed timer and token * Added a conditional for end=-1 * Update eventgentimer.py * Fixed timer and token (#162) * add extendIndexes feature (#154) * add extendIndexes feature * set extendIndexes as a list value * correct log level * upate doc, change num to weight * Test fix (#168) * Add sample functional test for replay mode * Add token replacement functional tests * skip failed case * Added a timeout * created a results dir * Update version.json * Fix previous pep8 format regression bug (#171) * Fix merge conflict bug * Delete tool orca related part (#178) * add test for jinja template test (#177) * clean up the logic about sampleDir and jinja_template_dir setting Add functional test for jinja template generator Fixes #174 Fixes #167 * revert the change about token, sample mode test cases * merge the change from develop branch * merge test conf from develop branch * use urllib 1.24.2 * format with pep8 * fix bug in template dir resolution * update test case to address multiline event * fix a issue that when setup eventgen with 200+ indexers, any exceptions will terminate setup and re-writing default configure files * Fix bug #179 urlparams is not set (#181) * fix issue-183 * add modinput ft (#185) * Issue 159 (#186) * Fixed timer and token * Added a conditional for end=-1 * Added timeMultiple logic * Time Multiple Fix * Add default pull request reviewers (#190) * Default to -1 (#189) * Changed verbose -> verbosity (#191) * Update README.md (#195) Like other Splunk products - Splunk Enterprise Security, Splunk Business Workflow ... - Splunk Event Generator does not need a definitive article "The" before the product name. * update doc for friendly reading and add backward capability section. (#193) * Update index.md (#194) Update index.md * change the sampleDir setting in test cases (#196) * return status=2 when all tasks finished. (#182) * return status=2 when all tasks finished. * Change release verion to 6.3.6 (#200) * Change release verion to 6.3.6 * Fix flaky tests (#204) * Fix flaky tests * Rename modinput to mod_input to avoid functional test fail * when hit an exception, make sure egx logs it to files * Fix circleci status badges (#208) * Clean up and interval error (#211) * Fix generatorWorks not work issue (#207) * Fix navigation error with installed with splunk stream (#214) * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * Add 3rd lib in app (#210) * Add 3rd lib in app * Bugfix/197 multiprocess not working (#218) * Fix issue 197 multiprocess not working * fix issue 219 (#220) * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * Add license credits (#222) * Feature/multi indexes (#224) * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * add extendIndexes feature * set extendIndexes as a list value * upate doc, change num to weight * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * add python code style lint and format * Add more unit tests for config module * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * fix no len dict out of range * clean duplicate code * Revert "Metrics output plugin" (#226) * [issue 217]disable logging queue in multiprocess mode (#223) * disable logging queue in multiprocess mode * Fixed fileName (#229) * Issue 201 (#221) * Removing unavailable servers * Removed nonresponding httpserver * Added a counter * fixed a bug * Added docs and spec * Addressing unused var * fixing url reference * fixed currentreadsize * Httpevent str formatting * fix #166 (#192) * Change release version to 6.4.0 and add release notes (#232) * fix modinput init error (#234)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pr fixes issue #197 .
multiprocessonly works on version6.3.4. And it causes great performance downgrade. This is caused by previous three fixes:httpevent.pyDetail comments are listed below.