fix #166 #192
Merged
fix #166 #192
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
li-wu
reviewed
May 5, 2019
li-wu
reviewed
May 5, 2019
li-wu
previously approved these changes
May 5, 2019
GordonWang
force-pushed
the
bug/backfill-perdayvolume
branch
from
717d126 to
29c2a9a
Compare
|
Branch is rebased. PR is updated.
|
li-wu
approved these changes
May 14, 2019
arctan5x
reviewed
May 30, 2019
|
|
||
| def _output_end(self): | ||
| pass |
There was a problem hiding this comment.
what is the purpose of this?
There was a problem hiding this comment.
this callback hook is added as a exit hook point for output plugin.
This hook is used in the counter output plugin to collect information.
The counter plugin is used for debug and issue triaging scenario.
arctan5x
reviewed
May 30, 2019
| from outputplugin import OutputPlugin | ||
|
|
||
|
|
||
| class CounterOutputPlugin(OutputPlugin): |
There was a problem hiding this comment.
How do we use this good stuff?
There was a problem hiding this comment.
if we want to use this dummy counter plugin, we can set outputMode=counter in eventgen conf stanza.
I did not update the readme doc because I think this is only used for debugging purpose. Do we have any place to hold the docs for developers? I can add some content to that place.
There was a problem hiding this comment.
What's wrong with exposing this option to customers?
GordonWang
commented
Jun 4, 2019
| CounterOutputPlugin.dataSizeHistogram[day] = CounterOutputPlugin.dataSizeHistogram.get(day, 0) + len(text) | ||
| CounterOutputPlugin.eventCountHistogram[day] = CounterOutputPlugin.eventCountHistogram.get(day, 0) + 1 | ||
|
|
||
| def _output_end(self): |
There was a problem hiding this comment.
this is where the output_end hook is used.
jmeixensperger
approved these changes
Jun 5, 2019
li-wu
added a commit
that referenced
this pull request
Jun 5, 2019
* Fix jinja template bug under SA-Eventgen app * Feature timeMultiple (#141) * Adjusting interval with timeMultiple * Update issue templates add the bug report and feature request issue templates * changing the stanzas to produce data * Windbag generator/count + end=0 edge cases (#145) * Generator handling for count = -1, end = 0 working properly * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * [Docs] update contribute docs (#148) * [Docs] update contribute docs * [Docs] update the contribute * Fix make docs bug and summary anchor link error * init unittest * [Build] add ut for timeparser functions * Add more UT for config module * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * Update docs * Post-PEP8 Fixes (#157) * skip sort, gitignore 3rd party libs * fixed yapf ignore, ran format across repo * Issue 160 (#163) * Fixed timer and token * Added a conditional for end=-1 * Update eventgentimer.py * Fixed timer and token (#162) * add extendIndexes feature (#154) * add extendIndexes feature * set extendIndexes as a list value * correct log level * upate doc, change num to weight * Test fix (#168) * Add sample functional test for replay mode * Add token replacement functional tests * skip failed case * Added a timeout * created a results dir * Update version.json * Fix previous pep8 format regression bug (#171) * Fix merge conflict bug * Delete tool orca related part (#178) * add test for jinja template test (#177) * clean up the logic about sampleDir and jinja_template_dir setting Add functional test for jinja template generator Fixes #174 Fixes #167 * revert the change about token, sample mode test cases * merge the change from develop branch * merge test conf from develop branch * use urllib 1.24.2 * format with pep8 * fix bug in template dir resolution * update test case to address multiline event * fix a issue that when setup eventgen with 200+ indexers, any exceptions will terminate setup and re-writing default configure files * Fix bug #179 urlparams is not set (#181) * fix issue-183 * add modinput ft (#185) * Issue 159 (#186) * Fixed timer and token * Added a conditional for end=-1 * Added timeMultiple logic * Time Multiple Fix * Add default pull request reviewers (#190) * Default to -1 (#189) * Changed verbose -> verbosity (#191) * Update README.md (#195) Like other Splunk products - Splunk Enterprise Security, Splunk Business Workflow ... - Splunk Event Generator does not need a definitive article "The" before the product name. * update doc for friendly reading and add backward capability section. (#193) * Update index.md (#194) Update index.md * change the sampleDir setting in test cases (#196) * return status=2 when all tasks finished. (#182) * return status=2 when all tasks finished. * Change release verion to 6.3.6 (#200) * Change release verion to 6.3.6 * Fix flaky tests (#204) * Fix flaky tests * Rename modinput to mod_input to avoid functional test fail * when hit an exception, make sure egx logs it to files * Fix circleci status badges (#208) * Clean up and interval error (#211) * Fix generatorWorks not work issue (#207) * Fix navigation error with installed with splunk stream (#214) * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * Add 3rd lib in app (#210) * Add 3rd lib in app * Bugfix/197 multiprocess not working (#218) * Fix issue 197 multiprocess not working * fix issue 219 (#220) * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * Add license credits (#222) * Feature/multi indexes (#224) * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * add extendIndexes feature * set extendIndexes as a list value * upate doc, change num to weight * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * add python code style lint and format * Add more unit tests for config module * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * fix no len dict out of range * clean duplicate code * Revert "Metrics output plugin" (#226) * [issue 217]disable logging queue in multiprocess mode (#223) * disable logging queue in multiprocess mode * Fixed fileName (#229) * Issue 201 (#221) * Removing unavailable servers * Removed nonresponding httpserver * Added a counter * fixed a bug * Added docs and spec * Addressing unused var * fixing url reference * fixed currentreadsize * Httpevent str formatting * fix #166 (#192)
li-wu
added a commit
that referenced
this pull request
Jun 5, 2019
* Fix jinja template bug under SA-Eventgen app * Feature timeMultiple (#141) * Adjusting interval with timeMultiple * Update issue templates add the bug report and feature request issue templates * changing the stanzas to produce data * Windbag generator/count + end=0 edge cases (#145) * Generator handling for count = -1, end = 0 working properly * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * [Docs] update contribute docs (#148) * [Docs] update contribute docs * [Docs] update the contribute * Fix make docs bug and summary anchor link error * init unittest * [Build] add ut for timeparser functions * Add more UT for config module * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * Update docs * Post-PEP8 Fixes (#157) * skip sort, gitignore 3rd party libs * fixed yapf ignore, ran format across repo * Issue 160 (#163) * Fixed timer and token * Added a conditional for end=-1 * Update eventgentimer.py * Fixed timer and token (#162) * add extendIndexes feature (#154) * add extendIndexes feature * set extendIndexes as a list value * correct log level * upate doc, change num to weight * Test fix (#168) * Add sample functional test for replay mode * Add token replacement functional tests * skip failed case * Added a timeout * created a results dir * Update version.json * Fix previous pep8 format regression bug (#171) * Fix merge conflict bug * Delete tool orca related part (#178) * add test for jinja template test (#177) * clean up the logic about sampleDir and jinja_template_dir setting Add functional test for jinja template generator Fixes #174 Fixes #167 * revert the change about token, sample mode test cases * merge the change from develop branch * merge test conf from develop branch * use urllib 1.24.2 * format with pep8 * fix bug in template dir resolution * update test case to address multiline event * fix a issue that when setup eventgen with 200+ indexers, any exceptions will terminate setup and re-writing default configure files * Fix bug #179 urlparams is not set (#181) * fix issue-183 * add modinput ft (#185) * Issue 159 (#186) * Fixed timer and token * Added a conditional for end=-1 * Added timeMultiple logic * Time Multiple Fix * Add default pull request reviewers (#190) * Default to -1 (#189) * Changed verbose -> verbosity (#191) * Update README.md (#195) Like other Splunk products - Splunk Enterprise Security, Splunk Business Workflow ... - Splunk Event Generator does not need a definitive article "The" before the product name. * update doc for friendly reading and add backward capability section. (#193) * Update index.md (#194) Update index.md * change the sampleDir setting in test cases (#196) * return status=2 when all tasks finished. (#182) * return status=2 when all tasks finished. * Change release verion to 6.3.6 (#200) * Change release verion to 6.3.6 * Fix flaky tests (#204) * Fix flaky tests * Rename modinput to mod_input to avoid functional test fail * when hit an exception, make sure egx logs it to files * Fix circleci status badges (#208) * Clean up and interval error (#211) * Fix generatorWorks not work issue (#207) * Fix navigation error with installed with splunk stream (#214) * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * Add 3rd lib in app (#210) * Add 3rd lib in app * Bugfix/197 multiprocess not working (#218) * Fix issue 197 multiprocess not working * fix issue 219 (#220) * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * Add license credits (#222) * Feature/multi indexes (#224) * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * add extendIndexes feature * set extendIndexes as a list value * upate doc, change num to weight * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * add python code style lint and format * Add more unit tests for config module * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * fix no len dict out of range * clean duplicate code * Revert "Metrics output plugin" (#226) * [issue 217]disable logging queue in multiprocess mode (#223) * disable logging queue in multiprocess mode * Fixed fileName (#229) * Issue 201 (#221) * Removing unavailable servers * Removed nonresponding httpserver * Added a counter * fixed a bug * Added docs and spec * Addressing unused var * fixing url reference * fixed currentreadsize * Httpevent str formatting * fix #166 (#192) * Change release version to 6.4.0 and add release notes (#232)
li-wu
added a commit
that referenced
this pull request
Jun 5, 2019
* Fix jinja template bug under SA-Eventgen app * Feature timeMultiple (#141) * Adjusting interval with timeMultiple * Update issue templates add the bug report and feature request issue templates * changing the stanzas to produce data * Windbag generator/count + end=0 edge cases (#145) * Generator handling for count = -1, end = 0 working properly * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * [Docs] update contribute docs (#148) * [Docs] update contribute docs * [Docs] update the contribute * Fix make docs bug and summary anchor link error * init unittest * [Build] add ut for timeparser functions * Add more UT for config module * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * Update docs * Post-PEP8 Fixes (#157) * skip sort, gitignore 3rd party libs * fixed yapf ignore, ran format across repo * Issue 160 (#163) * Fixed timer and token * Added a conditional for end=-1 * Update eventgentimer.py * Fixed timer and token (#162) * add extendIndexes feature (#154) * add extendIndexes feature * set extendIndexes as a list value * correct log level * upate doc, change num to weight * Test fix (#168) * Add sample functional test for replay mode * Add token replacement functional tests * skip failed case * Added a timeout * created a results dir * Update version.json * Fix previous pep8 format regression bug (#171) * Fix merge conflict bug * Delete tool orca related part (#178) * add test for jinja template test (#177) * clean up the logic about sampleDir and jinja_template_dir setting Add functional test for jinja template generator Fixes #174 Fixes #167 * revert the change about token, sample mode test cases * merge the change from develop branch * merge test conf from develop branch * use urllib 1.24.2 * format with pep8 * fix bug in template dir resolution * update test case to address multiline event * fix a issue that when setup eventgen with 200+ indexers, any exceptions will terminate setup and re-writing default configure files * Fix bug #179 urlparams is not set (#181) * fix issue-183 * add modinput ft (#185) * Issue 159 (#186) * Fixed timer and token * Added a conditional for end=-1 * Added timeMultiple logic * Time Multiple Fix * Add default pull request reviewers (#190) * Default to -1 (#189) * Changed verbose -> verbosity (#191) * Update README.md (#195) Like other Splunk products - Splunk Enterprise Security, Splunk Business Workflow ... - Splunk Event Generator does not need a definitive article "The" before the product name. * update doc for friendly reading and add backward capability section. (#193) * Update index.md (#194) Update index.md * change the sampleDir setting in test cases (#196) * return status=2 when all tasks finished. (#182) * return status=2 when all tasks finished. * Change release verion to 6.3.6 (#200) * Change release verion to 6.3.6 * Fix flaky tests (#204) * Fix flaky tests * Rename modinput to mod_input to avoid functional test fail * when hit an exception, make sure egx logs it to files * Fix circleci status badges (#208) * Clean up and interval error (#211) * Fix generatorWorks not work issue (#207) * Fix navigation error with installed with splunk stream (#214) * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * Add 3rd lib in app (#210) * Add 3rd lib in app * Bugfix/197 multiprocess not working (#218) * Fix issue 197 multiprocess not working * fix issue 219 (#220) * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * add metric_httpevent plugin * update log content in metric_httpevent * add doc for metric_httpevent * define httpevent_core and add fix eventgen-httpeventout log handler * restore samples file * restore unit test file * Add license credits (#222) * Feature/multi indexes (#224) * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * Bump version * add python code style lint and format * Add more unit tests for config module * Pep8 (#151) * Format to code standards, addressed linter errors/warnings * add extendIndexes feature * set extendIndexes as a list value * upate doc, change num to weight * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * Fix jinja template bug under SA-Eventgen app * Update docs (#146) * Updated docs, added release notes to changelog * add python code style lint and format * Add more unit tests for config module * calculate generate rate and use sequential index replacement * update dockerfile * randomize index replacement in each batch * fix no len dict out of range * clean duplicate code * Revert "Metrics output plugin" (#226) * [issue 217]disable logging queue in multiprocess mode (#223) * disable logging queue in multiprocess mode * Fixed fileName (#229) * Issue 201 (#221) * Removing unavailable servers * Removed nonresponding httpserver * Added a counter * fixed a bug * Added docs and spec * Addressing unused var * fixing url reference * fixed currentreadsize * Httpevent str formatting * fix #166 (#192) * Change release version to 6.4.0 and add release notes (#232) * fix modinput init error (#234)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.