Skip to content

Commit

Permalink
Done with docs, delete unused spec file
Browse files Browse the repository at this point in the history
  • Loading branch information
jmeixensperger committed Mar 15, 2019
1 parent 0be640a commit ca2e207
Show file tree
Hide file tree
Showing 4 changed files with 24 additions and 495 deletions.
2 changes: 1 addition & 1 deletion docs/PERFORMANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,4 +90,4 @@ As a result, for a eventgen cluster with one server and one controller, we found

In this architecture, the primary bottleneck is serializing and deserializing the data between processes. We added the reduce step of the outputqueue primarily to handle modular input and file outputs where we needed a limited number of things touching a file or outputting to stdout. Where we can parallelize this work, we can remove the majority of the CPU bottleneck of passing data around between processes.

For this reason, in July of 2014, we added a config option to disable using the outputqueue. In default/eventgen.conf, setting useOutputQueue to false or using `--disableOutputQueue` on the command line will output data directly from each GeneratorWorker rather than forcing the data back through the output queue.
For this reason, in July of 2014, we added a config option to disable using the outputqueue. Setting useOutputQueue to false or using `--disableOutputQueue` on the command line will output data directly from each GeneratorWorker rather than forcing the data back through the output queue.
32 changes: 21 additions & 11 deletions docs/PLUGINS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Plugin Architecture

As of version 3.0, Eventgen now allows for plugins which extend our core functionality. There are three types of Plugins:
Eventgen allows for plugins which extend our core functionality. There are three types of Plugins:

* Output
* Output plugins take generated lists of events and send the events to a specific target
Expand All @@ -12,7 +12,8 @@ As of version 3.0, Eventgen now allows for plugins which extend our core functio

## Anatomy of a Plugin

Plugins inherit from a OutputPlugin class and are placed in their appropriate directory, either in Eventgen app itself or inside another Splunk App's ``lib/plugins/<type>`` directory. Let's take a look at the simplest plugin available to us, the Devnull output plugin:
Plugins inherit from a base plugin class and are placed in their appropriate directory, either in Eventgen app itself or inside another Splunk App's ``lib/plugins/<type>`` directory.
Let's take a look at the simplest plugin available to us, the Devnull output plugin:

```python
from __future__ import division
Expand All @@ -37,13 +38,14 @@ def load():
return DevNullOutputPlugin
```

First, we import the OutputPlugin superclass. For output plugins, they define a constant MAXQUEUELENGTH to determine the maximum amount of items in queue before forcing a queue flush.
First, we import the OutputPlugin superclass. For output plugins, they define a constant MAXQUEUELENGTH to determine the maximum amount of items in queue before forcing a queue flush.

``__init__()`` is very simple. It calls its superclass init and sets one variable, firsttime. ``flush()`` is also very simple. If it's the first time, open the file /dev/null, otherwise, output the queue by writing it to the already open file.
``__init__()`` is very simple. It calls its superclass init and sets one variable, firsttime. ``flush()`` is also very simple.
If it's the first time, open the file /dev/null, otherwise, output the queue by writing it to the already open file.

Every Eventgen plugin defines a class and a ``load()`` method. The load() method is a universal function for determinig the class defined in the file.

Now, let's look at a slightly more complicated plugin, splunkstream.py in ``lib/plugins/output/splunkstream.py``. We're going to look just at the top of the class as its being defined:
Now, let's look at a slightly more complicated plugin, splunkstream.py in ``lib/plugins/output/splunkstream.py``. We're going to look just at the top of the class as its being defined:

```python
class SplunkStreamOutputPlugin(OutputPlugin):
Expand All @@ -57,11 +59,14 @@ class SplunkStreamOutputPlugin(OutputPlugin):
MAXQUEUELENGTH should look normal, but these other class variables need a little explanation.

### Configuration Validation
Config validation is a modular system in Eventgen, and plugins must be allowed to specify additional configuration parameters that the main Eventgen will consider valid and store. *Note that eventgen.conf.spec generation is not yet automated, which means plugins must ship with the default distribution and eventgen.conf.spec must be maintained manually.* Eventually spec file generation will be automated as well.
Config validation is a modular system in Eventgen, and plugins must be allowed to specify additional configuration parameters that the main Eventgen will consider valid and store.
*Note that eventgen.conf.spec generation is not yet automated, which means plugins must ship with the default distribution and eventgen.conf.spec must be maintained manually.*
Eventually spec file generation will be automated as well.

The main configuration of Eventgen validates itself by a list of configuration parameters assigned by type, and each of the configuration parameters is validated by that type. The settings list is required:
The main configuration of Eventgen validates itself by a list of configuration parameters assigned by type, and each of the configuration parameters is validated by that type.
The settings list is required:

* validSettings | Defines the list of valid settings for this plugin
* validSettings | Defines the list of valid settings for this plugin

The following lists are optional and likely to be used by many plugins:

Expand Down Expand Up @@ -116,9 +121,13 @@ def load():
return WindbagGenerator
```

For this generator plugin, notice we inherit from GeneratorPlugin instead of OutputPlugin. This plugin is also quite simple. In its ``__init__()`` method, it calls the superclass ``__init__()`` and it sets up two global variables, c, which holds the config (and is a Singleton pattern which can be instantiated many times) and a copy of the logger which we'll use for logging in most plugins.
For this generator plugin, notice we inherit from GeneratorPlugin instead of OutputPlugin. This plugin is also quite simple.
In its ``__init__()`` method, it calls the superclass ``__init__()`` and it sets up two global variables, c, which holds the config
(and is a Singleton pattern which can be instantiated many times) and a copy of the logger which we'll use for logging in most plugins.

Secondly, it defines a gen() method, which generates ``count`` events between ``earliest`` and ``latest`` time. In this case, we ignore the timestamp and return just event text. Then we call bulksend. This plugin has several performance optimizations: using a list constructor instead of a loop and using bulksend instead of send. Let's see how this could be implemented in a slightly less performant but easier to understand way:
Secondly, it defines a gen() method, which generates ``count`` events between ``earliest`` and ``latest`` time. In this case, we ignore the timestamp and return just event text.
Then we call bulksend. This plugin has several performance optimizations: using a list constructor instead of a loop and using bulksend instead of send.
Let's see how this could be implemented in a slightly less performant but easier to understand way:

```python
def gen(self, count, earliest, latest):
Expand All @@ -132,4 +141,5 @@ Here, we use ``send()`` instead of ``bulksend()`` and a loop to make it easier t

# Shipping a Plugin

When you've developed a plugin that you want to use in your app, shipping it with your app is easy. Place any Eventgen plugin in your Splunk app's ``bin/`` directory and we'll search for and find any plugins referenced by a ``outputMode``, ``generator`` or ``rater`` config statement.
When you've developed a plugin that you want to use in your app, shipping it with your app is easy.
Place any Eventgen plugin in your Splunk app's ``bin/`` directory and we'll search for and find any plugins referenced by a ``outputMode``, ``generator`` or ``rater`` config statement.
6 changes: 2 additions & 4 deletions docs/SETUP.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
## Install ##

There are a multiple ways to using Eventgen, and you should choose the method that best fits your use case. Below are the two major ways to use Eventgen - as a PyPI module and as a Splunk App. Follow the instructions below depending on your ideal use:
There are multiple ways to use Eventgen, and you should choose the method that best fits your use case.
Below are the two major ways to use Eventgen - as a PyPI module and as a Splunk App. Follow the instructions below depending on your ideal use:

* Install / Use Eventgen as a [Splunk App](#splunk-app-installation)

Expand Down Expand Up @@ -33,9 +34,6 @@ There are a multiple ways to using Eventgen, and you should choose the method th
* You have to run the "build" command to produce a Splunk app
* Harder to troubleshoot (especially in multiprocess mode)




---

## PyPI Installation / First Run #####
Expand Down
Loading

0 comments on commit ca2e207

Please sign in to comment.