forked from mozilla-services/heka
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGES.txt
1394 lines (910 loc) · 47.6 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
0.11.0 (2015-??-??)
Backwards Incompatibilities
---------------------------
Features
--------
Bug Handling
------------
0.10.0 (2015-??-??)
===================
Backwards Incompatibilities
---------------------------
* Major overhaul of filter and output plugin APIs to support disk buffering
(#1378).
* Go 1.4 now required for building.
* Deprecated the read_next_field() sandbox API; it is targeted for removal in
0.11.0 (#1602).
* Removed unused PipelinePack.Decoded attribute.
* LogOutput will write data to stdout instead of stderr (#1515).
* Using stftime literals for filenames during rotation in FileOutput plugin
(#1469).
* Implemented stftime format codes in: filenames in FileOutput plugin,
ESJsonEncoder, ESLogstashV0Encoder, Payload encoder (#1469, #1508).
* The package created by 'make deb' creates an "heka" user and ships an init
script and a systemd unit.
* The 'make deb' target requires fakeroot and debhelper to be installed.
* SplitterRunner interface now provides a `Done` method that should be called
whenever the splitter is no longer needed.
* `"Fields"` choice for the `fields` setting renamed to `"DynamicFields"` for
both ESJsonEncoder and ESLogstashV0Encoder.
Features
--------
* Added support for `write_message` to delete fields when passed nil values.
* Added a timer_event_on_shutdown configuration option for Filter/Output
sandboxes (#1460).
* Added `replace_dot` setting to StatFilter.
* A protobuf encoding of the current message is now always stored in
pack.MsgBytes prior to injection into the message router (#265).
* ProtobufEncoder now just copies the pack.MsgBytes into a new byte slice and
returns that.
* Added Linux CPU Stats Decoder and CPU Stats Filter.
* Added a new sandbox encoder, Schema InfluxDB Write Encoder, which updates
compatibility of sending data in Heka message fields to InfluxDB 0.9.0+
write API. It is required to use this encoder when integrating Heka with an
InfluxDB 0.9.0+ instance as the API to commit metrics has changed.
* Added support for abstract Unix domain sockets to the UdpInput.
* Added support for `can_exit` to inputs (defaults to false, except on
ProcessDirectoryInput spawned processes where it defaults to true)
* Added `field_mappings` setting to the ESJsonEncoder allowing custom names for
Heka message fields in ElasticSearch.
* Termination messages are now generated for SandboxFilters that fail
initialization by a SandboxManagerFilter.
* Added alert_throttle setting to HTTP status graph filter.
* Added DockerEventInput.
* Added splitters to the reports displayed by Heka after SIGUSR1 signal.
* Added `graphite` module with helpers allowing to generate graphite metrics
for counters and timeseries (#1461).
* Added Basic Auth and API key authentication to HttpListenInput (#1533).
* Added SSL/TLS support to HttpListenInput (#1534)
* Allow to overwrite ContainerName using a Container environment variable from
within the Docker container for the DockerLogInput. (#1545)
* Allow extracting environment variables from a Docker container to use as
fields for the DockerLogInput. (#1569)
* Dashboard now shows sandbox plugins in alphabetical order.
* Added 'dynamic_fields' config to ESJsonEncoder for selecting a subset of
Fields values
Bug Handling
------------
* Fixed visibility of synchronous decoders in reports (#1312).
* Fixed hang on SandboxFilter termination (#1509)
* More ProcessInput/ProcessDirectoryInput retry logic fixes (#1412 & #1418).
* ProcessInput fixed to no longer leak decoder goroutines when reconfigured
via ProcessDirectoryInput-driven config changes (#1444).
* Check configuration, ticker_interval in Stat Accumulator Input must be
greater than 0 (#1474).
* Switch from unmaintained `crowdmob` fork of GoAMZ dependency to `AdRoll`
(#1458).
* Added ability to reconnect after lost connection in `heka-flood` (#1536).
* Respect ElasticSearch URL path (#1558).
0.9.3 (2015-??-??)
==================
Bug Handling
------------
* Correctly honor "user-agent" heading config in HttpInput (#1520).
* Removed state from PluginMaker to remove race conditions during plugin
construction (#1532).
* Get decoder lock before cleaning up decoders during pipeline shutdown to
avoid race condition panics during exit (#1531).
0.9.2 (2015-04-22)
==================
Bug Handling
------------
* Added bounds check for truncated inputs lines to StatsdInput.
* Fixed ElasticSearch recovery after full queue when `queue_full_action` is
set to "drop" or "block".
* Fixed TcpOutput recovery after full queue when `queue_full_action` is set to
"drop" or "block" (#1484).
* Fixed bug where LogstreamerInput wasn't honoring `oldest_duration` setting
(#1437).
* ElasticSearch payload encoder will ensure there is a newline in the
end of the payload in order for the bulk API to work correctly (#1457).
* Fix a gzipped file seeking in logstreamer may cause an OOM exception.
* Fixed config parsing typo bug in heka-logstreamer command (#1436).
* Always check for decoder existence when a decoder is specified for an input
plugin (#1439).
* Check `IsStoppable()` on all input, filter, output plugins if they error at
startup time, only shut down Heka if false.
* Fixed typo causing panic in AMQPInput when splitter returns an error
(#1453).
* Handle chunked error responses in http_output.
* ElasticSearchOutput buffering now only retries messages when it's clear that
the problem related to failed communication with the ElasticSearch server
(#1401).
* Added handling of incomplete/trailing data for SplitBytes() method in
SplitterRunner (#1455)
0.9.1 (2015-03-13)
==================
Features
--------
* Added configurable max_message_size to heka-cat.
* Added `count` setting to TokenSplitter to allow splitting on every Nth
instance of the given delimiter.
* Added `deliver_incomplete_final` setting to SplitterRunner.
* Added `max_message_size` setting to UdpOutput.
Bug Handling
------------
* Fix the message matcher parser to clear the yySymType structure on each call
(#1409).
* Protect read_message("raw") from reading an empty pack.MsgBytes (#1405).
* Moved `SetSplitterRunner` out of the config's MakeRunner and into the
`NewSplitterRunner` function to make sure a splitter always has access to a
SplitterRunner if needed.
* Fixed SplitterRunner buffer readPos so it only increments when a read call
doesn't return an error (#1367).
* Ensured plugin list file is closed after writing to it in Dashboard plugin.
* Fixed exiting / restart behavior for ProcessInput to match documentation
(#1395).
* UdpOutput drops message if its size exceeds allowed UDP data size (#1393).
0.9.0 (2015-02-25)
==================
Backwards Incompatibilities
---------------------------
* Major overhaul of Heka's configuration loading code. This doesn't impact
most plugins, it's only a breaking change for plugins that happen to
instantiate and manage the lifecycles of other embedded plugins, e.g.
MultiDecoder, ProcessDirectoryInput, SandboxManagerFilter.
* All inputs now support `decoder`, `synchronous_decode`, and
`send_decode_failures` config options, automatically extracted by Heka's
config system.
* InputRunner now handles message decoding and delivery to the router,
according to the specifications of the input's configuration. This is
accomplished either directly via the `Deliver` method or, in cases where
decoding might need to happen in separate goroutines, through `Deliverer`
objects available from the `NewDeliverer` method.
* Hekad and all clients started using stdout for informational messages and
stderr for error messages.
* Stats Accum Input treats gauge inputs as float64 rather than int64 to match
statsd spec and other statsd implementations (#850).
* Introduction of Splitter plugins, accompanied by major changes to how input
plugins work to support the use of Splitter plugins. (#424)
* TcpInput and UdpInput now set a non-protobuf encoded message's Type field to
the configured input plugin name instead of the hard-coded string
"NetworkInput".
Bug Handling
------------
* Fix the shutdown hang caused by a Filter/Output plugin failure (#1324).
* Fix the RPM dependency errors on Centos7 (#1311).
* Don't output empty messages when use_framing = true (#1326).
* KafkaInput uses the oldest offset available instead of zero when no
checkpoint file exists and the offset method is set to manual. KafkaInput
also correctly closes the checkpoint file before removing it when it is
invalid (#1325).
* Handle empty byte fields in the sandbox interface (#1284).
* Reset header when discarding valid but oversized messages (#1221).
* Prevent the protobuf stream encoder from creating messages over
MAX_MESSAGE_SIZE (#1204).
* AMQPOutput now uses defined constants for delivery mode instead of hard
coded (and wrong) integer values (#1235).
* Hekad now respects --config parameter and loads default config path
instead of printing help and exit (#1239).
* MessageProtoParser now handles EOF gracefully, and returns all
complete records up to the end of the stream without having to
use GetRemainingData() (#1305).
* HttpListenInput no longer URL unescapes the HTTP request body (#1124).
* SmtpOutput now encodes email subject when necessary (#1277).
* All config files are now pre-loaded before any of the config is actually
loaded to prevent default plugins from being registered for each separate
config file, possibly overriding custom config.
Features
--------
* Added decode_message to the sandbox API (#1344).
* Added a SandboxInput plugin (#1333).
* Added a SandboxOutput plugin (#1303).
* `heka/plugins/amqp` package now exposes `GetAmqpHub` and `NewAMQPDialer`
functions for use by external packages.
* Added heartbeat monitoring sandbox filter plugin.
* Include the new lua_sandbox with support for shared library modules.
* Allow the sandbox process_message function to set the last error string
when it returns (#1191).
* Improve the sandbox inject_* error messages (#1156).
* Added ability to specify unit type and aggregation methods for stats graph
filter.
* Added option (immediate_start) to start process immediately in ProcessInput
(#1131).
* Added `hostname` setting to global `[hekad]` config options to allow
overriding the hostname value provided by Go's `os.Hostname` call (#1123).
* Added 'connect_timeout' option in ElasticSearch output plugin.
* Slightly improve error output in ElasticSearch plugin.
* Added support for HTTP authentication to ElasticSearchOutput.
* TcpOutput's disk buffering now supports specification of a max buffer size
and an action to take if the max size is reached, from 'shutdown', 'drop',
or 'block' (#1110).
* Added `send_interval` setting to SmtpOutput.
* Added `timestamp` format setting to ESLogstashV0Encoder (#1142).
* LogstreamerInput now uses first two "magic" bytes to identify gzip files
instead of relying on `.gz` suffix.
* Added on-disk buffering to ElasticSearchOutput. It is enabled by default.
* Added TLS support to ElasticSearchOutput (#1259).
* Added ability for HttpListenInput to capture specified HTTP request headers
and write them as message fields.
* Added ability for Statsd input handler to treat a malformed stat line
one at time without skipping all the "good" stats in a multi-line input
(more like statsd itself).
* Added `message_interval` setting support to heka-flood test configuration.
* Added `cert_path` setting for TLS support to DockerLogInput.
* Added new fields to messages generated by HttpListenInput, including
RemoteAddr, Path, EnvVersion, Hostname (the name of the server that
handled the request), and Host (the host to which the client sent the
request) (#1328).
* Added `hostname_keep` option to RsyslogDecoder (#1338).
* Added an option to Schema InfluxDB Encoder to send the data to InfluxDB as
a list of series rather than the default of a single series for all fields
in the message.
* Added an option to Schema InfluxDB Encoder that excludes all base fields
from being sent to InfluxDB to reduce the network traffic and storage
demands if these fields aren't useful.
* Added `read_only` setting to AMQPInput to support read-only users.
* Added ability to specify MAX_MESSAGE_SIZE via configuration file of hekad
and heka-flood (#1208).
* Changed default plugin_chansize setting from 50 to 30 (#1242).
0.8.3 (2015-01-08)
==================
Bug Handling
------------
* Fixed LogstreamerInput to use a separate DecoderRunner for every
LogstreamInput created, rather than sending all of the streams through a
single decoder.
0.8.2 (2015-01-06)
==================
Bug Handling
------------
* Fix rsyslog sandbox decoder test to use current year and location for
timestamp parsing, since syslog timestamp format makes that assumption.
* Ensure that geoip_decoder is not included in release binaries.
0.8.1 (2014-12-17)
==================
Bug Handling
------------
* Fix leaky file descriptor bug on http_output.go.
* Only stamp PipelinePack diagnostics in cases when a pack actually matches a
message matcher, instead of for every matcher every time (#1167).
* StatsdInput allows to specify (via max_msg_size option) size of message read
from UDP (#1165).
* AMQPOutput now recycles packs even when a publish error causes the output to
exit / restart (#1178).
* Fixed HttpOutput TLS section parsing (#1163).
* Fixed LogstreamInput extraneous journal saves that caused high disk IO
when EOF is reached.
0.8.0 (2014-10-29)
==================
Backwards Incompatibilities
---------------------------
* Audited the use of Logger and Type headers on internal message (#1024). These
changes may break some existing message matchers with regards to
plugin/sandbox termination messages.
* Sandbox decoder linux_cpustats has been renamed to linux_loadavg and the
corresponding filter cpustats was renamed to loadavg (#1094).
* ESLogstashv0 encoder will now set the @type field to match the ES record
type specified in the `type_name` setting. This is Logstash behaviour in
V0. You can revert back to the old method with `use_message_type = true`
Bug Handling
------------
* linux_diskstats.lua will now parse a line with no leading space (#1141).
* Protect globals.Stopping with a RWMutex.
* Added support for `fields` config setting to ESLogstashV0Encoder; it was
already documented, but hadn't been implemented (#1096).
* Fixed deadlock race condition in AMQPOutput shutdown sequence (#824).
Features
--------
* Added Kafka Input/Output plugins (#1148).
* Optimized prefix/suffix testing in the message matcher (#1040).
* LogstreamerInput now supports seekjournal file hashing for files that are
less than 500 bytes in length; hash is generated against file contents with
0 bytes prepended to make 500 bytes of hash content (#972).
* Support environment variables in config files (#1023).
* Add Dockerfiles and example Docker usage to repo
* Encoders can now return (nil, nil) to express that they've swallowed the
input message without generating any output.
* SandboxEncoder now supports return value of -2 from the process_message call
to specify that no output is being generated.
* Added schema_influx.lua encoder.
* Added CBUF Librato Encoder.
* Added option to disable re-using TCP connections to the ElasticSearch output
* Added separate env loading file and support for NUM_JOBS env var to Windows
build (#971).
* Added DockerLogInput (issue #1092).
* Added support for "Epoch", "EpochMilli", "EpochMicro", and "EpochNano"
timestamp formats to the time parsing code used by the Payload*Decoder
plugins (#963).
* Added ability to drop big message entirely or to keep first part of it (#1134)
0.7.3 (2014-10-28)
==================
Bug Handling
------------
* Fail gracefully on LogstreamerInput missing `file_match` setting (#1105).
* Fixed issue where ProcessInput wasn't propagating environment variables and
working directory to repeatedly run (vs. one time, long running) processes
(#1108).
* UdpInput now removes a Unix datagram socket file at shutdown time if one is
created at startup.
* Fixed issue with keeping previous EOF state in LogStreamer (#1119).
* ProcessInput now honors documented restart behavior (#1117).
0.7.2 (2014-10-01)
==================
Bug Handling
------------
* Fail more gracefully if user doesn't specify a scheme in the server URL for
the ElasticSearchOutput (#1069).
* Correctly compute http timeout interval for ElasticSearchOutput when using
HTTP indexing.
* Fixed issue w/ inaccurate payload sizes in heka-flood.
* Allow severity to be modified by the MessageTemplate logic for use in
ScribbleDecoder (#1084).
* Render pack diagnostic idle time in seconds (as specified) instead of
nanoseconds.
* Prevent orphaned matchers from remaining in the router's matcher lists in
cases where a filter or output registered in one TOML file overrides a
filter or output of the same name in an earlier TOML file.
0.7.1 (2014-09-02)
==================
Bug Handling
------------
* Fixed handling of TcpInput and TcpOutput keep_alive_period default value
handling, and added docs (#1054).
* Fixed OSX lua_sandbox build error.
* Fixed load ordering of nested MultiDecoders (#1045).
0.7.0 (2014-08-27)
==================
Backwards Incompatibilities
---------------------------
* Switched to using fork of gomock, import location changed from
`code.google.com/p/gomock` to `github.com/rafrombrc/gomock`.
* Move *GlobalConfigStruct out of pipeline package's global namespace, it is
now stored as an attribute on PipelineConfig. Any code that used to call
pipeline.Globals() now must instead access *PipelineConfig.Globals. The
`WantsPipelineConfig` interface has been introduced to give plugins access
to the active *PipelineConfig if it's needed in the ConfigStruct or Init
methods (#951).
* Removed deprecated `pipeline.GetHekaConfigDir` API call, which has been
replaced by `PrependBaseDir` and `PrependShareDir` for a few versions
now.
* Removed the regular expression template support from the message matcher
(issue #960).
* Updated the AMQP input / output configuration values to use underscore
delimited words for long config options (issue #953).
* LogstreamerInput now errors on nonexistent `log_directory` instead of
creating the folder (issue #1066).
Bug Handling
------------
* Fixed the MySQL slow query grammar to handle logs with no newline.
* Fixed packet tracking idle packs error output formatting.
* Prevent panics during shutdown when a restarting plugin has been restarted
(issue #957).
* Added support for a Logstreamer symbolically linked log_directory root
(issue #741).
* Fixed lots of race conditions in the tests.
* Eliminated race condition in use of global AMQPHub in AMQP input / output
plugins (#953).
Features
--------
* Added StatMetric Influx Encoder.
* Added RstEncoder.
* Added support for '__ignore_root' tag in add_external_plugin to allow
external packages to be added which do not have any .go files
in their root directory (issue #955).
* Switched from using goprotobuf to gogoprotobuf with marshal_all and
unmarshal_all extensions for significant performance increase.
* Added stats graph sandbox filter to make it easier to generate graphs of
statsd / graphite metrics (issue #966).
* Added support for the majority of repository URI formats when cloning external
Go packages (issue #937).
* Added the ability to test for field existence in message matcher (issue #958).
* Added support to ESJSonEncoder and ESLogstashV0Encoder to properly encode
field arrays.
* Added a filter to monitor all process message failures (issue #948).
* Added UdpOutput.
* Added IrcOutput.
* Added support for filter and output plugins to optionally exit without
causing Heka to shutdown via config option.
* Added support for custom HTTP Headers in HttpListenInput and DashboardOutput.
0.6.1 (2014-08-27)
==================
Bug Handling
------------
* CarbonOutput using UDP transport now uses multiple packets to send stats
data to Carbon server when output would be longer than the 64KiB max size
for UDP packets (issue #1035).
* StatsdInput now trims all whitespace around incoming stat names instead of
just trailing newlines (issue #1011).
* Fixed silent failures in ElasticSearchOutput bulk indexing operations.
* Fixed permanent loss of HTTP connection in certain cases when ElasticSearch
has been restarted (issue #1008).
* Fixed default sandbox script type when loading dynamic plugins.
0.6.0 (2014-07-09)
==================
Backwards Incompatibilities
---------------------------
* Go 1.3 now required for building.
* Changed handling of message stream framing. Before stream framing was
presumed to always be required when encoding with protocol buffers, and not
required when using other encodings. Now all outputs support an optional
`use_framing` setting that will determine whether or not framing will be
applied when using the OutputRunner's Encode method (issue #922).
* All files in a config directory that do not end in ".toml" will now be
ignored and not loaded as a part of the Heka configuration (issue #750).
* Removed the PayloadJsonDecoder which is replaced by the Lua sandbox cjson
module (issue 826).
* Removed LogfileInput and LogfileDirectoryManagerInput which were replaced
by LogstreamerInput in 0.5.0 (issue 914).
* MultiDecoder no longer sets prepends name of the decoder instance to the
message type value for every message. This was impacting performance, is
almost never what is needed, and is almost always overwritten by the nested
subdecoders anyway.
* MultiDecoder now has a `subs` option that refers to other top level decoder
config definitions, instead of nested sub-configs and a separate `order`
option specifying the order. This makes for less typing and the ability to
reuse decoder definitions as both standalone decoders and across multiple
MultiDecoders (issue #485).
* ElasticSearchOutput now uses an Encoder plugin instead of MessageFormatters.
Created ESJsonEncoder, ESLogstashV0Encoder, and ESPayloadEncoder
(implemented as a SandboxEncoder in es_payload.lua) to replicate prior
formatter behavior.
* Changed the Lua sandbox API. inject_message is no longer overloaded, output
has been renamed to add_to_payload, and inject_payload has been introduced.
* Changed TcpOutput, FileOutput, and LogOutput to use Encoder plugins for
output formatting instead of bespoke formatting implementations.
* The rsyslog decoder now further parses the %SYSLOGTAG% variable. The
fields.syslogtag no longer exists and is replaced by fields.programname and
message.Pid (issue #677).
* Removed no-longer-used 'decoder_poolsize' global config setting.
Bug Handling
------------
* Fixed the ElasticSearchOutput to always use UTC times in the bulk API header
(issue #504).
* Fixed the SandboxDecoder panic when failing the Decode() after successfully
injecting a message (issue #910).
* StatsdInput no longer spins up a new goroutine for each stat (issue #359).
* MultiDecoder no longer prevents shutdown if a nested SandboxDecoder crashes
during startup (issue #896).
* MultiDecoder using `cascade_strategy = "all"` now passes all generated packs
to all nested decoders instead of skipping the remainder if an earlier pack
in the sequence fails to decode (issue #896).
* MultiDecoder no longer tries to recycle the original pack, leaving that job
to the DecoderRunner as intended (issue #896).
* Updated SmtpOutput to use a slice instead of a map to generate SMTP headers
so the order will remain consistent when using Go 1.3 or later.
* Use FindProcess instead of syscall.Kill to make the 'pid_file' configuration
setting work on Windows (issue #807)
* Fix the SandboxFilter ReportMsg panic on termination/shutdown (issue #816)
* MultiDecoder with no 'order' set now fails on init instead of panicking on
first message.
* Fix panic on SIGUSR1 caused by no reports in a given plugin category (issue
#832).
* Fix file_match config of plugin logstreamer_input (issue #893).
* StatFilter now correctly handles values from integer or float fields (issue
#612).
Features
--------
* Added general purpose HttpOutput (issue #820).
* Added message processing stat reporting to the MultiDecoder for each
subdecoder and in aggregate (issue #719).
* Added memory statistics to the Heka report output, for self monitoring.
* Added a message type configuration option to the rsyslog decoder (issue #907).
* Added `elasticsearch` Lua module to generate BulkAPI index JSON (issue #875).
* Added support for Lua sandbox preservation versioning (issue #701).
* Added support for unix datagram sockets to the UdpInput using net "unixgram"
(issue #790).
* Add LogstreamInput seekjournal reporting to dashboard output (issue #445).
* Added a HyperLogLog library to the Lua sandbox using the Redis
implementation http://antirez.com/news/75.
* Added an alert encoder to make the alert messages easier to read.
* Introduced Encoder plugin type (issue #417).
* Added a bloom filter to the Lua sandbox and created a unique items filter.
* Turned the MySQL slow query log examples into a deployed decoder and filter.
* Added an Nginx error log decoder (issue #785)
* Added the ability to preserve the webserver log line in message payload
(issue #784)
* Added the optional 'pid_file' configuration setting (issue #777).
* Add anomaly, alert, annotation modules (issue #677)
* Added 'sample_denominator' global config setting to allow tweaking the
sample interval when computing timing of certain operations, replacing prior
hard-coded DURATION_SAMPLE_DENOMINATOR constant (issue #625).
* Added an Apache access log decoder based on the Apache 'LogFormat'
configuration directive.
* Added BufferedOutput. Extracts the queuing functionality out of TcpOutput.go
into a general purpose lib for use in any output module. Callers get
messages buffered to disk while another goroutine consumes and forwards
data. Any errors encountered can cause the sending goroutine to backoff and
resend data.
* Added http_timeout to elasticsearch output to prevent slow or stale
connections from holding up the flow. (issue #769)
* Add query parameters to the Message as Fields in the HttpListenInput
* Add QueueTTL option to AMQPInput to allow specifying message expiration.
* Added optional tls sub-section to AMQPInput and AMQPOutput for configuring
AMQPS TLS settings.
* Add detection and handling of gzipped files to LogstreamerInput (issue #648).
* Added optional TCP Keep-Alive parameters to TcpInput/TcpOutput plugins.
0.5.2 (2014-05-16)
==================
Bug Handling
------------
* FileOutput no longer panics when using `format = "text"` and payload is nil
(issue #843).
* Fix SandboxDecoder pass-through case so decoders that only use write_message
and not inject_message will emit messages correctly (issue #844).
* Fixed TcpInput so it will only override the default delimiter for a
RegexpParser if a delimiter is specified in the config.
* Fix the FileOutput panic when HUP'ed. Modified the CheckWritePermission
utility function to use a unique filename for each check. (issue #808)
* Terminated plugins are now removed from the SandboxManager quota (issue
#774).
* Fixed SIGUSR1 triggered text report to stdout to work with updated
heka.all-report JSON data structure (issue #762).
* MultiDecoder now uses message.GetType() instead of Message.Type so
emitted message type will be generated correctly (issue #761).
* LogstreamerInput no longer causes Heka to fail to start on empty or
whitespace-only journal files (issue #755).
* The cbufd_host_aggregator filter now properly reclaims expired hosts.
* Pull in a new lua_sandbox to fix JSON encoding of empty strings in the
sandbox plugin output() call.
* Escape regexp meta characters (notably '\') to prevent a panic in the LogstreamerInput on Windows
0.5.1 (2014-03-18)
==================
Bug Handling
------------
* Skip *.bak, *.tmp, *~, and .files in a config dir as a non-breaking band-aid
until we require an explicit naming convention in 0.6 (see issue #750).
* heka-logstreamer command now supports config directories in addition to
single config files, just like hekad itself (issue #742).
* Logstreamer package's NewLogstreamSet function no longer lowercases the
match part names when constructing match translation maps since the
PopulateMatchParts method doesn't actually expect the names to be
lowercased.
* Added support for use of "missing" value in Logstreamer translation maps to
allow users to place missing values at the end of the list instead of the
beginning (issue #735).
0.5.0 (2014-03-06)
==================
Backwards Incompatibilities
---------------------------
* ProcessInput no longer supports a separate'name' config setting, it uses the
specified plugin name from the TOML section header like all of the other
plugins.
* Removed Stdout_chan and Stderr_chan from ManagedCmd and CommandChain, client
code now has direct access to Stdout_r and Stderr_r io.Reader providers.
* The PluginHelper interface DecoderRunner prototype has changed (issue #717)
to allow for the base name e.g. "ProtobufDecoder" and a full instance name
e.g. "MyInput-ProtobufDecoder" of the decoder to be specified. This allows
multiple decoders of the same type to show up on the DashboardOutput and
sandbox decoder state preservation to work properly. Also the DecoderRunner
UUID interface method was removed.
* FileOutput `flushinterval` config setting changed to `flush_interval` to
match config naming conventions.
* SandboxDecoder, SandboxFilter, and SandboxManagerFilter now all use
`${SHARE_DIR}/lua_modules` as the default `module_directory` setting.
SandboxDecoder and SandboxFilter both now interpret relative paths to lua
source code to be relative to ${SHARE_DIR} instead of ${BASE_DIR}.
* DashboardOutput `static_directory` setting now defaults to
`${SHARE_DIR}/dasher` instead of `${BASE_DIR}/dashboard_static`.
* The sandbox preservation data is now stored in the
{base_dir}/sandbox_preservation directory instead of with the plugin source.
On the initial restart no preserved data will be restored unless it is
manually moved to this directory first. (issue #626)
* The Heka utilities (flood, sbmgr, sbmgrload, inject) have been namespaced
with a `heka-` prefix for their respective binaries. I.e. Flood has been
renamed heka-flood., etc.
* MultiDecoder now gets its name from the TOML section name like the rest of
the plugins, instead of a separate 'name' config option.
* Major reorganization of the `pipeline` package, moving the implementation of
most plugins to sub-packages of a separate `plugins` package.
* Removed the wrapper 'table' element from the JSON serialization (issue #525)
i.e., {"table":{"value":1}} is now simply {"value":1}. The change also
removes the special '_name' metadata arribute; the top level _name element
should be created in the Lua structure if it is required.
* In the process of removing the core sandbox code from Heka (issue #464), the
sandbox was streamlined to only load the base library by default. All sandbox
plugins must now explictly load additional libraries with the require
function.
* Removed DecoderSet method from PluginHelper interface (and DecoderSet
abstraction entirely) and replaced it with Decoder and DecoderRunner methods
that return a Decoder or a DecoderRunner by name.
* Changed Decoder interface to support one input pack generating multiple
output packs.
Bug Handling
------------
* Network parsers now return all records available in the parse buffer
(issue #732).
* TcpInput now stops a given connection's decoder when the connection is
closed, preventing memory pooling (issue #713).
* StatsdInput now doesn't fail with multiple stats in a single UDP packet.
* Set default StatAccumInput stat namespace prefix values even when
`legacy_namespaces` is set to false (issue #630).
* Fixed cpuprof file from being closed right after opening so no data was
being logged.
* Fixed LogfileInput so it will only override the default delimiter for a
RegexpParser if a delimiter is specified in the config.
* PluginWrapper will now check for the WantsName interface when creating a
plugin, and will set the plugin's name if appropriate.
* SandboxDecoder now explicitly logs a fatal error before triggering shutdown
to ensure error message is actually emitted.
* Message severity now defaults to the highest RFC 5424 value (i.e. 7) implies
low severity, rather than zero, which implies `emergency`, (issue #518).
* 'flood' command now outputs every send error (even 'connection refused'), and
always increments 'messages sent' count even when there is a sending error
(so setting 'num_messages' still works even if hekad stops responding,
etc.).
* stat_accum_input will not fail when flushing a timer where the percentile
is equal to the min value.
Features
--------
* Added ProcessDirectoryInput.
* InputRunners now support being specified as 'transient', meaning their
lifespan should be managed by the code that creates the InputRunner and not
Heka's pipeline code.
* HttpInput: now supports configuring the HTTP method, HTTP headers and HTTP
Basic Authentication
* TLS Listeners can specify a 'client_cafile' which limits the CAs that a
client cert can be chained to. This provides a mechanism for TLS Client AUTH.
* TLS Senders can specify a 'root_cafile' which limits the CAs that a
server cert can be chained to. This provides a mechanims for TLS Server AUTH.
* Added StopDecoderRunner function to PluginHelper API so inputs can manually
decommission decoders when they're no longer being used.
* The SandboxManagerFilter can now control the sandbox usage limits (issue #95)
* Added support for send_nsca to NagiosOutput as an alternative to direct http
submission; also a way to explicitely specify service_description and host
to match Nagios config
* Added configurable network types to TcpInput "tcp", "tcp4", "tcp6", "unix" and
"unixpacket" (issue #539)
* Added configurable network types to UdpInput "udp", "udp4", "udp6"
(issue #539)
* Added flush_count config setting to FileOutput to complement existing
flush_interval. Also added flush_operator setting which can be "AND" or "OR"
to specify how flush_count and flush_interval should be combined.