forked from beave/sagan
-
Notifications
You must be signed in to change notification settings - Fork 1
/
ChangeLog
714 lines (503 loc) · 36.9 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
2020/10/?? - Sagan 2.0.0 released.
* New JSON parsing and rule options! For more information on how Sagan
can do JSON processing, see:
https://sagan.readthedocs.io/en/latest/sagan-json.html#sagan-json
New signature options:
"json_content" - https://sagan.readthedocs.io/en/latest/rule-keywords.html#json-content
"json_nocase" - https://sagan.readthedocs.io/en/latest/rule-keywords.html#json-nocase
"json_strstr" - https://sagan.readthedocs.io/en/latest/rule-keywords.html#json-strstr
"json_pcre" - https://sagan.readthedocs.io/en/latest/rule-keywords.html#json-pcre
"json_meta_content" - https://sagan.readthedocs.io/en/latest/rule-keywords.html#json-meta-content
"json_meta_nocase" - https://sagan.readthedocs.io/en/latest/rule-keywords.html#json-meta-nocase
"json_meta_strstr" - https://sagan.readthedocs.io/en/latest/rule-keywords.html#json-meta-strstr
* Removed all libcurl dependencies. It was over kill and cause memory leaks for Bluedot.
* Renamed all "Bro" references to "Zeek".
* Statistics can now be written in a JSON format similar to Suricata's JSON stats.
This will replace the "Perfmon" CSV stats in version 2.0.1.
* New "event_id" rule option. This autmagically gets the event ID (for
example, from Microsoft Windows, etc) from standard syslog or JSON if
present.
* New "chown-fifo" configuration option. If the FIFO (named pipe) is not
owned by Sagan, its ownership is changed if this is set to "true".
* Better validation of signatures upon loading.
* Added new "metadata" option. This works exactly the same as the Suricata
"metadata" option. This rule option is not parsed by the Sagan engine.
It is used to add notes, comments, etc to a signature.
* Added "normalize" data to EVE (liblognorm normalization data).
https://github.com/beave/sagan/commit/ddf5699fe867b8a372c4ada9c70f62f216b87b78
* "candlerb" @ Github submitted some bug fixes & code clean ups. Thanks!
- Logic cleanup around where "routing.c" by
https://github.com/beave/sagan/pull/143
- Fix logic for combined matches of program and syslog_*
https://github.com/beave/sagan/commit/3e177556a114e69161074bfcc05d14b37d42bee3
- Do not forget rule conditions like syslog_facility, syslog_level etc
https://github.com/beave/sagan/commit/c85d9f6c26d4fb85c4f68ab16ed35c462f3c0624
- Update list of debug options in manpage
https://github.com/beave/sagan/commit/c85d9f6c26d4fb85c4f68ab16ed35c462f3c0624
* Fixed "follow flow" option that would only follow up to 512 flows.
* Removed "Snortsam" and Unified2 support.
* New rule option "event_type" add. These is essentially an alias for
"program". This makes it less confusing when working with logs like
Suricata that have an "event_type" field.
* First GeoIP destination bug.
* Added new "client-stats" output option. This take a log line from a
reporting "host" every X seconds and records it. This can be useful for
tracking what is sending data along with a small example type.
* Re-worked how Sagan writes out EVE data. This allows EVE files to be
truncated, renamed, etc and makes it easy for "log rotation" routines.
* Lots of memory & CPU usage improvements.
2019/07/03 - Sagan 1.2.2 released.
* Sagan has new and improved documentation! Go to:
https://sagan.readthedocs.io!
The new documentation system makes it a part of the development cycle. We are
continuing to add and improve our documentation.
* New performance enhancements - This version was "stress tested" in our lab to
identify bottlenecks. The first major issue we noticed was pthread_mutex_lock()/
pthread_mutex_unlock() calls and how expensive they are. For simple lock
scenarios, we now use __atomic_add_fetch() and __atomic_sub_fetch(). For bigger
lock issues, we introduced "batching" in Sagan. This allows Sagan to "lock"
memory for more than one object (log line) at a time. To control how many logs
lines get sent to a thread, adjust the "batch-size" in the sagan.yaml. By
default, the "batch-size" is set to 1. This allows Sagan to perform like you
would normally expect. In situations of 5-10k EPS or higher, consider
adjusting the "batch-size" to a higher value. For more information see:
https://sagan.readthedocs.io/en/latest/high-performance.html
https://github.com/beave/sagan/commit/ece4be4d1b1440bfaa8cedaf56d2477ecc73d07a
https://github.com/beave/sagan/commit/ece4be4d1b1440bfaa8cedaf56d2477ecc73d07a
* Suricata style 'xbits' and Sagan 'flexbits' - Sagan has aligned the
'xbits' keyword usage with Suricata. If you understand how Suricata xbits
function, then you understand how the new Sagan 'xbits' function. What had
previously been known as 'xbits' (prior to this version) are now known as
'flexbits'. In certain situation, 'flexbits' may be more flexible in that
they can be set on TCP/IP port usage and direction.
https://github.com/beave/sagan/commit/eb8cd0e4024a294183c2ec9aeef45920ce900a22
https://github.com/beave/sagan/commit/26e783bbe9b4c0755bb7ccbd3f49db03cecb07d5
https://github.com/beave/sagan/commit/fe6bda08b90cf04505dff086369f6ad60c53480d
https://github.com/beave/sagan/commit/4e60945c0df14eafc9f030ce03fc4d2ff54e841b
i * New 'xbits' in Redis support - Normally 'xbits' are stored to a mmap() file.
This is done to not only store xbit values but allows multiple Sagan processes
to "share" (intra-process communications) xbit data. This also allows Sagan
persistent xbit storage (xbits can survive processes restarts). The 'xbit' in
in Redis concept allows Sagan to share 'xbits' between multiple Sagan systems
over a network. That is, a Sagan system physically in one location can share
xbit data with another system in another physical location via the network.
In the sagan.yaml, you can set your Redis server location (TCP/IP address,
password, etc). You can also set the 'xbit-storage' type to 'redis' (defaults
to mmap). Once done, 'xbits' will be stored to Redis rather than mmap() files.
While this sort of network level communication of threats is great, it comes at
a cost of network speed and latency. Any xbits that fire in rapid succession
might perform poorly. In those cases, you may want to consider using 'xbits'
for high level communications and detection and 'flexbits', which is mmap()
only, for low level communications and detection.
Sagan 'xbits' are stored using the following key format (via SET):
sagan:{cluster name}:{xbit name}:{triggering IP}
For example:
sagan:my_company_name:brute_force:192.168.2.1
Data about the xbit it stored in JSON can can be retrieved using "GET {keyname}".
https://github.com/beave/sagan/commit/1ba4421d66db70a5398f3a71df88bd0671f1dedf
* "cluster-name" is added to Sagan - You can define a 'cluster-name' within
the sagan.yaml. This is currently only used by Redis xbits. This allows
you to control what xbits can be shared in Redis between Sagan instances.
https://github.com/beave/sagan/commit/34a21187d9f6a239453c1dc51f3c6d52938020a1
* The "external" output module has been removed from the sagan.yaml. The "external"
rule keyword is still valid. This calls an external program to processes data
from an event.
https://github.com/beave/sagan/commit/477131b00cc856b88d70e9672e48df0432095d6d
* When using "external" within a rule, the data is now passed in a sane JSON
format via stdin to the program you are calling. This makes processing the data
much easier.
https://github.com/beave/sagan/commit/5186ff1ca6718e3a2944a603f85ed155dc679209
* Fixed various Bluedot JSON parsing issues. Added Bluedot lookup returns to EVE
output.
* Added "program" field to EVE output.
* You can now enabled/disabled base64 payload encoding in the sagan.yaml file by
using the 'alerts-base64' configuration option. By default, to remain compatible
with Suricata style EVE output, it is enabled. If set to 'false', Sagan will
dump the data as plan text.
https://github.com/beave/sagan/commit/1b438950a9b866686a21f316343ff63b752c69ad
* Fixed an issue where using the "ignore" option can consume all worker threads.
https://github.com/beave/sagan/commit/af5991c908b5739587b7e6f95deb8a931484c68a
* Removed support for the "selector" in Sagan. This was remove to keep the Sagan
code clean. It was also our opinion that this would function better at the
syslog daemon level rather than within Sagan.
https://github.com/beave/sagan/commit/e4894e67deeb4d110a432e5672892ec21b3abb03
* Sagan now shuts down more cleanly. Sagan will attempt to "wait" for any worker
threads to complete there tasks. The default "wait" time is 15 seconds. Upon
reaching 15 seconds, Sagan will for worker threads to abort.
https://github.com/beave/sagan/commit/e0253f5de9a052a62d8856c7928be403bba4661a
* In certain situations, the "after" keyword would misfire. This was due to a
timestamp update issue that was out of place.
https://github.com/beave/sagan/commit/e48eef785058619f654aa8826cb161d09d87d3fd
* This version makes the "saganpeek" (src/tools/saganpeek.c) more useful. This
tool allows you to "see" what is currently in Sagan memory. For example, what
has a 'threshold' on it, what 'xbits' are set, what 'after' has tripped. This
utility give you statistics along with the "offending" log line.
* Multiple bug fixes that addressed stability. New JSON bug fixes.
2018/11/07 - Sagan 1.2.1 released.
* Sagan can now read JSON via the FIFO. Traditionally, Sagan has used
a pipe delimited format. This means that you can have your syslog daemon
(rsyslog, syslog-ng, nxlog, etc) send Sagan data in JSON through the FIFO.
There is a new input JSON mapping file (json-input.map) to assist with
JSON mapping of input.
https://github.com/beave/sagan/commit/cfe8d434c326038367f6c99301f0e3f2bd1f934f
https://github.com/beave/sagan/commit/7e65c05ae61355a290c81869e4cccf9e3d911e1b
* New JSON "message" & "program" parsing and auto detection. When enabled,
this allows Sagan to read in JSON data from the syslog "message" field. In some
cases (third party "splunk" forwarding) the JSON will start within the syslog
"program" field. This option allows Sagan to automatically detect the JSON
and find the best mapping for the data. There is a new mapping file
"json-message.map". After JSON is decoded, Sagan "scores" the mappings.
The best score "wins".
https://github.com/beave/sagan/commit/17876f1c2635af18ff2360f2a405d0fe0946783d
https://github.com/beave/sagan/commit/10507916f4721e4efa11916d10ae030b68ddc494
* Fixed flow issue where destination wouldn't be honored in certain situations.
https://github.com/beave/sagan/commit/70117eaacd6ee084a1df81d46c89144f87ac26dd
* Fix issue with "after" that cause false positives.
https://github.com/beave/sagan/commit/f6553fecbb15dc70e4d5181724db42bc7ca530ba
* Due to many changes, "saganpeek.c" had to be altered to support new
"threshold" and "after" options.
https://github.com/beave/sagan/commit/81ecf5e99768d75eb9a798b41d68e41b3dc96a27
https://github.com/beave/sagan/commit/c2b401cb9d15ebb6dcec09d47717335fde9817ca
* New --enable-libfastjson configure option
https://github.com/beave/sagan/commit/a261e84b5a8247069a2acbdf6161263b248a125e
* "rev" and "sid" are now proper uint32_t and uint64_t
https://github.com/beave/sagan/commit/830dd53e05474f0a2c534f1efdb9827154adebe5
https://github.com/beave/sagan/commit/aead360a7a25139a8dd23980a687bc37e345447d
* Complete re-write of "after" and "threshold". The new system is more
flexible and easier to maintain. This allows the rule writer to specify
multiple conditions for a "threshold" or "after".
https://github.com/beave/sagan/commit/58cb296df6b017a0d85c55625da4327891555dc8
https://github.com/beave/sagan/commit/7dc8c2784f8ad915e8d7738661746ba5dbc1950f
https://github.com/beave/sagan/commit/7afd601a8f6f8dfd36dfc4799c09d7db85b4340a
https://github.com/beave/sagan/commit/7afd601a8f6f8dfd36dfc4799c09d7db85b4340a
https://github.com/beave/sagan/commit/8e6ca162b9e77893cbb4e852bdd6dfcbc90cb95f
https://github.com/beave/sagan/commit/c3697e24d344f9ac91b41dda3e8b5f75abe8618b
https://github.com/beave/sagan/commit/cec9e0950ef73273aafbd5e229d56e5e10512845
https://github.com/beave/sagan/commit/ea2dd731331f7dd8b0b1ccc17c0b311241b97b4a
* Added experimental "xbit_upause" rule option. This causes a rule to "pause"
for X microseconds before performing an xbit operation.
https://github.com/beave/sagan/commit/152d688dff0e6772574e3cdd201424b40ec15f9a
* New "rule-tracking" yaml options. This allows tracking of rules that have
never fired verses rules that have fired. This can be useful in rule
tuning.
https://github.com/beave/sagan/commit/1ac1dc091ef55d66ae024c5c047ed264cf25574d
https://github.com/beave/sagan/commit/49d371de716227b22054e2a7014c79b1eb7ae00c
* Added "skip_networks" yaml option to GeoIP and Bluedot. This option tells
Sagan to "skip" lookups for defined network.
https://github.com/beave/sagan/commit/cd38188f4f5537dc57d21ef507eaedd9cad30cc6
https://github.com/beave/sagan/commit/b7522b2de9eb81e62b92c8f63bfba8dc356bfc2e
* Various GeoIP fixes. Change ./configure options from --enable-geoip2
to --enable-geoip
https://github.com/beave/sagan/commit/f08daca514e816e209bca7808431ceabdf58431b
https://github.com/beave/sagan/commit/8f1cf3aafe26802ae42a1bd4ea93b8883949e042
https://github.com/beave/sagan/commit/03d28dd612e45ae236790cf1ec95965285f16c68
https://github.com/beave/sagan/commit/a4d892a1ada24f995caa11ca9660445cd97eba2f
https://github.com/beave/sagan/commit/4d8b918cbec6ec9d603eda62ccd49d6a89966859
* When using NXLog as a syslog receiver, NXLog doesn't handle named pipes/
FIFOs. We created a "help" program so that NXLog can write to FIFOs more
efficiently.
https://github.com/beave/sagan/commit/79703feaf043f2f4f40e179a7850535b52b196be
* Is_IP() and Is_IP6() is now one function.
https://github.com/beave/sagan/commit/2ba2b2749345bebe0dd3d6b6e903ef763e2ac134
* Better thread safety upon exit. On systems with high loads, Sagan would
sometimes segfault upon exit. This corrects that issue.
https://github.com/beave/sagan/commit/854a3296bf46fd189cdc8860f008d3f30130f315
https://github.com/beave/sagan/commit/6575a1e3f9bd16f9dcb43c77dcaab839e1e35b41
* Re-write of how Sagan produces JSON. Sagan can now store _all_ logs in a
JSON output format. This makes it easy to get all logs into back-ends like
Elasticsearch, etc.
https://github.com/beave/sagan/commit/6c8de84c805be280547e433b4c15f2ca6aeba217
https://github.com/beave/sagan/commit/de85f5e30a91237cae681d1811445e50a5a7bfd4
* Removed duplicate rule set load in default sagan.yaml of "windows-security.yaml".
Also correct in the default sagan.yaml that cisco-acs.rules is now
cisco-ise.rules. Thanks msnriggs!
https://github.com/beave/sagan/commit/52386ed383623a16f47ccba6143402c7e8e15b61
* In certain situations, Sagan would segfault when an non-IP address was being
looked up in Bluedot.
https://github.com/beave/sagan/commit/fd172823868700450abe038a7b0e84583e47ee30
* Some minor memory fixes and cleanups identified by Valgrind.
https://github.com/beave/sagan/commit/2ae04fad81bef49640d8aa94854e55a5c9d60871
https://github.com/beave/sagan/commit/79549b69dc5c3589d386cd8e6fe1edb494e6fc46
https://github.com/beave/sagan/commit/5ac62b15fae3dddd5a775674b81da59afce44252
https://github.com/beave/sagan/commit/32241f73c81cab15db92b632aaa7f5c7447fcde1
* Fixes for Bluedot. Added max-ip-cache, max-hash-cache, max-url-cache, and
max-filename-cache to Bluedot processor. Added new DNS "ttl" option to
Bluedot processor to limit the number of times Sagan will lookup the
Bluedot host. Added some new statistics to Bluedot output. Added a new
Bluedot IP queue for dealing with many lookups at a time.
https://github.com/beave/sagan/commit/308236f494c4c8d2e285957f4c8fc7392d58149f
https://github.com/beave/sagan/commit/3a8d6bfb709fd49cf75e28d68bf966ec8d8e1505
https://github.com/beave/sagan/commit/6304d01330848daba8be0f566c2389d84c1a84af
https://github.com/beave/sagan/commit/30fe45ce5e3ee571240a97e30ede636edbb9438b
https://github.com/beave/sagan/commit/a2e5b0a9163fa7dd75e67072153cab1bc99fc09c
https://github.com/beave/sagan/commit/adf9c5f89878ebee7c79fcc4a4f861d80db86769
https://github.com/beave/sagan/commit/1d17d43833b0e49d7f52570d7ece6a4ae3fb6d7d
https://github.com/beave/sagan/commit/32e083cb346abf68f8b840311d9bafd381088230
2018/06/14 - Sagan 1.2.0 released.
* "meta_content" size limitation/truncation issue fixed.
https://github.com/beave/sagan/commit/6640d7384b19712b32603ecde9cc9472e10a5573
* Added new "parse-ip" option to the sagan.yaml. These help
control how "parse_src_ip" and "parse_dst_ip" function. New
options are "ipv6" (enabled by default) and "ipv4-mapped-ipv6"
(disabled by default). This converts IPv4 mapping in IPv6
(for example ffff::192.168.2.1 becomes 192.168.2.1).
* New Bluedot TTL/Host option to prevent overloading DNS servers.
https://github.com/beave/sagan/commit/319f63ef5dab9b9fccab85707cf44701363a78dc
* Better "stripping" of hashes when using the "parse_hash" rule
option.
https://github.com/beave/sagan/commit/d3b0578dbd872544b12a7afca7ef617f9a4b739e
* Fixed issue with "flow.c" when dealing with many flows. Thanks to
Michael Riggs!
https://github.com/beave/sagan/issues/113
https://github.com/beave/sagan/commit/cb5d6e6aaab6a2c2b7dbbc534e6c9fa6cf5197eb
* A lot of processors/bluedot.c fixes that would result in a segfault
under extreme loads.
* Fixed unsigned char[MAXIPBIT] data alignment issues in various places
that might lead to a failure of IP comparisons to fail.
* Removed some left behind "debug" code.
2018/05/29 - Sagan 1.1.9 released.
* Added IPv6 support. Thanks to Kenneth Shelton @ Netwatcher
(https://netwatcher.com).
* Added multi-tenancy through the use of a "selector". Thanks
to Kenneth Shelton @ Netwatcher (https://netwatcher.com).
* "xbits", "after" and "threshold" now store the log
information, signature name and ID used during triggering.
This data can be displayed using the "saganpeek" utility when
this data is using mmap storage.
* Can now use pipes with "meta_content". For example "|3a
3a|"
https://github.com/beave/sagan/commit/50ef014739d0e745e95769b32821d3372b81c430
* Moved liblognorm/normalization structure to thread local.
This should improve normalization preformance.
https://github.com/beave/sagan/commit/2862ecad742f2e32e6fda3af54e03f3ab03bdcf0
* Some getaddrinfo() calls had to be converted back to
inet_pton() due to performance/errors.
https://github.com/beave/sagan/commit/366cc257ab7f03f37b8e5a06520e5b14b5178095
* New "parse_hash" enhancements. These allow extraction of
MD5, SHA1, SHA256 automatically from logs. Useful for Sysmon
support.
https://github.com/beave/sagan/commit/2e6cedafdc9febc1886664734fdff8b33507dce1
* Removed legacy code for isset/isnotset/set for support for &
and |. This added unneeded complexity to the code.
https://github.com/beave/sagan/commit/4e9ddc431b0d2ca733ea75a36a3bf8f336e0df7b
* Fix for issue https://github.com/beave/sagan/issues/107.
Rules should respond to "pri" as well as "priority" rule
options.
https://github.com/beave/sagan/commit/8eb159872c6c0ff99e4435fa2f5f1596dd1dbed7
* Added "force-ipv4" Sagan configuration option. Unified2 can
be somewhat strange with Unified2. This option keep the IP
version type "4" even if an IPv6 address is encountered.
https://github.com/beave/sagan/commit/2dd9b66520c4d8be8c659834669e11e7358d9593
* Sagan will now pass liblognorm JSON output to "external" (rule option)
programs. This allows third party "external" programs to access already
normalized data.
https://github.com/beave/sagan/commit/84970a9bb8f8c3720dbda22914aaa16359687d58
* Fixed Bro-Intel address loading issue. https://github.com/beave/sagan/issues/104
Thanks to Psipher Diaz for reporting this.
https://github.com/beave/sagan/commit/8ee6c6ed1cf5ed27b14fc2ccaccb64674f058886
* Fixed Blacklist segfault:
https://github.com/beave/sagan/issues/111
* Flow checks where broken. Got those fixed.
* Added "xff" EVE/JSON output support.
* Fixed i386 compile time error (Thanks Pierre Chifflier).
* New "xbit: noeve" option to prevent EVE output while xbit is being set.
* A lot of minor bug fixes. For full lists, please see the
Sagan issues page:
https://github.com/beave/sagan/commits/master
* Fix for rules with lots of flows:
https://github.com/beave/sagan/issues/113
* Major re-write of parsers/ip.c. This is used by "parse_src_ip" and
"parse_dst_ip". Now does everything in one pass and "caches" results for
the next rule.
2017/07/25 - Sagan 1.1.8 released.
* Big stability fixes in this release. Mostly involving protecting data with in
global structures (xbit, threshold, after, client tracking, etc). If upgrading,
you'll want to remove all old/legacy IPC data from /var/sagan/ipc or whatever
what you have "ipc-directory" configuration option.
https://github.com/beave/sagan/commit/d10a92c9d0a6ec39f26c149179ce0494a4ceba96
https://github.com/beave/sagan/commit/d10a92c9d0a6ec39f26c149179ce0494a4ceba96
https://github.com/beave/sagan/commit/d2802f9d6cbe26836195e5400e92a707d0cf0062
* Speed up's....
https://github.com/beave/sagan/commit/466a5ea722209ec63cec544c52622f261659dbae
https://github.com/beave/sagan/commit/295221aa6836b017c578663b0185ac63535bfa84
* Sagan threads are now "named". This allows for easier debugging. Thread names
are SaganMain, SaganWorker, SaganKey, SaganPlog, SaganSignal, etc.
https://github.com/beave/sagan/commit/6767ec707044de62166d00fb7f35d7d3050921c5
* Fixes for "parse_hash" rule option. This allows Sagan to automatically extract
MD5, SHA1, SHA256 hashes for lookups.
https://github.com/beave/sagan/commit/58f00606f7912b4c122a4c20eb4321c081842882
* Beta Redis "xbit" sharing code added. This is a work in progress. This allows
Sagan to "share" xbits between devices on the network.
https://github.com/beave/sagan/commit/d25b8e92e00f136623f08f2aee5da442073e5978
https://github.com/beave/sagan/commit/77e33538a054c09ed615a01c6be0e0fc7b373c83
https://github.com/beave/sagan/commit/6a1d6c04c652d4f598e2951949d812578ad53085
https://github.com/beave/sagan/commit/1f288bb6d178ed15859d7fb3cf8ae1ea0ea1e445
https://github.com/beave/sagan/commit/8c2031e21f86a4ec84c62a6d7e0b46d386c3da68
https://github.com/beave/sagan/commit/6828a0ac9cab651046739cd371db8f14cbe0029e
https://github.com/beave/sagan/commit/823163aa8dfe1a205d03d6c975ac0f5a8673c0f8
https://github.com/beave/sagan/commit/d1656a1ba8eb6ffa68fc8224d09d88d97d89e1ec
https://github.com/beave/sagan/commit/c61c0908177129503e839a81007ec2e8139698cb
https://github.com/beave/sagan/commit/8e42d3450afc3c66b5b013b1924df2aed6b13226
* Split after.c and threshold.c from processors/engine.c
https://github.com/beave/sagan/commit/b445cd3aba2c1c6c6128bf819337206f6f253653
https://github.com/beave/sagan/commit/7385473512c6b3aa9ed01a34f9dbc4519a471d3f
* Typo fix issue #95 reported by "CyberTaoFlow".
https://github.com/beave/sagan/commit/b490cf2a9cc3b9d702b1f0c12468a248d2e3e283
* Removed the confusing "email by priority" option. If you want Sagan to e-mail
you event(s), the rule must contain the 'email' rule option.
https://github.com/beave/sagan/commit/8149e923bb37d7a5ff0ec88ff5c5be57c2388ab1
2017/05/31 - Sagan 1.1.7 released.
* The largest and most fundamental change relates to how Sagan rules are written. As of this release, Sagan
rules mimic Snort/Suricata rules. In the past, the first part of a Sagan rule (for example, "alert tcp
$EXTERNAL_NET -> $HOME_NET 22") would be applied to the log message. Sagan would catorgize the log as
being from a "tcp" source and using port "22". This is no longer the case. If Sagan can determine from
the log data the protocol, source port, destination port, source address and destinations address, tradition
"Snort" like logic can then be applied. For example, a Sagan rule can now apply logic to those logs that match
the criteria. The rule writer may only want the rule logic to be applied to logs that are determined to be
from a TCP source and on port 22. The idea is to more closely match Snort and Suricata rule functionality.
In certain situations, you may want to applied a "default" protocol, source or desitination port. For example,
OpenSSH logs do not contain protocol information but we know OpenSSH uses TCP. In those cases, Sagan can apply
defaults by using the new "default_proto" (tcp/udp/icmp), "default_dst_port" and "default_src_port". These new rule
rule options can use hard coded numeric or $VARIABLE options (for example, "default_dst_port: $SSH_PORT");
More information about these changes can be found with the pull request & commits at:
https://github.com/beave/sagan/pull/92
https://github.com/beave/sagan/commit/7eb5caa4ac40f12471f63a3fc9f4759164cd40f9
https://github.com/beave/sagan/commit/709a58f86fd66fb881a4aca2253628d69f4005de
Special thanks a "YoichSec" (Yoichi Sagawa) for the pull/code to make this happen!
* Large cleanup effort involving Snortsam code. This mainly cleaned up many compile time warning messages.
https://github.com/beave/sagan/commit/3a63cb483784ead9a646c66edead9d1610cab98d
* Fixed issue of "external" output not being called correctly.
https://github.com/beave/sagan/issues/93
* Bro-intel support was broken in 1.1.6. This bug has been fixed.
https://github.com/beave/sagan/issues/90
* A large effort and code clean up to standardize the way threads handle data between functions. This greatly increases
the codes "readablity" and brings stability in data handling. More information can be found at:
https://github.com/beave/sagan/issues/89
https://github.com/beave/sagan/commit/9621976e98dec9ffd3ef2d82b844d61ff1efe8ee
https://github.com/beave/sagan/commit/bcb7c0f40d7afc2ee76c93db73d3e2dc7e3d1e2d
https://github.com/beave/sagan/commit/29231c0b7bb094c4c43b4984a966e41835584636
https://github.com/beave/sagan/commit/678a6019443a438ee44917e1d7e2b4b4e45a05c2
https://github.com/beave/sagan/commit/e7cb805e62bb5b41f6f85d1453050e5fc0fe3748
https://github.com/beave/sagan/commit/033fe56b6d67ed1a894dfe3ebd6637d9e16e405b
https://github.com/beave/sagan/commit/033fe56b6d67ed1a894dfe3ebd6637d9e16e405b
https://github.com/beave/sagan/commit/a8ad0bd3f7172750982967b1fac5c12dbb2dedfc
https://github.com/beave/sagan/commit/f797e3620105e825deef744f0a413ec8890a62d0
* Error in SMTP output. Bug caused Sagan to always send e-mails reguardless of priority.
https://github.com/beave/sagan/issues/88
* Many minor bug fixes and cleanups.
2017/03/20 - Sagan 1.1.6-r1 released.
* Resolved minor compile time issues.
2017/03/16 - Sagan 1.1.6 released.
* New Suricata EVE "Alert" compatible JSON output! (still beta!). See the "eve-log" in the sagan.yaml. Test with EveBox!
* Microsecond resolution added in logging for fast, alert.log & unified2.
https://github.com/beave/sagan/commit/5bf1a65270e50a8e1882e4c3a870589dd22b54ad
* Rules will now be distributed with Sagan released (see the "rules" directory).
* Fixed issue #83 - SMTP YAML configation verses rule option bug.
https://github.com/beave/sagan/issues/83
https://github.com/beave/sagan/commit/b2f7e91d6906e87ac436eba91e002bd5729a9940
* Merge YoichSec (https://github.com/YoichSec) new xbit feature to track by source & destination ports.
https://github.com/beave/sagan/pull/85
https://github.com/beave/sagan/commit/546867a4ab7c861168acb861abfc9104c1811da6
* Removed "static __thread" from Sagan_u32_Time_To_Human() function.
https://github.com/beave/sagan/commit/b6a6d01b8119f7576378e4ed1129193043ae9f8e
* Fix for Between_Quotes function related to content:! & meta_content:! rule options.
https://github.com/beave/sagan/commit/2541f2a9e37b56e1ef3bd4a2d5e9c036a9f50e79
* Removed "static __thread" from Between_Quotes() function.
* Removed "static __thread" from DNS_Lookup() function.
* Removed Hex Validate() function (no longer needed)
https://github.com/beave/sagan/commit/188322de08c0ef2ba1b2f11b68e5a5f33bf09c66
* Move strlcpy/strlcat prototypes.
https://github.com/beave/sagan/commit/7e038dc4098c71a4ce7106d91617227166ce21c7
* Complete re-organization of source tree. Dropped all "sagan-?.c" prefixes.
2017/02/15 - Sagan 1.1.5 released.
* Travis-CI (travis.yml) support added to test build Sagan automatically.
* Sagan_Log issue that cause segfault in certain situations with "xbits".
https://github.com/beave/sagan/commit/ce76ebe41f51b21d38a3ece60adf2ad69a5a795d
* Large effort to clean up autoconf/automake fules.
https://github.com/beave/sagan/commit/57a457951670ddb3803fe3bd20a1e4b637d3ddc3
https://github.com/beave/sagan/commit/911e895d2d246c5605bd6fd42da20cb2a3e5f75b
* --daemon bug fix by Ray Ruvinskiy. This fix moves the --daemon logic earlier in the Sagan code (pre-rule loading, etc). This allows for Sagan to load with less memory being used. See commit for more details.
https://github.com/beave/sagan/commit/0a286f06e42a7d509ac662a19bdcc72b777a426f
https://github.com/beave/sagan/commit/f86694adafa76092bdcc1378333227d7945de104
* Remove "force-fifo-ownership" YAML option from configuration/code.
https://github.com/beave/sagan/commit/3d828ca5c91357231e71842c0ac85c2a58cbb9c0
* "include" in YAML configuration would fail to load in certain situations. This has been corrected.
https://github.com/beave/sagan/commit/b27a578082e93eb4f407cdaaa281d57ed4ab88d2
* Bluedot initialization was being called at the wrong time.
https://github.com/beave/sagan/commit/172759a1b6e73eae584b1e6a738a2bf2d3303626
2016/12/30 - Sagan 1.1.4 released.
* Sagan has adopted YAML for future configuration file formats. For more information on YAML see https://en.wikipedia.org/wiki/YAML. The new YAML format allows for a more stream line & easier to understand configuration. The new 'sagan.yaml' is modeled after the Suricata YAML file.
* With the adoption of YAML, Sagan now requires libyaml to be installed.
* configure.in libyaml checks added.
* Sagan "flowbits" are now known as "xbits". "xbits" represents what Sagan is doing with data rather than "flowbits".
* Added new "fast" output format. Works the same as "fast" output format for Suricata/Snort.
* Test for OpenBSD RWX mem segments. If not available, then disable PCRE JIT. Brought up by Victor Julian.
https://github.com/beave/sagan/commit/65aa4dd4d8880d2c54ddbb67a39089d5e6001b69
* Released from Hamburg, Germany after 33c3! :)
2016/11/07 - Sagan 1.1.3 released.
* New "dynamic_load" processor. This allows rules to be created that can load other rules. For example, it is now possible to "detect" new logs coming in and have Sagan automatically load rules for them.
* PCRE JIT support added. This allows faster PCRE processing if your library supports it.
* Modifications to the configure.ac to detect different PCRE library versions and functions. This was taken from the Suricata project.
https://github.com/beave/sagan/commit/04c74d56ac1eff7679636c16fdf853ae4e70c23a
* Fixed bug that caused a segfault on reload (SIGHUP).
https://github.com/beave/sagan/commit/87cd03dc1887e4fa1cef385974180cac77ca1c2d
* Code clean up/redefinition of _Sagan_Event and _Sagan_Proc_Syslog arrays.
https://github.com/beave/sagan/commit/db70eed853719521675e11e95b556edf94565542
2016/09/23 - Sagan 1.1.2 released.
* Added "srcport" for "threshold" and "after".
* Fixes around "dstport" for threshold" and "after"
* Fixed display of source/destination IP addresses in tools/sagan-peek.c
* Change "artistic" style from "GNU" to "Linux". This allows better formatting of source code on smaller screens.
* New pthread mutex to address corruptions of mmap / files.
* tools/sagan-peek.c now has command line option to specify mmap file location. Various other improvements.
* Better error checking in src/processors/sagan-track-clients.c
* New MD5, SHA1 and SHA256 in src/parsers/parse-hash.c
* Minor Bluedot lookup fixes.
* Fixes for "after" and "threshold" when dealing with "usernames". (Username tracking).
* Added "file" option for sagan.conf to reference a file to load. For example, "var MYVAR file:/path/to/my/values.txt".
For more information, see https://github.com/beave/sagan/issues/75
2016/08/17 - Sagan 1.1.1 released.
* Added meta_offset, meta_depth, meta_within and meta_distince Champ Clark ([email protected])
* Major changes/restructuring to sagan-meta-content.c - Adam Hall ([email protected])
* Fix major memory issue when loading large sets of rules. Related to meta_content! (Champ Clark)
* Minor build fix when liblognorm is disabled - Champ Clark
* Minor stdbool.h issue & added PKG_PROG_PKG_CONFIG to configure.ac
* Remove old unneeded liblognorm headers. Champ Clark
* memset replacement of strlcpy, minor clean up, minor bug fixes. (Adam & Champ)
2016/07/06 - Sagan 1.1.0 released.
The Basics:
-----------
* Sagan now "remembers" where it left off between restarts/reboots/etc.
* You can now create rules that focus on certain IP address or IP address ranges (ie - $EXTERNAL_NET/$HOME_NET).
* Sagan can treat "old" Bluedot IP reputation threat Intel differently than "new" threat intel.
* We added "qdee.pl", a SDEE poll routine to the "extra" directory.
* A lot of bugs were fixed
The Details:
------------
* Moved all "threshold", "after", "flowbits", and "client tracking" to mmap files. This means that Sagan "remembers" between restarts where it "left off"!
* Introduced "tools/sagan-peek.c" which allows you to exmaine Sagan mmap files. Useful in debugging or just "seeing" what Sagan is "tracking".
* $EXTERNAL_NET and $HOME_NET now function as expected. Previous versions of Sagan did not have any concept of $EXTERNAL_NET/$HOME_NET and were ignored. Adam Hall @ Quadrant made Sagan "aware" of "traffic flow". Values in a rule for source/destination are tested _after_ normalization.
* Added "mdate" (modification date) and "cdate" (creation data) to Bluedot. This allows Sagan to not trigger "aged" Bluedot Threat Intel. For example, do _not_ alert if an IP address is seen and the Intel is over X hours/days/months/years old.
* Threholding based on 'dstport' merged, thanks to Bruno Coudoin. See: https://github.com/beave/sagan/commit/44d6752acf27d61bcd57e35f930b0f6e11dadbc7
* Added parsing for IPTables "SPT" and "DPT"t port for iptables, thanks to Bruno Coudoin. https://github.com/beave/sagan/commit/9de9cffd224a44f93c80eca62e6ead617a4b97a6
* Added "qdee" to the "extra" directory. This allows Sagan to parse older style Cisco IDS output. This polls using the SDEE protocol. See https://github.com/beave/sagan/commit/61c4a7dd611161697785c889630dd3c8333ec8b5
* Removed support for libjsonc (json-c) and moved to libfastjson.
The Bugs Fixed:
---------------
* Correct issue for when Sagan cannot open a file (-F/--file) due to permissions.
* Removed unused "SigArgs" array.
* Clean exit when Sagan cannot load Maxmind GeoIP2 data file.
* Change "normalize: {type}" to "normalize;". All normalization rules now come from one file. This keeps Sagan in line with liblognorm development.
* Sagan now "warns" the user if old style "normalize" is encountered. See: https://github.com/beave/sagan/commit/ba3de9e43bc8623b361e34ce06a2e7808e045f88 and https://github.com/rsyslog/liblognorm/issues/206
* Fix json_object_object_get_e) compile time warnings. See: https://github.com/beave/sagan/commit/e9bdea5b7fa5b25c1d7e740a4c856c70a1046d1d
* Minor ARM CPU fixes.
* Various "meta_content" fixes. When using "meta_content" with large amounts of search data would sometimes cause failures.
* Major bug fixes involving "client tracking". Thanks to Adam Hall @ Quadrant Information Security!
* Sagan now attempts to create the FIFO if it is not detected. Thanks to Cabrol Perales.
* A lot of smaller bug fixes. See: https://github.com/beave/sagan/commits/master
2015/11/19 - Sagan 1.0.1 released.
* Add unified2 "extra data" field to store the orignal syslog
source.
* Added threshold by "username".
* Minor code clean up.
2015/10/23 - Sagan 1.0.0 released.