Kafka feature 3105 v5 #4096

CosmosSun · 2019-08-08T03:34:16Z

This changeset ensures the kafka output ability for eve.

A new output method convenient for data analysis.
Configure with --enable-rdkafka option when you need outpout eve with kafka.
Update eve kafka output conf doc

Continuation of #4093.

Make sure these boxes are signed before submitting your Pull Request -- thank you.

I have read the contributing guide lines at https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Contributing
I have signed the Open Information Security Foundation contribution agreement at https://suricata-ids.org/about/contribution-agreement/
I have updated the user guide (in doc/userguide/) to reflect the changes made (if applicable)

Link to redmine ticket:
https://redmine.openinfosecfoundation.org/issues/3105
Describe changes:

Kafka output method convenient for data analysis
Update eve kafka output conf doc

PRScript output (if applicable):

This changeset ensures the kafka output ability for eve. - A new output method convenient for data analysis. - Configure with --enable-rdkafka option when you need outpout eve with kafka. - Update eve kafka output conf doc

norg · 2019-08-08T22:08:27Z

Doc looks fine to me, a bit more explanation for new users might be helpful

jryberg · 2019-08-17T16:30:50Z

This is really awesome!! Is there anything blocking this merge?

CosmosSun · 2019-08-18T03:55:52Z

This is really awesome!! Is there anything blocking this merge?
https://redmine.openinfosecfoundation.org/issues/3105#note-3
maybe i should solved all these possible cases

jlucovsky · 2019-08-19T22:13:27Z

src/output-json.c

+                gethostname(hostname, 1023);
+                json_ctx->file_ctx->sensor_name = SCStrdup(hostname);
+            }
+            if (json_ctx->file_ctx->sensor_name  == NULL) {


nit: 2 blanks between sensor_name and ==

jlucovsky · 2019-08-19T22:22:27Z

src/util-log-kafka.c

+			   int error_code,
+			   void *opaque, void *msg_opaque) 
+{
+    rk = rk;


lhs = rhs ... what's this for? L108-L112

jlucovsky · 2019-08-19T22:23:48Z

src/util-log-kafka.c

+    rd_kafka_topic_conf_t *topic_conf;        
+    char tmp[16];        
+    char errstr[512];
+    kafka_ctx = (SCLogKafkaContext*) SCCalloc(1, sizeof(SCLogKafkaContext));


Make sure kafka_ctx is freed on error paths/early exits.

jlucovsky · 2019-08-19T22:25:10Z

src/util-log-kafka.c

+        exit(EXIT_FAILURE);
+    }
+
+    conf = rd_kafka_conf_new();


Make sure you free conf with rd_kafka_topic_conf_destroy on error paths/early exits.

jlucovsky · 2019-08-19T22:31:03Z

src/util-log-kafka.c

@@ -0,0 +1,203 @@
+/* Copyright (C) 2007-2019 Open Information Security Foundation


Nit: change date to just 2019

jlucovsky · 2019-08-19T22:31:48Z

src/util-log-kafka.c

+#include "util-log-kafka.h"
+#include "util-logopenfile.h"
+
+#ifdef HAVE_LIBRDKAFKA


IMO, this guard should be moved prior to the first #include statement.

jlucovsky · 2019-08-19T22:35:26Z

src/util-logopenfile.c

+        if (!log_ctx->sensor_name) {
+            char hostname[1024];
+            gethostname(hostname, 1023);
+            log_ctx->sensor_name = SCStrdup(hostname);


Where will sensor_name be freed if SCConfLogOpenKafka fails?

victorjulien · 2019-08-26T08:12:00Z

Please submit a new PR with the requested updated, thanks!

victorjulien · 2019-08-27T08:00:41Z

Btw @dbcfd how do you feel about this? Asking because of your https://github.com/dbcfd/surikafka project.

@CosmosSun why do you think this should be a part of Suricata itself instead of using something like https://github.com/dbcfd/surikafka ?

CosmosSun · 2019-08-27T10:34:48Z

Btw @dbcfd how do you feel about this? Asking because of your https://github.com/dbcfd/surikafka project.

@CosmosSun why do you think this should be a part of Suricata itself instead of using something like https://github.com/dbcfd/surikafka ?
I have not seen surikafka before, so i add a similar feature.
It may provide a new output type directly.

dbcfd · 2019-08-27T15:44:34Z

My general feeling is that suricata should focus on using file and UDS for output, to reduce maintenance cost associated with other output types (e.g. kafka and redis). Continuing to add additional output types increases maintenance cost for features that may see limited use, and can be achieved by using a unix pipe like approach, where the output handler is external to suricata. This also allows the output handler to have a different release cycle, to address possible bugs, library upgrades, etc.

I feel even stronger about this with kafka, due to its significant configuration and failure modes. Kafka is more likely to introduce performance impacts than redis, especially if it is configured as a blocking, don't drop events mode with an ack level for broker and replicates.

Sleep 250 microseconds instead of 100 as running in KVM cause the old value to use 100% CPU for these threads. Perf testing suggests no measurable impact for the non-KVM case. Ticket: OISF#4096

Sleep 250 microseconds instead of 100 as running in KVM cause the old value to use 100% CPU for these threads. Perf testing suggests no measurable impact for the non-KVM case. Ticket: OISF#4096 (cherry picked from commit 17a38f1)

zhouhanAC · 2023-08-16T02:10:35Z

You should use rd_kafka_poll after each call to rd_kafka_produce, or call rd_kafka_poll regularly, otherwise the VmRSS of the process will continue to grow, causing memory leaks:

`int LogFileWriteKafka(void *lf_ctx, const char *string, size_t string_len)
{
LogFileCtx *log_ctx = lf_ctx;
SCLogKafkaContext *kafka_ctx = log_ctx->kafka;
int partition = kafka_ctx->partition % (log_ctx->kafka_setup.partitions);

if (rd_kafka_produce(kafka_ctx->rkt, partition,
		RD_KAFKA_MSG_F_COPY,
		/* Payload and length */
		(void *)string, string_len,
		/* Optional key and its length */
		NULL, 0,
		/* Message opaque, provided in
		 * delivery report callback as
		 * msg_opaque. */
		NULL) == -1) 
{
	SCLogError(SC_ERR_KAFKA,
			"%% Failed to produce to topic %s "
			"partition %i: %s\n",
			log_ctx->kafka_setup.topic_name, partition,
			rd_kafka_err2str(
				rd_kafka_errno2err(errno)));
	/* Poll to handle delivery reports */
	rd_kafka_poll(kafka_ctx->rk, 0);
    return -1;
}
kafka_ctx->partition++;

rd_kafka_poll(kafka_ctx->rk, 0 /*non-blocking*/);
return 0;

}`

victorjulien · 2023-09-05T06:50:52Z

@zhouhanAC please report bugs in our issue tracker instead of very old and closed PRs, thanks!

output: Add kafka output ability for eve

d9f400a

This changeset ensures the kafka output ability for eve. - A new output method convenient for data analysis. - Configure with --enable-rdkafka option when you need outpout eve with kafka. - Update eve kafka output conf doc

CosmosSun requested review from norg and a team as code owners August 8, 2019 03:34

CosmosSun closed this Aug 8, 2019

CosmosSun reopened this Aug 8, 2019

CosmosSun closed this Aug 8, 2019

CosmosSun reopened this Aug 8, 2019

jlucovsky reviewed Aug 19, 2019

View reviewed changes

victorjulien closed this Aug 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka feature 3105 v5 #4096

Kafka feature 3105 v5 #4096

CosmosSun commented Aug 8, 2019 •

edited

Loading

norg commented Aug 8, 2019

jryberg commented Aug 17, 2019

CosmosSun commented Aug 18, 2019

jlucovsky Aug 19, 2019

jlucovsky Aug 19, 2019

jlucovsky Aug 19, 2019

jlucovsky Aug 19, 2019

jlucovsky Aug 19, 2019

jlucovsky Aug 19, 2019

jlucovsky Aug 19, 2019

victorjulien commented Aug 26, 2019

victorjulien commented Aug 27, 2019

CosmosSun commented Aug 27, 2019

dbcfd commented Aug 27, 2019

zhouhanAC commented Aug 16, 2023

victorjulien commented Sep 5, 2023

		@@ -0,0 +1,203 @@
		/* Copyright (C) 2007-2019 Open Information Security Foundation

Kafka feature 3105 v5 #4096

Kafka feature 3105 v5 #4096

Conversation

CosmosSun commented Aug 8, 2019 • edited Loading

norg commented Aug 8, 2019

jryberg commented Aug 17, 2019

CosmosSun commented Aug 18, 2019

jlucovsky Aug 19, 2019

Choose a reason for hiding this comment

jlucovsky Aug 19, 2019

Choose a reason for hiding this comment

jlucovsky Aug 19, 2019

Choose a reason for hiding this comment

jlucovsky Aug 19, 2019

Choose a reason for hiding this comment

jlucovsky Aug 19, 2019

Choose a reason for hiding this comment

jlucovsky Aug 19, 2019

Choose a reason for hiding this comment

jlucovsky Aug 19, 2019

Choose a reason for hiding this comment

victorjulien commented Aug 26, 2019

victorjulien commented Aug 27, 2019

CosmosSun commented Aug 27, 2019

dbcfd commented Aug 27, 2019

zhouhanAC commented Aug 16, 2023

victorjulien commented Sep 5, 2023

CosmosSun commented Aug 8, 2019 •

edited

Loading