Skip to content

Commit

Permalink
Syslog rate limit design (#1049)
Browse files Browse the repository at this point in the history
Logging in SONiC is organized with rsyslogd. Each container has its own rsyslogd instance plus a daemon running on a host. The rsyslogd instance which is running on the host is used to collect the messages from within containers and store them at /var/log/syslog path. rsyslog config file are generated from templates:

Container scope:
Multi ASIC: https://github.com/Azure/sonic-buildimage/blob/master/files/image_config/rsyslog/rsyslog-container.conf.j2
Single ASIC: https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-base/etc/rsyslog.conf

Host scope: 
https://github.com/Azure/sonic-buildimage/blob/master/files/image_config/rsyslog/rsyslog.conf.j2
Currently, each container has hardcoded message rate limiting to avoid receiving flooded log messages:

$SystemLogRateLimitInterval 300
$SystemLogRateLimitBurst 20000
There is no rate limiting configured on host side for now.

The SystemLogRateLimitInterval determines the amount of time that is being measured for rate limiting. The SystemLogRateLimitBurst defines the amount of messages, that have to occur in the time limit of SystemLogRateLimitInterval, to trigger rate limiting. For example, SystemLogRateLimitInterval=300, SystemLogRateLimitBurst=20000, it means that if one daemon generate more than 20000 messages in 300 seconds, rsyslogd will start to drop messages after that(FIFO).

This feature allows user to configure SystemLogRateLimitInterval and SystemLogRateLimitBurst for host, containers.
  • Loading branch information
Junchao-Mellanox authored Nov 8, 2022
1 parent b9841f8 commit bd2cfde
Show file tree
Hide file tree
Showing 6 changed files with 388 additions and 0 deletions.
1 change: 1 addition & 0 deletions doc/syslog/images/cli_rate_limit_config_flow.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit bd2cfde

Please sign in to comment.