Log rotation via Logs.MaxBackups broken

[ghost]

Old logs are not being removed, e.g. on one of the gateway servers (saturn).

$ ls -la /ipfs/ipfs_master/repo/logs
total 1527944
drwxr-xr-x 2 root root     36864 Jun 15 08:02 .
drwxr-xr-x 5 root root      4096 Jun 15 08:47 ..
-rw-r--r-- 1 root root 104857600 Jun 14 21:57 events-2015-06-15T01-57-53.411.log
-rw-r--r-- 1 root root 104857600 Jun 14 22:45 events-2015-06-15T02-45-07.608.log
-rw-r--r-- 1 root root 104857600 Jun 14 23:57 events-2015-06-15T03-57-21.684.log
-rw-r--r-- 1 root root 104857600 Jun 15 00:59 events-2015-06-15T04-59-41.068.log
-rw-r--r-- 1 root root 104857600 Jun 15 01:48 events-2015-06-15T05-48-37.328.log
-rw-r--r-- 1 root root 104857600 Jun 15 02:45 events-2015-06-15T06-45-13.662.log
-rw-r--r-- 1 root root 104857600 Jun 15 03:37 events-2015-06-15T07-37-06.694.log
-rw-r--r-- 1 root root 104857600 Jun 15 04:06 events-2015-06-15T08-06-41.607.log
-rw-r--r-- 1 root root 104857600 Jun 15 04:46 events-2015-06-15T08-46-56.850.log
-rw-r--r-- 1 root root 104857600 Jun 15 05:30 events-2015-06-15T09-30-31.561.log
-rw-r--r-- 1 root root 104857600 Jun 15 05:59 events-2015-06-15T09-59-14.303.log
-rw-r--r-- 1 root root 104857600 Jun 15 06:27 events-2015-06-15T10-27-13.424.log
-rw-r--r-- 1 root root 104857600 Jun 15 07:15 events-2015-06-15T11-15-25.221.log
-rw-r--r-- 1 root root 104857600 Jun 15 08:02 events-2015-06-15T12-02-49.921.log
-rw-r--r-- 1 root root  96501760 Jun 15 09:05 events.log

[rht]

What is the server version and ipfs config Log (to help with bisect)?

[whyrusleeping]

@rht its been this way pretty much since we set it up. I don't think it ever worked.

[ghost]

The commit is 988b158 on all of them.
MaxSizeMB seems to works. earth's config lacks MaxAgeDays, but shows the same problem.

$ ansible solarnet -a 'docker exec ipfs bash -c "ipfs version ; ipfs config Logs"'
uranus | success | rc=0 >>
ipfs version 0.3.4
{
  "Filename": "/ipfs/repo/logs/events.log",
  "MaxAgeDays": 0,
  "MaxBackups": 1,
  "MaxSizeMB": 100
}

jupiter | success | rc=0 >>
ipfs version 0.3.4
{
  "Filename": "/ipfs/repo/logs/events.log",
  "MaxAgeDays": 0,
  "MaxBackups": 1,
  "MaxSizeMB": 100
}

neptune | success | rc=0 >>
ipfs version 0.3.4
{
  "Filename": "/ipfs/repo/logs/events.log",
  "MaxAgeDays": 0,
  "MaxBackups": 1,
  "MaxSizeMB": 100
}

pluto | success | rc=0 >>
ipfs version 0.3.4
{
  "Filename": "/ipfs/repo/logs/events.log",
  "MaxAgeDays": 0,
  "MaxBackups": 1,
  "MaxSizeMB": 100
}

venus | success | rc=0 >>
ipfs version 0.3.4
{
  "MaxAgeDays": 0,
  "MaxBackups": 1,
  "MaxSizeMB": 100
}

earth | success | rc=0 >>
ipfs version 0.3.4
{
  "MaxBackups": 1,
  "MaxSizeMB": 100
}

mercury | success | rc=0 >>
ipfs version 0.3.4
{
  "MaxAgeDays": 0,
  "MaxBackups": 1,
  "MaxSizeMB": 100
}

saturn | success | rc=0 >>
ipfs version 0.3.4
{
  "Filename": "/ipfs/repo/logs/events.log",
  "MaxAgeDays": 0,
  "MaxBackups": 1,
  "MaxSizeMB": 100
}

[rht]

I c. At least 'works' here in my node (ipfs 0.3.4, latest master as of 7d434da)

$ ls -lha ~/.ipfs/logs
total 287M
drwxr-xr-x 2 rht users 4.0K Jun  9 14:16 .
drwxr-xr-x 5 rht users 4.0K Jun 15 22:24 ..
-rw-r--r-- 1 rht users 250M Jun  9 14:16 events-2015-06-09T07-16-04.044.log
-rw-r--r-- 1 rht users  37M Jun 15 22:24 events.log

$ ipfs config Log
{
  "MaxAgeDays": 0,
  "MaxBackups": 1,
  "MaxSizeMB": 250
}

[rht]

I did remove the logs on Jun 9 (I forgot the reason), so this is likely due to my node doesn't have enough traffic to fill the logs. Will test with smaller MaxSizeMB then.

[rht]

I switched MaxSizeMB to 1. It does work on my node (MaxAgeDays / MaxBackups). Here are 3 different snapshots of ~/.ipfs/logs

total 1.2M
drwxr-xr-x 2 rht users 4.0K Jun 15 22:52 .
drwxr-xr-x 5 rht users 4.0K Jun 15 22:47 ..
-rw-r--r-- 1 rht users 1.0M Jun 15 22:52 events-2015-06-15T15-52-01.360.log
-rw-r--r-- 1 rht users 160K Jun 15 22:52 events.log

total 1.8M
drwxr-xr-x 2 rht users 4.0K Jun 15 22:52 .
drwxr-xr-x 5 rht users 4.0K Jun 15 22:53 ..
-rw-r--r-- 1 rht users 1.0M Jun 15 22:52 events-2015-06-15T15-52-01.360.log
-rw-r--r-- 1 rht users 756K Jun 15 22:54 events.log

total 1.3M
drwxr-xr-x 2 rht users 4.0K Jun 15 22:54 .
drwxr-xr-x 5 rht users 4.0K Jun 15 22:53 ..
-rw-r--r-- 1 rht users 1.0M Jun 15 22:54 events-2015-06-15T15-54-56.267.log
-rw-r--r-- 1 rht users 292K Jun 15 22:56 events.log

[ghost]

So is this with MaxAgeDays=0 and MaxBackups=1?

[rht]

Right, unchanged config, or rather, the default except for MaxSizeMB.

[rht]

Linking the issue nodes #1018

[ghost]

rht__: i was updating those Log configs on the gateways right now
noticed it says Logs there, which seems to be old
it's just Log nowadays
the repo.config.Logs has been kept in place, but isn't anymore
so it's just silently discarded without a migration path
:X
i guess that means it assumes MaxBackups = 0

[ghost]

Yeah here's the PR that removes the Logs config section: #642

[jbenet]

noticed it says Logs there, which seems to be old
it's just Log nowadays
the repo.config.Logs has been kept in place, but isn't anymore
so it's just silently discarded without a migration path

oh no! that was silly. :(

[ghost]

And after the next rotation the old logs got removes so that now there's only one old backup being kept.

[rht]

I guess #1018 will have to wait.