Re-enabling Push (after a long period of it being disabled) leads to the entire backlog being sent out

[nmagedman]

Description:

We had disabled Push Notifications for a long time while trying to manage an ongoing problem with high server CPU. When we eventually re-enabled it, RC pushed out a long backlog of notifications.

Steps to reproduce:

1. Go to Administration > Push
2. Change the "Enable" toggle-switch to OFF.
3. Restart RC
4. Let the system run without push notifications for several months.
5. Upgrade RC from 2.x to 3.x
6. Go to Administration > Push
7. Change the "Enable" toggle-switch to ON.
8. Restart RC

Expected behavior:

Comments made before the time Push was re-enabled would not be pushed out the mobile devices.
Only new comments should be pushed out.

Actual behavior:

I'm receiving a backlog of notifications going back several months, from before I disabled Push.
I'm not certain whether the notifications also include those from the several-month window during which Push was disabled.

Server Setup Information:

• Version of Rocket.Chat Server: 3.0.3
• Operating System: Ubuntu Linux
• Deployment Method: tarball
• Number of Running Instances: 15 (5 servers, 3 processes per server)
• DB Replicaset Oplog: Enabled
• NodeJS Version: 12.14.0 - x64
• MongoDB Version: 3.4.20

Additional context

Similar problem mentioned in an old, unanswered, Community Support post from a year ago.
https://forums.rocket.chat/t/problem-with-old-notifications-being-sent-after-enabling-mobile-push-notifications/3932

Relevant logs:

Perhaps incidental, but Push-related:

Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]: server.js:204 System ➔ error Error sending push to gateway (3 try) -> Error: failed [400] {"cod
e":108,"error":"bad request sent to apn","requestId":"2a7fd82a-1dbd-487d-980c-acedb68c5ea2","status":400}
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Object.exports.makeErrorByStatus (packages/http.js:176:10)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Request._callback (packages/http.js:140:24)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Request.self.callback (/opt/rocket-chat/bundle/programs/server/npm/node_modules/meteor/h
ttp/node_modules/request/request.js:185:22)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Request.emit (events.js:210:5)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Request.EventEmitter.emit (domain.js:475:20)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Request.<anonymous> (/opt/rocket-chat/bundle/programs/server/npm/node_modules/meteor/htt
p/node_modules/request/request.js:1161:10)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Request.emit (events.js:210:5)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Request.EventEmitter.emit (domain.js:475:20)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at IncomingMessage.<anonymous> (/opt/rocket-chat/bundle/programs/server/npm/node_modules/me
teor/http/node_modules/request/request.js:1083:12)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at Object.onceWrapper (events.js:299:28)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at IncomingMessage.emit (events.js:215:7)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at IncomingMessage.EventEmitter.emit (domain.js:475:20)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at endReadableNT (_stream_readable.js:1184:12)
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     at processTicksAndRejections (internal/process/task_queues.js:80:21) {
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:   response: {
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     statusCode: 400,
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     content: '{"code":108,"error":"bad request sent to apn","requestId":"2a7fd82a-1dbd-487d-980c-acedb68c5ea2","status":400}\n',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     headers: {
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       'access-control-allow-headers': 'Content-Type, Authorization, Content-Length, X-Requested-With',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       'access-control-allow-methods': 'GET, PUT, POST, DELETE, OPTIONS',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       'access-control-allow-origin': '*',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       'access-control-expose-headers': 'Content-Type, Authorization, Cache-Control, Expires, Pragma, X-powered-by',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       'cache-control': 'private, no-cache, no-store, must-revalidate',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       'content-length': '111',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       'content-type': 'application/json; charset=utf-8',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       date: 'Sun, 05 Apr 2020 11:00:29 GMT',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       expires: '-1',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       pragma: 'no-cache',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       'x-powered-by': 'Rocket Fuel and Rocketeers',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       connection: 'close'
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     },
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     data: {
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       code: 108,
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       error: 'bad request sent to apn',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       requestId: '2a7fd82a-1dbd-487d-980c-acedb68c5ea2',
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:       status: 400
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:     }
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]:   }
Apr  5 11:00:29 i-009c3440d1c696c8d rocket.chat[5571]: }

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sampaiodiego]

this should have been fixed by #17357