Push notifications stop after ~24 hours #1218
Comments
|
Correct, notifications worked for 1 or 2 days max then stopped also for my Ring Intercom |
If someone was interested in conducting this investigation, what steps would they need to take? Setting up |
|
Great question @smockle! The hard part with mitmproxy is that the Ring App uses certificate pinning, which makes any MITM setup not work by default. There are ways around this, but they require a rooted device and a few pieces of software to get everything working. It's not a route I recommend others try to go down 😅. Honestly I think this issue might be one that we can debug without necessarily seeing the traffic from the App. Based on the assessment from @tsightler in the other issue, I have a suspicion that the Another suspicion is that users experiencing this issue might be on a setup where they are not getting a consistent hardware id. This would present in the form of many "devices" showing up in the Control Center during that first 24 hour period. @salemsayed I'm curious, after you cleared your devices and waited a day, did you have just a few devices in Control Center, or was the list very long again? |
Unfortunately, the code in ring-mqtt already does this, at least it attempts to by calling subscribeToDingEvents()/subscribeToMotionEvents() when it detects this by checking the value on every poll event (20 seconds). The thing is, subscribing to the events doesn't appear to fail, but the observed behavior in the logs is that, once these values start showing up as false, they will revert back to unsubscribed every few minutes even after calling the subscribe function. @smockle If you do decide to go down the path of mitmproxy, below is a post from a ring-mqtt user that did just that when they wanted me to implement a function that we didn't know how the API worked. It might help if you decide to go down that route, it didn't seem too hard. tsightler/ring-mqtt#528 (reply in thread) I actually want to set this up myself, but time just hasn't been available for me lately. |
Just two devices right now, an unnamed device and homebridge-ring device. |
Same here. Have been deleting and re-authenticating the devices for days now. Each time it's just the two: unnamed and Homebridge ring. |
|
Hi, I use the ring-client-api in an app for the Homey Smart Home Platform (https://homey.app/) and there are quite some users of the app that are reporting the same exact problems with the notifications. The strange thing is that not all users seem to be affected by the bug, the app has ~5000 users but only a few compared to that amount have reported it. That’s all information I can add for now, I’m currently on vacation and have currently no way to do any troubleshooting myself. When I get more relevant information from my users I will post it here. Edit: For anyone interrested, this is the code that sets up the connection with the Ring api: |
|
Thanks for the info @daneedk! That sounds in-line with the other experiences described so far. I personally use the standard Ring App notifications so I wasn't sure if this was affecting me. Enabled notifications this morning and mine are working just fine, so I'm not able to repro/debug 😞. It sounds like there is a lot of variance in who is experiencing this. For those having issues, what version of Node.js are you on? I'm on 18.16.0 without issues. |
|
@daneedk That sounds exactly the same as what I'm seeing from ring-mqtt users as well. I estimate at least 6000 users of ring-mqtt based on various analytics data (it's probably higher, but so far only a few sporadic reports. Almost all started around May 4th and it's the first time it ever impacted my setup. However, since resetting all authorized users, my setup works perfectly for two weeks now. However, its clear that other users see this work only for about 24 hours. For whatever reason, I've noticed that a lot of people with the issue do not have a Ring subscription, but not everyone. Perhaps there's just a higher percentage of Ring users that don't have a subscription than I realized. I'm also not sure that anyone that has reported the issue to this point is a US resident, so I wonder if the problem is happening in other regions vs the US. However, I think it would be really good if we could start collecting this basic information from users experiencing the 24 hour issue to see if there's anything common. My suggestions for questions:
However, I'm also open to other suggestions. Not being able to reproduce the issue makes it particularly difficult to find any solution. One thing, at least one user has reported that the native Home Assistant Ring integration has the same problem, but it still uses the old polling method vs push notifications (as far as I know anyway). |
|
This problem occurs on both the old and new Homey. The old Homey uses nodejs v12.22.11 and the new Homey, which is really new and it’s firmware is still in Release Candidate phase, uses v18.16.0. The old Homey has not been available in the US so no US based users for that one, the new Homey will be available in the US, but I’m not sure if any are shipped yet (I’ll try to get that confirmed) The first report about this particular problem was posted on the Homey Community forum on the 10th of may. My app has a dedicated support topic on that forum, I will ask users that are experiencing the problem to answer the questions above. |
|
|
|
|
Thanks for the responses everyone!
@tsightler you may have nailed the problem in this paragraph. I think the symptoms we are seeing likely stem from new Data Residency laws, which require data to only be stored within the region of the user. In some of the new laws, there is an acceptable period for data to be stored in another region, but only temporarily (something on the scale of 24-48 hours). If Ring is storing the session and push notification data in the US, it would make sense that they would only be able to store that same info for 24-48 hours for users in Spain/Germany. Assuming this theory is correct, I'm guessing the apps are now configured to recreate the user session and push notification registration at least once a day. We should be able to do something similar from |
|
Hi, I’m having the same problem here in Canada. When did you notice the problem? Did you remove all device and re-authenticate, if so, did this work and for how long? What devices do you have? What region/country are you located in? Do you have a ring subscription? |
|
@dgreif I think the theory of changes regarding data residency may make sense. Around the same time reports started of notification issues there was also another issue reported with ring-mqtt that I was pretty sure was indirectly related to the infrastructure changes. The ring-mqtt code polls the device health API every 5 minutes to update various attribute data. Around the same time notifications started having issues some users also reported an issue where this data was generating an error in ring-mqtt due to null data. Investigating this issue, it appeared to only impact EU users. Querying the API still returned a result, but the JSON response mostly contained null values, especially for personally identifiable data like wifi network name. After about 24-48 hours everyone that reported the problem indicated that the behavior returned to normal. I had thought that maybe Ring was making changes to keep personally identifiable data, like wifi names, in-region.
One thing though, does this not happen on restart? So far users have reported restarting doesn't help. |
|
@dgreif @tsightler the theory about it impacting mostly EU users sounds plausible, most users of my app are in the European region. There are also some in Australia and New Zealand, I can contact a user from Australia and check with him.
|
|
Having the issue here as well. No dings and no motion events coming through to Homebridge. Ring App works. Node.js version is 18.16.0
|
|
I've released v
Great question! The original setup actually didn't create a session (even on restart) unless we got a 404 back on a standard data request. Now I'm creating a session before any data requests go out, as well as on a 12 hour interval. |
|
For users of ring-mqtt that may be monitoring this issue, I've updated the dev branch to include 11.7.6 of ring-client-api to make this easier to test. Simply change the "branch" config option from "addon" to "dev" and restart the addon/docker container and it will pull down the latest dev branch for testing purposes. If this works I'll push out a full update later this week. |
|
Well, I'm in the U.S., Michigan, and I have a Ring subscription. I've changed nothing since the Motion Detection on the cameras stopped getting to Homebridge. Except for now I've updated the plugin. None of these cameras are passing motion: |
|
@MoTechnicalities Did you follow the instructions of deleting all authorized clients and re-authenticating. |
|
The issue I was bringing attention to here: #1219 got closed as a duplicate of this post. Just letting you know that the issue 1219 is not resolved after upgrading to the latest version 11.7.6, the button on the ringdoorbell is still not pushing any notifications to homekit Thought you might like to know |
|
@benjackson1111 Same question, have you followed the steps to remove all devices and re-authenticate? Does this fix it for some period of time? |
|
Thanks for reporting in! @jchristianj thanks for pointing it out, those 404 logs are normal after you create a new refresh token. Nothing to worry about there. @daneek that error definitely seems related to the refresh token as that's what you would get if you had an invalid refresh token. I'm not able to reproduce any issues with it, so hopefully you can find something while debugging on your end 🤞. @tammotheus that's an odd one, but likely some issue with communication between homebridge and homekit. I'd try rebooting all the things 😄 Side note, I'll be AFK much of the weekend so unfortunately I can't promise a quick turnaround on fixing anything with this beta. A few negative reports so far, but hopefully others are able to upgrade with less issues and try out the changes in regard to double devices and notifications after 24 hours/restarts. |
|
@dgreif many thanks for this fast solution. :-) regards and have a good weekend |
|
The beta is rock solid so far. |
|
It appears to be working for me as well on initial testing. Thanks so much. Great job! |
|
Is there anything else I can try? I have updated to the latest beta, removed the camera in the home app, re-added it, refreshed the token in the homebridge config....and its worse than before. At least I'd get it working for 24 hours, now the button on the doorbell doesn't trigger anything, even after following the instructions as above. What could I be missing when everyone else is having success?! Sorry to the bearer of bad news! |
|
@benjackson1111 I would suggest simply following the staps again very carefully. As you state, others have shown that this method works, so my guess is that you likely did something out-of-order without realizing it. I'd suggest making sure that no ring-homebridge connection is made and don't try to re-authenticate until you are sure there are no devices listed in the Ring Control Center Authorized devices list. Some users have had to remove all devices to get it working, not just the ones associated with ring-homebridge/ring-client-api/ring-mqtt/etc. It's very critical that the authorized device created is 100% a new device or the push notifications will not be enabled. |
|
ok thanks I’ll try again I appreciate the confidence kick!! On Jun 3, 2023, at 9:37 PM, tsightler ***@***.***> wrote:
@benjackson1111 I would suggest simply following the staps again very carefully. As you state, others have shown that this method works, so my guess is that you likely didn't something out-of-order without realizing it. I'd suggest making sure that no ring-homebridge connection is made and don't try to re-authenticate until you are sure there are no devices listed in the Ring Control Center Authorized devices list. Some users have had to remove all devices to get it working, not just the ones associated with ring-homebridge/ring-client-api/ring-mqtt/etc. It's very critical that the authorized device created is 100% a new device or the push notifications will not be enabled.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@benjackson1111
|
|
Thank you Cooper. I did something similar and it worked! The notification was faster than ever too. Like the moment I pressed the button and before the sound effect ended the gate was buzzing (what I have the button press do is “turn on” a sonoff ewelink relay that buzzes my security gate to let delivery people in). Used to take a second or two. I’ll test and see if it keeps working after homebridge restart. Thank you all!On Jun 4, 2023, at 4:21 AM, CooperCGN ***@***.***> wrote:
@benjackson1111
I couldn’t get it to work at first so here’s the steps I took
Removed Ring child bridge from HomeKit
Deinstalled the plug-in in Homebridge and did a full restart of my raspberry pi
Removed every device in the ring control center, even the associated iOS devices so list was completely empty
Installed beta plugin in Homebridge, linked it to my ring account and set it as child bridge
Restart Homebridge via config interface
Added child bridge and its devices to HomeKit again
working now
logging into ring app on my iPhone and iPad again
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@dgreif I pushed out an update to ring-mqtt that uses the beta ring-client-api and, so far, things seem pretty good. Of course, it's not even been a full 24 hours yet, but at least I'm not flooded with issues from upgrade problems so far. |
|
Working great on this end @dgreif thank you tons! Been over 24-hours and restarted server multiple times with no issues |
|
@tsightler I can confirm your latest ring-mqtt update has fixed my notifications issue and I have yet to see any other issue arise yet. |
|
@dgreif I've been able to make some changes to my Ring Doorbell and Cameras app for Homey to make it work with the I've just got one minor issue left, the authorisation (which is part of my app) still results in a device with the For now I've published a test version of my app to the Homey App Store and instructed my users in the Ring Doorbell and Camera topic on the Homey Community Forum on how to proceed. Thanks again for your great work! |
|
For me it's working flawless, notifications (ding and motion) survived multiple homebridge reboots during the last days. |
|
Running 11.7.7 When did you notice the problem? |
|
Latest beta looks to be working fine here, passed the 24H with no issues, will keep testing and report back. |
|
@dgreif Thanks again! 11.8.0-beta.0 works fine so far after resetting/reinstalling the child bridge. Notifications come through over the past 3 days after daily reboots. |
|
Given the positive results so far, I feel confident we have a working solution. I've released |
|
Same here - i’ve been testing rigorously and it hasn’t faulted since the beta
… On Jun 8, 2023, at 8:57 PM, Dusty Greif ***@***.***> wrote:
Given the positive results so far, I feel confident we have a working solution. I've released 11.8.0 with the changes. I'll keep this issue open for a few more days for additional feedback in case users have continued notifications issues after updating.
—
Reply to this email directly, view it on GitHub <#1218 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANIPG4VXAQ7EHSZ74OD7AGLXKKNKRANCNFSM6AAAAAAYFHZSUU>.
You are receiving this because you were mentioned.
|
|
Same, flawless since 11.8.0 beta - no notification issues for 5 days including multiple restarts. Thank you so much. |
|
Same, I’ve published a stable version of the Ring Doorbell and Camera app for Homey that uses 11.8.0-beta.0 so all users could benefit from it and no problems have been reported. Just reports how happy users are it’s working again. Thanks for your effort! |
|
So, I updated and no change. Do we need to refresh the token as well as updating?On Jun 8, 2023, at 11:59 PM, benjackson1111 ***@***.***> wrote:
Same here - i’ve been testing rigorously and it hasn’t faulted since the beta
On Jun 8, 2023, at 8:57 PM, Dusty Greif ***@***.***> wrote:
Given the positive results so far, I feel confident we have a working solution. I've released 11.8.0 with the changes. I'll keep this issue open for a few more days for additional feedback in case users have continued notifications issues after updating.
—
Reply to this email directly, view it on GitHub <#1218 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANIPG4VXAQ7EHSZ74OD7AGLXKKNKRANCNFSM6AAAAAAYFHZSUU>.
You are receiving this because you were mentioned.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>
|
Yes, you still must follow the steps from the Notification Troubleshooting section of the wiki to remove all previous devices and re-authenticate. The update will not fix accounts where notifications are not working already, but will hopefully prevent the problem from happening again in the future. |
|
Beta was working great, updated to stable and it's back to not working, do I need to refresh tokens again? |
|
If it was working on the beta, I'm surprised it would stop working on the full release. I'd suggest following the steps from the Notification Troubleshooting page and see if that fixes it |
I just loaded the 11.8.0 release version (I already had the 11.8.0-beta0 version working) and restarted the child bridge and all seems to be working as advertised. Great Job! |
Did that and it's working again. Weird that it stopped working after upgrading from beta, will keep monitoring. |
|
Glad to hear it's working @salemsayed! I think we can call this "done" 🎉 For anyone landing here with notification issues in the future, please ensure you are on the latest version and follow the Notification Troubleshooting steps. If you still have problems after that, please open a new GitHub issue. |
and others
In #1212, we confirmed that clearing existing devices and re-authenticating seems to fix Push Notifications. However, some users are finding that the notifications only work for ~24 hours before they need to do the process again. The native apps don't seem to have this issue, so they must be sending some update to keep notifications working long term. We should investigate this further and find a way to avoid the daily bug.
See #1212 (comment) for more details.
The text was updated successfully, but these errors were encountered: