Healthcheck stopped after update Upstream #7652
Comments
|
Hi, I tried to debug the issue, I put a print in for loop block in file healthcheck.lua following below ...
self:renew_periodic_lock()
local cur_time = ngx_now()
for _, checker_obj in ipairs(hcs) do
if checker_obj.name == "5fb68818-d502-41ae-abcc-a41ff1405783:permission-dev" then
print("found permission-dev")
end
if checker_obj.checks.active.healthy.active and
(checker_obj.checks.active.healthy.last_run +
checker_obj.checks.active.healthy.interval <= cur_time)
then
...I updated upstream Conclusion
|
|
Hi, After searching about pairs and ipairs, I think issue is when I update upstream, old upstream object in hcs has been removed by GC, it means we have gaps in table => iterate over hcs stopped when got nil => new upstream object inserted by Ref: https://sodocumentation.net/lua/topic/676/tables#avoiding-gaps-in-tables-used-as-arrays I changed from ipairs to pairs, it working well for _, checker_obj in ipairs(hcs) do |
|
our project has the same problem！ |
|
by the way, how do i roll back kong to 2.2.0？ |
|
we found a new problem. when we update target by konga, and set target unhealth or health, konga show msg: [31merror: �[39mKongProxyController request error { message: 'Upstream serviceXXXXX has no balancer' } |
|
|
mark |
|
how to fix it? |
@PorcupineBoy You can workaround following the pull request |
now iam using the kong version was 2.7.0.0-enterprise-edition . |
|
When this PR will be merged ? We are hitting severe production issues due to active healthchecks being disabled when we update the upstream. Is this fix safe to apply ? We are using kong 2.5.1 |
|
I've ported the fix Kong/lua-resty-healthcheck#78 to lua-resty-healthcheck version 1.5.0 used in kong 2.8.0 here Kong/lua-resty-healthcheck#93, There are steps In my PR description how to reproduce the issue in kong 2.8.0. I hope it helps to have it merged. |
|
This issue was fixed thanks to @kayano. If you want to keep using Kong 2.x series, you need at least Kong 2.7.2 or 2.8.1 to have the fix included. |
Summary
Upstream information
Kong debug log
The text was updated successfully, but these errors were encountered: