[Smartswitch][Chassisd] Race condition causes incorrect update of control_plane_state and data_plane_state

[gpunathilell]

Description

Sometimes the control_plane_state and data_plane_state are updated incorrectly
when the DPU is powered off this data is set to down for data_plane_state and control_plane_state from the DPU_STATE key from the dpu https://github.com/sonic-net/sonic-platform-daemons/blob/c61323f41abc9afefcc581e0586ea30fe656a147/sonic-chassisd/scripts/chassisd#L1221
However, Sometimes this data is not updated when we run show system-health dpu all on the switch
since we are copying the complete data from the key and writing it back on the switch - , sometimes the data written by the DPU is overwritten - This could be the cause of the issue
https://github.com/sonic-net/sonic-platform-daemons/blob/c61323f41abc9afefcc581e0586ea30fe656a147/sonic-chassisd/scripts/chassisd#L787

Steps to reproduce the issue:

Use image with all Smartswitch PMON related PRs integrated on Switch and DPUs

1. Run show system-health dpu all
2. Make sure that data_plane_state and control_plane_state are up
3. Run the command again after powering off DPU, the states are still shown as up, even though dpu is powered off
   Issue is seen intermittently, not 100% reproducible

Describe the results you received:

Incorrect dpu states

Describe the results you expected:

Updated data_plane_state and control_plane_state

[oleksandrivantsiv]

@prgeor FYI