Make a distributed query failed when timeout

[zhuoyuan-liu]

Currently, when the nodes push new distributed query results back to osctrl, it checks if the query is completed. However, for some of the nodes, it may become inactive and would never send the results back, which would result in the distributed query in active forever.

I would suggest adding a timeout section for each distributed query, i.e. if the query is not completed within X time, we mark the distributed query as failed/inactive and remove the relevant records in DistributedQueryExecution table.

[javuto]

This is a good idea, do you think the timeout should be per environment or globally?

[zhuoyuan-liu]

I think it would be great if we have some default values and accept different values for each distributed query. It would be great to combine with my another proposal. #529

[javuto]

Landed #536 for osctrl-admin, now you can run queries and specify a timeout in hours, and also no expiration as well. This is set per query.