[techsupport] improve robustness

[stepanblyschak]

Signed-off-by: Stepan Blyschak [email protected]

What I did

1. Execute the command in a separate bash process under timeout. Due to with '--foreground' the child processes are not killed and still have the file descriptor opened to piped process which will basically hang forever.
   From the "timeout" man page:

       --foreground

              when not running timeout directly from a shell prompt,

              allow COMMAND to read from the TTY and get TTY signals; in
              this mode, children of COMMAND will not be timed out

So, a command like this:

timeout --foreground 5s some_process_that_spawn_childs_that_produce_output | tee some_file

Will actually hang forever. The '--foreground' option is still needed. So the proposed solution is to run that construct in another bash process:

timeout --foreground 5s bash -c "some_process_that_spawn_childs_that_produce_output | tee some_file"

2. Moved hw-mgmt-dump to collect_mellanox function and execute it under timeout.
3. Handle global timeout: When global timeout is reached the SIGTERM is sent to generate_dump process, upon SIGTERM the generate_dump process will finalize its work by creating a dump of what already has been collected, providing a dump even when timeout is reached. The generate_dump will be killed in case it cannot create dump after initial SIGTERM.

How I did it

I implemented the above mentioned features.

How to verify it

Artificially hang the FW, execute "show techsupport" and verify even in case timeout is reached the dump is generated.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

[liat-grozovik]

if timeout expired and file does not exist, we should cont but IMO should have an error message identify timeout expired and no hw mgmt file. .

[stepanblyschak]

Fixed

[liat-grozovik]

@yxieca , @qiluo-msft could you please help to signoff?

[Staphylo]

@stepanblyschak @liat-grozovik
Could you clarify the rationale for moving the hw-dump (/usr/bin/hw-management-generate-dump.sh) collection to be specific to mellanox products?
We found it to be really useful and ended up implementing this hook for our platforms.
We only realized now that this data is no longer collected, thus affecting support capabilities.
Would it be fine to move this collection back to common code or should we implement another hook in generate_dump for our products?