Support for NVIDIA Nsight Systems Profiling of DJL Serving

[Lokiiiiii]

Description

This PR introduces support for performance profiling and debugging in DJL Serving using NVIDIA Nsight Systems. The changes enable detailed profiling and performance analysis to optimize the model server's operation.

Key Features

Debug Mode Activation

• Introduced a DEBUG_MODE environment variable:
  • When set, the model server will run with performance profiling enabled.
  • Ensures smoother debugging workflows.

Nsight Systems Integration

• Automatic installation of NVIDIA Nsight Systems when DEBUG_MODE is enabled.
• Nsight Systems version can be specified using the NSIGHT_VERSION environment variable.

Profiling Configuration

• Added environment variables to control profiling behavior:
  • NSYS_PROFILE_DELAY: Time delay (in seconds) before starting profiling, to capture relevant operations.
  • NSYS_PROFILE_DURATION: Duration (in seconds) of profiling. It is recommended to keep this under 10 minutes to avoid large report sizes.
  • NSYS_PROFILE_TRACE: Specify APIs and libraries to trace during profiling (e.g., cuda,nvtx,osrt,cudnn).

Report Handling

• Profiling reports are automatically saved and uploaded to an S3 bucket if the S3_DEBUG_PATH environment variable is provided.

Testing

• Verified the installation and usage of NVIDIA Nsight Systems in debug mode.
• Validated environment variable handling and error scenarios.
• Verified in SageMaker

Documentation

• Created the troubleshooting guide to include details on using the new debugging and profiling features.

[siddvenk]

Few questions:

• Is it possible to leverage this profiler on systems without NVIDIA GPUS? It looks like there are different profiles we can capture, so i'm curious if this has utility for Neuron/CPU images. If no, before install maybe we do a nvidia smi check or something similar to exit quickly on non gpu systems
• Not necessarily for this PR, but since this does require network access to download the tool at runtime, it won't work in network isolation use-cases. We may want to consider extending the current containers with a new debug build stage that installs this tool and can be used directly for JS. Thoughts?
• I would prefer a different flag name instead of DEBUG_MODE. LMI_DEBUG_MODE, LMI_ENABLE_PROFILER or something similar is clearer to me. I'm not sure if DEBUG_MODE would conflict with a flag in any of the deps we pull in

[Lokiiiiii]

* Is it possible to leverage this profiler on systems without NVIDIA GPUS? It looks like there are different profiles we can capture, so i'm curious if this has utility for Neuron/CPU images. If no, before install maybe we do a `nvidia smi` check or something similar to exit quickly on non gpu systems

I am no expert but the tool can be used for CPU profiling from what I have seen so far. So I don't think the test is necessary.

* Not necessarily for this PR, but since this does require network access to download the tool at runtime, it won't work in network isolation use-cases. We may want to consider extending the current containers with a new debug build stage that installs this tool and can be used directly for JS. Thoughts?

Good idea but requires a much deeper discussion -

1. How do we keep bloat minimal and containers cache-able ?
2. How do we get Hosting to copy debug artifacts ?

I am currently introducing this for developers' sake. If it proves to be truly useful, we can think about making this a self-serviceable customer feature.

* I would prefer a different flag name instead of `DEBUG_MODE`. `LMI_DEBUG_MODE`, `LMI_ENABLE_PROFILER` or something similar is clearer to me. I'm not sure if DEBUG_MODE would conflict with a flag in any of the deps we pull in

Ack. I will fix this in the next itr.