Address RMMAllocator error for UCX Endoscopy Tool Tracking application

[mocsharp]

This PR addresses a RMMAllocator crash when running the application with realtime set to false (play the video as fast as possible).

[error] [rmm_allocator.cpp:190] Unexpected error while allocating memory [00007]('video_replayer_allocator') : std::bad_alloc: out_of_memory: RMM failure at:bazel-out/k8-opt/bin/external/rmm/_virtual_includes/rmm/rmm/mr/device/pool_memory_resource.hpp:424: Maximum pool size exceeded
[error] [memory_buffer.hpp:79] video_replayer_allocator Failed to allocate 1229760 size of memory of type 1. Error code: GXF_FAILURE

As @grlee77 suggested, the fix is to explicitly set the RMM Allocator's initial and maximum sizes.

Adds the ability to enable/disable data flow benchmarking through configuration.

[tbirdso]

@mocsharp could you please provide more details on the error and root cause? Are the RMMAllocator crash and benchmarking option related?

[mocsharp]

@mocsharp could you please provide more details on the error and root cause? Are the RMMAllocator crash and benchmarking option related?

Updated description.

[tbirdso]

One question on defaults, code looks good otherwise. Will wait for approval from @grlee77 to merge.

[tbirdso]

ping @grlee77 for review

[jjomier]

@grlee77 can you please review?

[grlee77]

Thanks. The change looks good to me. I will add an additional description motivating why it may be necessary.

@mocsharp, please either update the default value of benchmarking to false in the YAML or change the comment describing the default to be consistent. I will go ahead and approve, so you can merge after that without requesting another review

[grlee77]

@mocsharp could you please provide more details on the error and root cause? Are the RMMAllocator crash and benchmarking option related?

@tbirdso:
Copying some additional context from a prior discussion outside of GitHub. Item 2 below is why an allocator may need a bit larger memory capacity than one would naively estimate based on the tensors used in a single compute call.

In practice there can be some degree of "parallel" operation of operators even with the GreedyScheduler due to how CUDA kernels are launched aysnchronously from the host. Although the greedy scheduler can only call compute on one operator at a time, it is possible the compute method may return immediately after a CUDA kernel is launched allowing a subsequent compute method to be called while the work from that previous kernel is still being executed on the GPU. The ability to return once the kernel is launched is good from a performance standpoint, but it can lead to a couple potential issues that may cause confusion to application authors:

1. Times reported for individual operators by Data Flow Tracking or GXF's job statistics (HOLOSCAN_ENABLE_GXF_JOB_STATISTICS=true) reflect the time spent in compute which may be misleadingly short in the case the kernels launched continued to run on GPU after compute has returned. For instance, if a subsequent operator then does an operation like device->host copy that requires synchronization, the remainder of the kernel computation time from the compute call from the previous operator would show up in that downstream operator's compute .
2. If compute returns while computation is still being done on a tensor, an upstream operator is then free to be scheduled again. If that upstream operator was using a BlockMemoryPool or other pool, the memory from the prior compute call would still be in use. Thus the operator needs space to allocate a second tensor on top of the original one. This means the author has to set a larger number of required blocks (e.g. 2x as many) than they would have otherwise estimated.

I think v2.6 had some changes to InferenceOp where the operator's compute can return while the GPU kernels are still running (due to async launch of CUDA kernels) so this may be more likely to occur in newer releases than it was previously

[tbirdso]

Thanks all, looks good!