[OpenCL] Implement save/load pre-compiled programs

[echuraev]

Using pre-compiled programs might significantly improve inference time of the first run.

• Added methods SupportPreCompiledPrograms which reports if the module supports using pre-compiled programs.
• Method GetPreCompiledPrograms returns string with bytes of pre-compiled programs.
• Method SetPreCompiledPrograms allows user to pass pre-compiled programs to the module.

[tvm-bot]

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

• cc @elvin-n _{See #10317 for details}

_{Generated by tvm-bot}

[echuraev]

cc: @tqchen, @csullivan, @srkreddy1238

[echuraev]

@tvm-bot rerun

[srkreddy1238]

@echuraev thanks for this feature.

I too had a similar problem statement with CLML tuning cache management at runtime which need to be generated once and reused later. In CLML, there existed multiple sub graphs related to each tvm module and for now the tuning cache is indexed by subgraph symbol and are stored under one file by serializing them through DMLC::Strm. This works by maintaining one file per tvm module. This approach adds additional overhead for the user to maintain and specify the tuning cache for each tvm module.

I think we have a generalized problem statement here where there is a need of cache management for the tvm runtime.

Probably we could come up with an unified approach where

• There will be a tvm_runtime cache specified by environment variale or graph_runtime api interface.
• Each tvm compiled module will have a unique hash key generated at compile time and is accessible from tvm module interface.
• A new file utility interface to load/store the binary blobs generated by any runtime as a key & value pair.

The flow would be like

1: Runtime (OpenCL/CLML ..etc.) will form a key (concatenating [ tvm_module hash + runtime + purpose ...etc.] )
2: Try to fetch from runtime cache.
3: If not exist regenerate and save into cache.
4: Use the stored cache

This will simplify and minimize the end user hassle to a level of just specify a cache folder and relax.

In the implementation side we have

• tvm Module passing a unique key from compilation to runtime probably via grpah_json.
• Cache API to serialize and load/store the key, value paired binary blobs.
• Runtime specific changes to use this cache interface.

[elvin-n]

Supporting the idea of caching data, I am against of API introducing work with file system. We can provide serialize interfaces that users can implement and feed to TVM API or we can provide load/store functions operating with binary in memory, but it must not be file system dependent API.

[srkreddy1238]

Supporting the idea of caching data, I am against of API introducing work with file system. We can provide serialize interfaces that users can implement and feed to TVM API or we can provide load/store functions operating with binary in memory, but it must not be file system dependent API.

You mean, TVM will prepare in memory serialized blob for all the cache, application is responsible for retrieving followed by storing on their own and later user will input the the binary blob back to TVM (similar to binary input for load_params) ?

[echuraev]

@srkreddy1238 Thank you for your review! The idea of caching data looks promising, but I agree with @elvin-n that the TVM should provide only API for serializing objects to binary format but shouldn't work with file system. So user application should be responsible for writing/reading this serialized blob to/from disk.

Do you think that caching mechanism should be implemented in this PR? I would prefer to see such implementation in separate PR.

[srkreddy1238]

User management will impose additional challenge.

Every time the application should retrieve and save the same as we are unaware of change to the cache blob in the current module load. Or the application should query about a change due to the current run. I feel all these complicates the app interface.

@echuraev cache implementation incur more changes outside cl runtime and can be handled outside this PR scope once there is an agreement on the design.

[srkreddy1238]

A suggestion, otherwise good to go.

[srkreddy1238]

LGTM

[echuraev]

@tqchen could you please review this PR once again?