Porting to Linux

[snakelizzard]

I am porting the project to Linux. Seems easy, but I have several questions not covered in the readme.

1. What Onnx extension do you use in Axodox.MachineLearning.Shared/Assets? Am I understanding correctly, this is https://github.com/microsoft/onnxruntime-extensions?
2. What is custom_op_cliptok.onnx?
3. I loose dependency on axodox-common and rewrite Tensor methods toHalf/toFull to be done without DirectX. Do you have any other suggestions on porting?
4. I see the controlnet branch with recent commits. Can you please share a plan for future features? Are you planning to add more samplers?

[axodox]

1. Yes the onnx extension is from the mentioned library, and it is used to implement the clip tokenizer used in custom_op_cliptop.onnx, which is used for text tokenization in Stable Diffusion. It would be possible to implement the tokenizer on CPU as well, however I reused this one from a Microsoft example with C# + Stable Diffusion.
2. I suggest to keep the dependency on axodox-common as most of the revelant parts are Linux compatible, and also as I will rely on the lib going forward as well, so it will be hard to maintain the code if it is removed.
   a. Instead I suggest to provide some Linux implementations for a few methods, most importantly the texture resize and load functionality in TextureData. Otherwise much of the code covered by the PLATFORM_WINDOWS ifdef is not required for Stable Diffusion, or the MachineLearning lib and you can ignore those for the Linux port.
   b. You can keep using the DirectXMath.h on Linux with clang or gcc, there are tutorials to do so. So no need to port the half conversions.
3. My current plan is to add controlnet and related feature extractors (e.g. depth, openpose estimators), after that I will work on 3D model generation and visualization using the libs and adding stuff for that. I am also considering AI upscaling tools to be implemented in the coming months. I might also add more samplers, but that is not an immediate requirement for me to do my main target of 3D model generation.

[axodox]

If we build a linux version of the library we should also standardize the build process, packaging. Some questions to decide:

• Which compiler to use (e.g. clang, gcc)?
• Packaging mode (e.g. vcpgk, nuget)
• Build setup (with cmake)
• Configuration for the AppVeyor build so common versioning is kept

Some more questions:

• An alternative for WIC (Windows Imaging Component) for Linux, e.g. cairo, as this is the largest Windows dependency in the lib
• Support for different execution providers (e.g. CUDA) since DirectML does not support Linux outside WSL

And probably there are more questions to discuss, my Linux dev experience is not extensive.

[snakelizzard]

I can say about my use case. And my understanding.

• There is no big difference between clang/gcc until it comes to some specifics of optimization. The source code is still simple enough not to cause any problems. I build with gcc by default.
• I do not think packaging is a priority. It is nice to have a script for conan.io, but still, few people use it.
• On WIC: That is why I prefer to reduce dependency to axodox-common. There are only two things borrowed classes:
  • TextureData should not care about image formats. Each potential implementation Qt, Cairo/Pixman, WIC, OpenCV, Android native already has own way to make conversion.
  • aligned_allocator is header only. By the way, it is better to change _aligned_malloc to std::aligned_malloc in this class.
• I am going to try CUDA.

[axodox]

TextureData should not care about image formats. Each potential implementation Qt, Cairo/Pixman, WIC, OpenCV, Android native already has own way to make conversion.

It is also used to write image metadata and image resizing, however it might not be invoked for that directly from MachineLearning. So I can accept this analysis.

However, there are some more classes I think (e.g. Logger, file-io, events and threading related code) which are in use or will be used in the future. So architecturally common should be kept, even if some parts do not have windows versions and it will make the initial setup a little more complicated.

Any general code which is not directly tied to machine learning, should go into common.