Replace Makefiles with Just

[YJDoc2]

This is an exploratory addition of just as a command runner.

Currently I have only added justfiles for existing makefiles, but I feel in future we can completely replace the scripts/* with just if it works out.

I personally feel that syntax of just is better than plain makefile/shell scripts, especially in case of if-else etc.

This PR is more as a rfc and testing out just rather than merging.

Also, adding just will add an dependency on just for our build system (as a whole, not only CI)

Comments and questions welcome!

Update : as discussed below, we are moving to just instead of Makefile, so that will be done in this PR itself.

[yihuaf]

I would vote in favor of just.

[utam0k]

I'm a super beginner at just. please tell me what is the most benefit to using just in youki's repo🙏

[YJDoc2]

Hey
So Just is a plain command runner. There is no efficiency or additional feature benefit from just instead of using makefiles as we are doing now, but I feel following differences are worthwhile :

• no need for .PHONY and such makefile particular stuff. just always runs commands and does not do a timestamp comparison. For most of our usecases, we also need the same as we don't really do manual timestamp based compiling . Instead cargo checks if file has been updated and re-compiles accordingly . Thus we must mark a lot of targets in makefile .PHONY to work. as you can see justfile looks and reads much cleaner.
• I feel it has a cleaner way to set and use variables compared to make
• I feel the if-else and such control structures are better in just than in make.

@yihuaf might also be able to add some more points.

Just to be clear, this PR is not intended for merging, this is just RFC for checking if this would be a good idea

[yihuaf]

makefile is two system in one (sort of). The first part is the recipe or command runner where make <command> runs a specific command/recipe. The second part is a build system where file changes are automatically detected. For youki and most of non-C/C++ projects, make is almost exclusively used as a command runner. In fact, the build-system part of the make would make things harder, such as the requirement to use .PHONY. With that being said, make is an extremely mature system. While there are a lot of quirkiness around using makefile exclusively as a command runner, almost all of these quirkiness are known and has known workarounds.

justfile is an attempt to build a new generation of command runner that aims to keep most of what makefile is doing well, and fix the quirkiness of make with a more ergonomic UX experience. It is written in rust so it is very popular among rust projects and the ecosystem. The author listed out how justfile improves on makefile in the readme here: https://github.com/casey/just.

@YJDoc2 already has a good list of pros. From my personal experience, I agree that the variable management is cleaner in justfile and embedding small scripts, especially bash and python becomes much easier. For example, there is no need to escape variables in the embedded bash script, whereas makefile requires $$() escape. It is much less error-prone in my opinion. I like how just handles recipe command with argument. For example, we can specify just build <debug/release> and it can be a single recipe rather than two.

The cons is introducing a new dependency where just needs to be installed upfront. just does make it easy to install as well as having a CI action to set things up. just has an include syntax that is only recently merged, so I don't know how well it works. Previously, just can only recursively call other justfile. Also, while just syntax is relatively easy, we do need to learn the new syntax whereas more people who have learned make.

All in all, I think the trade offs is worth it and embracing more into the rust ecosystem can be a good thing by itself.

[utam0k]

It is much less error-prone in my opinion. I like how just handles recipe command with argument. For example, we can specify just build <debug/release> and it can be a single recipe rather than two.

@YJDoc2 @yihuaf Thanks 😍 I really want this one.
I also understand the trade-offs. Even with that in mind, it would be worthwhile to introduce it. Justfile is a more fun way to program. This is one of youki's motivations.

However, I have one concern. I don't want to have to maintain justfile and Makefile together for a period of time. That would lead to confusion. Let's switch at once. So can we have a feature branch or a PR that switches them all at once?

[utam0k]

Hey @containers/youki-maintainers, could you share your thoughts on using justfile instead of Makefile? I would really appreciate your input as it could potentially impact the development experience. Thank you!

[yihuaf]

However, I have one concern. I don't want to have to maintain justfile and Makefile together for a period of time. That would lead to confusion. Let's switch at once. So can we have a feature branch or a PR that switches them all at once?

I agree with your concern. There are a few ways we can make this transition smooth, depending on how much time we can commit in the next a few weeks.

We don't have a lot of makefiles, so switching just the makefile over to justfile can be done in a single PR, along with the readme instructions and the CI changes. I am happy to commit the time to make the switch next week. @YJDoc2 has done a lot already with this PR. We can then make the improvements as follow up PRs, such as combining targets using more features from justfile.

Or, if we prefer a piecemeal approach, we can wrap the makefile with a top level justfile. We simply pass the arguements to justfile to makefile so we build an abstraction to hide the fact that we can using makefile underneath. This allows us to port over in multiple PRs and won't affect user experience.

I think either approach would work, but I think a single person should draft the PR to setup the readme, the CI, and the switch to use justfile. We should coordinate on this.

[utam0k]

We don't have a lot of makefiles, so switching just the makefile over to justfile can be done in a single PR, along with the readme instructions and the CI changes. I am happy to commit the time to make the switch next week. @YJDoc2 has done a lot already with this PR. We can then make the improvements as follow up PRs, such as combining targets using more features from justfile.

My preference is for this approach.

[YJDoc2]

Hey @utam0k @yihuaf Thanks for your comments!

I think I have already converted all the makefiles into corresponding Justfiles in this PR, I'll need to update the CI (so to completely remove the makefiles) and update the docs and readme. I'd be happy to do this in this week.

Just to note, this will add just as a necessary dependency for developing youki.

[YJDoc2]

Hey, I know this has been delayed a lot, but I got busy in some other stuff. Apologies 😓 I'll get this done by this weekend to a mergable state. Apologies again 😓

[utam0k]

Hey, I know this has been delayed a lot, but I got busy in some other stuff. Apologies 😓 I'll get this done by this weekend to a mergable state. Apologies again 😓

No need to apologize, but I'm looking forward to using the just you introduce. Please take your pace.

[l0rem1psum]

This might not value add anything to the current discussion, but has anyone explored the option of using cargo-make? Based on my limited experience, cargo-make looks and feels more promising. Not trying to discredit the current effort, just throwing things for discussion.

[yihuaf]

This might not value add anything to the current discussion, but has anyone explored the option of using cargo-make? Based on my limited experience, cargo-make looks and feels more promising. Not trying to discredit the current effort, just throwing things for discussion.

We had good experience with just previously on other projects. I have not used cargo-make before, but using toml to specify commands and scripts is a no go for me. It is way too restrictive and not powerful enough compared to just. It is OK for simple stuff, but not powerful enough for what we are looking for.

[yihuaf]

Hey, I know this has been delayed a lot, but I got busy in some other stuff. Apologies 😓 I'll get this done by this weekend to a mergable state. Apologies again 😓

Take your time and let us know if we can help out in anyway :)

[codecov-commenter]

Codecov Report

Merging #1823 (90959b4) into main (510cc8a) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1823   +/-   ##
=======================================
  Coverage   65.26%   65.26%           
=======================================
  Files         129      129           
  Lines       14800    14800           
=======================================
  Hits         9659     9659           
  Misses       5141     5141           

[YJDoc2]

!github easy-benchmark

[yihuaf]

Great!