New build system

[elegios]

This PR implements an approach for #707, that should be much more concise than the previous approach, and that additionally supports two test runners: make and tup.

The PR is presently a draft, for two primary reasons:

• Not all special test cases are implemented yet. Most of what's mentioned in Refactoring our builds and tests #707 is implemented, but we should have some more eyes on it, as well as specifications for the remaining test cases.
  • mi tune in particular appears to not respect the --output flag, which will cause some issues
• It has only been tested on my machine thus far, thus it's not entirely certain I correctly handle other setups. To test and see that things work on other machines I would appreciate the following:
  1. Checkout this branch.
  2. Test running make, make clean, and make test. The first two should work without issue, while the last should exhibit some test failures (feel free to look into the causes and add exceptions to misc/test-spec.mc if you have time and/or feel responsible for them, I don't necessarily have much time to look into it presently).
  3. Other make targets that could be useful to test, but aren't quite as important/different from before: install, uninstall, lint, fix, and cheat.

---

Update:

• This now includes all tests from the original suite as far as I can tell, with some exceptions:
  • The javascript tests ran diff between the two implementations, but the system only allows single input tests right now, so we can't quite do that right now.
  • The mlang-pipeline tests should test symbolize.mc, which was apparently part of the previous suite and worked, but now it does not work. I've not investigated, because no helpful error message is produced.
• The idea is to test every single .mc file in the repo, and this is mostly the case now, so there are many more tests than before. Some files are harder to test correctly though, e.g., the lrk generator generates parsers, but there's no convenient way to provide files to parse to test their semantics.
• It's still the case that it would be good to test on other systems before merging. It seems very solid on my machine, but "it works on my machine" hardly seems sufficient for this sort of thing.

[larshum]

I tried running the latest version on my Mac now. The make and make clean commands worked fine, but I got a slight error for make test:

exec misc/test --bootstrapped smart
realpath: illegal option -- -
usage: realpath [-q] [path ...]
make: *** [test] Error 1

The realpath command is given a --relative-to=... flag, which isn't supported by the ancient version used in MacOS (I don't know the exact version, but the manual page is from 2011). I describe how I worked around it below. After getting that to run, I managed to get some (boot) tests to run.

---

To fix the problem with realpath, you can install coreutils through brew install coreutils and then you also need to update the PATH as described there to overwrite the built-in versions with more recent ones.

Commands also provided by macOS and the commands dir, dircolors, vdir have been installed with the prefix "g".
If you need to use these commands with their normal names, you can add a "gnubin" directory to your PATH with:
  PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"

Alternatively, we could use grealpath for MacOS to avoid having to update the path.

[elegios]

Hm. It would probably be better if we could avoid adding extra dependencies if we don't really need to. In this case we know that . is inside $dir, i.e., the latter is a prefix of the former, so it might be possible to do some string manipulation in bash instead or something. I've pushed a version that does that now, could you test that it works for you as well? (ideally without the workaround you applied)

[larshum]

Hm. It would probably be better if we could avoid adding extra dependencies if we don't really need to. In this case we know that . is inside $dir, i.e., the latter is a prefix of the former, so it might be possible to do some string manipulation in bash instead or something. I've pushed a version that does that now, could you test that it works for you as well? (ideally without the workaround you applied)

Yes, that worked for me (without the workaround).

[johnwikman]

I have a couple of major and minor questions with this PR:

Major

• The new repository structure is OK, but I think that the test directory does not belong under src. That means that we are putting things like microbenchmarks and other miscellaneous examples together with the things that actually make up Miking. I'm content with src/stdlib, but I really think that we should have the test directory at a top level parallel to src. Can we change the tup config to handle this? Alternatively submit PRs to the tup's upstream repository to get the necessary functionality to handle this?
• All tests are now handled in misc/test-spec.mc. We are now essentially wholly relying on Miking to tell us if Miking works, which I see as a testing fallacy. Having the test-spec.mc is not completely wrong, and we are today already relying on Miking's utests to tell us if the code is working. What I am worried about is that we are locking ourselves into a system where it is only Miking that can tell us if it is working, i.e. a lack of automated external sanity checks. Can we add this somehow in tup or in another way?

Minor

• Why are there three different files for tup under src? Can't we just have a single one?
• The misc directory seems to contain quite a lot of fluff at the moment, small scripts that I don't care about. (gen-tup-rules, watch, with-tmp-dir, etc.) Can these be move to a separate directory? I don't think these belong together with the test spec.

[elegios]

1. This is definitely a thing we can do, it's just more fiddly. Right now the tests can look for everything under src (we can skip that prefix in the path, we only need one glob call, some other similarly minor things), in such a case it would have to look in both directories. It would also mean duplicating the tup-related files from src to test. My thinking with this arrangement is that we have a number of separate components/sub-projects, all of which live under src (because they're source code), some of which are relevant to the outside world (main, stdlib, maybe boot) and some are not (test, maybe boot).
2. This is a very valid point. The main mitigating factor is that all miking is doing during this testing is listing what tests to run and how, other tools actually run and check things.
3. Slightly simplified, tup expects one tupfile per directory in the project, but allows a tupdefault as a shorthand, any directory without a tupfile is treated as though the closest tupdefault was there. Tuprules is easy to include in other files, i.e., it's for single source of truth stuff. The top-level Tupfile defines how to build boot and mi itself, which is different from how all the tests work, so it can't be just the tupdefault file. Annoying, I agree, but this seemed the most reasonable solution to me. (If we split out test we need to either copy these three files to there, or move them to the root of the directory)
4. They can be moved. I'd argue to keep the test and watch scripts together, and it probably makes sense to have test-spec.mc in the same place, but beyond that it's just a matter of opinion where things are. Maybe misc/scripts for gen-tup-rules, elide-cat, repo-ignored-files, and with-tmp-dir, leave test, watch, and test-spec.mc directly under misc, and ideally find some place for mcore-comby.json?

Sidenote, my ideal for test and watch is if they were just commands of the top-level Makefile, but you can't pass arguments to make targets (conveniently) to my knowledge, so that's not an option. We could change it so we don't have make as the top entry, either adding some script (requiring ./whatever) or something like just (which would be another dependency, which is meh).

[johnwikman]

I think that answer my minor questions perfectly! Having these three tup-files is fine for the moment. If we insist on using tup (which has nice features) as our new build system, then we should also try to upstream changes to tup that make our lives easier in using the tool. Let's do open source the way it should be done 🙂

The second major question is still a bit open for me. Even though it's being piped to make, Miking is still acting as a man-in-the-middle telling make what to do. On top of this the test-spec.mc file is quite complex, imports a lot of features from stdlib, and is being run with the bootstrapped mi compiler instead of the OCaml reference. This presents a couple of issues to me:

• The complexity of the file makes it more prone to mistakes (even though it is well documented),
• something being broken in common parts of stdlib means that the test program doesn't run or that it may produce erraneous output, and
• almost everything that test-spec.mc is testing are its own dependencies due to running with mi.

I still have hesitations about this. If test-spec.mc was written in something like Python (or its statically typed cousin) then I would consider this resolved.