diff --git a/_posts/2021-01-09-Julia-1.6-what-has-changed-since-1.0.md b/_posts/2021-01-09-Julia-1.6-what-has-changed-since-1.0.md new file mode 100644 index 00000000..b5165720 --- /dev/null +++ b/_posts/2021-01-09-Julia-1.6-what-has-changed-since-1.0.md @@ -0,0 +1,629 @@ +--- +title: "Julia 1.6: what has changed since Julia 1.0?" +layout: default +tags: + - julia +--- + +Julia 1.0 came out well over 2 year ago. +Since then a lot has changed; and a lot hasn't. +Since Julia 1.0 was a commitment to no breaking changes; +but that is not to say no new features have been added to the language. + + + +Julia 1.6 is the first feature release since 1.0. +Prior to 1.0 all releases were feature releases; they came out when they were ready. +Since then all releases (except 1.6) have been timed releases. +The release candidate is cut from the main branch every 3 months; and they are released after an [extensive round of additional checking](https://github.com/JuliaLang/julia/pull/35846), which often takes weeks to months. +Julia 1.6 was soft-slated to be the new long-term-support (LTS) version that would have bug-fixes backported to it for the next few years. +The current LTS is Julia 1.0, which has now had 5 patch releases made. +Since it was going to be supported for a long-time people wanted to make sure everything good got in; thus it was a feature release. +The core developers have demurred on if 1.6 will actually be selected to be the new LTS (even if it wasn't it won't ascend to being LTS til it stops being the current Stable). + +My impression now is that they feel like it has too many cool new things; and that a few things didn't quite make it in even with the extended release cycle. +So it's looking likely to me that 1.7 will actually be the LTS; but that it might also be a feature release -- possibly this time a much shorter release period than usual. +In practice I think for a lot of package maintainers 1.6 will be a LTS, in that that is the oldest version they will make sure to continue to support. +There have been too many cool new things (as this post will detail) to stay back to only 1.0 features. +Already a lot of packages have dropped support for Julia versions older than 1.3. + + +This post is kind of a follow-up to my [Julia 1.0 release run-down]({{site.url}}//2018/06/01/Julia-Favourite-New-Things.html). +But it's going to be even longer, as it is covering the last 5 releases since then; and I am not skipping the major new features. +I am writing this not to break down release by release, +but to highlight features that had you only used Julia 1.0, you wouldn't have seen. +Full details can be found in the [NEWS.md](https://github.com/JuliaLang/julia/blob/backports-release-1.6/NEWS.md), and [HISTORY.md](https://github.com/JuliaLang/julia/blob/backports-release-1.6/HISTORY.md) + + +## Front-end changes +### Soft-scope in the REPL +Julia 1.0 removed the notion of soft-scope from the language. +I was very blasé about the change to for-loop bindings in my [1.0 release post](https://www.oxinabox.net/2018/06/01/Julia-Favourite-New-Things.html#for-loop-variable-binding-changes). +In fact, I didn't even mention this particular change. +It was [#19324](https://github.com/JuliaLang/julia/pull/19324) for reference. + +This was undone in in Julia 1.5 with [#33864](https://github.com/JuliaLang/julia/pull/33864) for in the REPL only. +Now in the REPL, assigning to a global variable within a for-loop actually assigns that variable, rather than shadowing it with a new variable in that scope. +The same behavior outside the REPL now gives a warning. + +Personally, this change never affected me because I never write for-loops that assign variables at global scope. +Indeed basically all I code I write is inside functions. +But I do see how this causes problems for some interactive workflows, especially e.g. when demonstrating something. +See the [main GitHub issue](https://github.com/JuliaLang/julia/issues/28789) and the longest [Discourse thread](https://discourse.julialang.org/t/another-possible-solution-to-the-global-scope-debacle/15894), though there were many others. +It took over a year of discussion to work out the solution, particularly because many of the more obvious solutions would be breaking in significant ways. + +### Deprecations are muted by default + +This was me in [this PR](https://github.com/JuliaLang/julia/pull/35362). +It's not something I am super happy about. +Though I think it makes sense. +Using deprecated methods is actually fine, as long as your dependencies follow SemVer, and you use `^` (i.e. default) bounding in your Project.toml's `[compat]`, which everyone does, because it's enforced by the auto-merge script in the General registry. + +Solving deprecations is a thing you should chose to do, actively. +Not casually, when trying to do something else. +In particular, when updating to support a new major release of one of your dependencies, you should run your (integration) tests to see if they succeed. +If they don't succeed, then one removed support for the new version and run Julia with deprecation warnings turned on (or set to error); to identify and fix them. +Probably also you are checking the release notes, since you are intentionally updating. + +The core of the reason is because it was actually breaking things. +Irrelevant deprecation warning spam from dependencies of dependencies was causing JuMP and LightGraphs to become too slow to be used. +Since they were from dependencies of dependencies the maintainers of JuMP and LightGraphs couldn’t even fix them, let alone the end users. + +Deprecation warnings are still turned on by default in tests, which makes sense since the tests (unlike normal use), are being run by maintainers of the package itself, not its users. +Though it still doesn't make a huge amount of sense to me, since spam from deprecation warnings floods out your actual errors that you want to see during testing. +For this reason Invenia (my employer) has disabled deprecation warnings in the tests for all our closed source packages, and added a new test job that just does deprecation checking (set to error on deprecation and with the job allowed to fail, just so we are informed). + +Hopefully one day we can improve the tooling around deprecation warnings. +An ideal behavour would be to only show deprecation warning if directly caused by a function call made from within the package module of your current active environment. +I kind of know what we need to do to the logger to make that possible, but it is not yet something I have had time to do. + + +### Code highlighting for `code_llvm` and `code_native` +This was added in [#36634](https://github.com/JuliaLang/julia/issues/36634). +This functionality was first implemented in [ColoredLLCodes.jl](https://github.com/kimikage/ColoredLLCodes.jl), where it worked by monkey-patching the InteractiveUtils stdlib. +That package does still work on Julia 1.0. + + +**Julia 1.0:** + +![Julia 1.0 code_llvm]({{site.url}}/posts_assets/Julia-1.0-1.6-changes/julia-1.0-code-llvm.png) + +**Julia 1.6:** + +![Julia 1.6 code_llvm]({{site.url}}/posts_assets/Julia-1.0-1.6-changes/julia-1.6-code-llvm.png) + +The REPL itself still doesn't have syntax highlighting for Julia code though. +The [OhMyRepl](https://github.com/KristofferC/OhMyREPL.jl) package does provide that, and works all versions of Julia. +It is a lot more than a series of regexes though, so I don't think we are going to see it built into Julia too soon. +Probably one day, though, as its big dependencies are also required if the parser wants to move to be written in Julia (though I also don't expect that any time soon). + +### Clearer Stacktraces + +Just like colored `code_llvm` colored stack-traces also originated in a package that was doing some nasty monkey-patching: [ClearStacktrace.jl](https://github.com/jkrumbiegel/ClearStacktrace.jl). +This was added into Base itself in [#36134](https://github.com/JuliaLang/julia/pull/36134). + +**Julia 1.0:** + +![Julia 1.0 stacktrace]({{site.url}}/posts_assets/Julia-1.0-1.6-changes/julia-1.0-stacktrace.png) + +**Julia 1.6:** + +![Julia 1.6 stacktrace]({{site.url}}/posts_assets/Julia-1.0-1.6-changes/julia-1.6-stacktrace.png) + +The first thing you probably notice is the colored package names, to make it more clear where the error is coming from. +You also should notice the dimming of type parameters to make complicated types easier to read. +Also the addition of argument names, and showing functions with keyword arguments as they are written, rather than as a weird `#DataFrame#654` internal function. +The whole thing just looks more modern and polished. + +### Time Traveling Debugger + +[Julia 1.5 added built-in support](https://julialang.org/blog/2020/05/rr/) for [`rr`](https://rr-project.org/) (Linux only for now). +This is not useful so much for debugging Julia code, but for debugging Julia itself. +If you get into one of the very rare situations where the language is doing something truly nonsense; you can send a bug report recorded via `rr`, and someone can debug *exactly* what is happening. +Event to the extent of [diagnosing faulty RAM](https://julialang.org/blog/2020/09/rr-memory-magic/). +`rr` is an impressively cool piece of tech; it boils down to the fact that linux's `fork` is incredibly cheap, because it takes advantage of [copy-on-write](https://en.wikipedia.org/wiki/Copy-on-write); so it basically `fork`s before every system call, and serializes what is going on; so you have a full track of everything that happened. +The `rr` integration is really nice and easy to use; I had someone who was part of our architecture and operations team (i.e. who writes way more CloudFormation than Julia) run it to submit a support request; and they had no real troubles. + + +## Syntax + +### `import as` + +>The syntax import A as B (plus import A: x as y, import A.x as y, and using A: x as y) can now be used to rename imported modules and identifiers ([#1255](https://github.com/JuliaLang/julia/issues/1255)). + +This will be familiar to Python folks who love to do: `import numpy as np`. +I hope we never see it used that ubiquitously in julia. +I am quite happy with `using Foo` which imports into scope everything the author of `Foo` exported. +Python people find that super weird and scary; but it's fine. +Further, most things you use don't come from the module that defined them anyways; they are overloaded functions e.g. from `Base`. +The real value of `import FooBar as fb` is not to have a short abbreviation so you can do `fb.quux`. +That was already possible via `const fb = FooBar`. +It is to handle cases where the package name itself conflicts with an identifier. +For example (as has often occured) if one has uses a `Pipe` in the REPL, and then later wants to load [Pipe.jl](https://github.com/oxinabox/Pipe.jl/) via `import Pipe` then one gets a name clash before it can be resolved. +Now one can do `import Pipe as PipingPipe`. + +### NamedTuple/keyword arguments automatic naming +This feature felt weird when I first read about it, but it has quickly grown on me. +How often do you write some code that does some processing and calls some other method passing on some of its keyword arguments? +For example: +```julia +# Primary method all others redirect to this +foo(x::Bar; a=1, b=2, c=3) = ... + +# Method for if x is given as components +foo(x1, x2; a=10, b=20, c=30, comb=+) = foo(Bar(comp(x1, x2)); a=a, b=b, c=c) +``` +This new feature allows one to avoid writing `(...; a=a, b=b, c=c)`, and instead write `(...; a, b, c)`. +This does come with the requirement to separate keyword arguments from positional arguments by `;`, but I have always done this, and the [BlueStyle guide requires it](https://github.com/invenia/BlueStyle#keyword-arguments). +It feels like we are fully leveraging the distinction of keyword from positional arguments by allowing this. +In contrast, the distinction vs e.g. C# and Python that allow any positional argument to be passed by name, is to make the name of positional arguments not part of the public API, thus avoiding changing it being a breaking change. + +The same syntax can be used to create `NamedTuple`s. +```julia +julia> product = ["fries", "burger", "drink"]; + +julia> price = [2, 4, 1]; + +julia> (;product, price) +(product = ["fries", "burger", "drink"], price = [2, 4, 1]) +``` +This is particularly cool for constructing `NamedTuple`s of `Vector`s, which is a valid [Tables.jl Table](https://tables.juliadata.org/stable/#Tables.columntable). + +It is interesting to note that for the logging macros introduced in Julia 1.0, this is how they have always worked. +E.g. `@info "message" foo bar` and `@info "message" foo=foo bar=bar` display the same, which has always felt natural. + +### Lowering of `'` +In Julia 1.0 `'` was lowered directly into a call to `Base.adjoint`. +This meant it was impossible to redefine what `'` meant in your current module. +Now it lowers to a call to `var"'"`, which is something you can overload. +People often think it is cool to overload this to automatic differentiation so that `f'(x)` gives you the derivative of `f(x)`. +Furthermore, it can now have Unicode suffixes added to it, to define a new suffix operator. +For example, the one in `Base` which makes `A'ᵀ` give `transpose(A)` (rather than the adjoint/hermitian transpose which is returned by `A'`). + +## Performance: References to the Heap from the Stack +This was promised in 2016 as a feature for 1.0 (released 2018), but we actually didn't get it until 1.5 (released 2020) with [#33886](https://github.com/JuliaLang/julia/issues/33886). +In short, the process of allocating memory from the heap is fairly slow*, whereas allocating memory on the stack is basically a no-op. +Indeed, Julia benchmarking tools don't count allocations on the stack as allocations at all. +One can find extensive write ups of heap vs. stack allocations and how it works in general (though some mix the C specific factors with the CPU details). +In Julia all mutable objects live are allocated on the heap. +Until recently, immutable objects that contained references to heap allocated objects also had to live on the heap, i.e. only immutable objects with immutable fields (with immutable fields with...) could live on the stack. +But with this change, all immutable objects can live on the stack, even if some of their fields live on the heap. +An important consequence of this is that wrapper types, such as the `SubArray` returned from `@view x[1:2]`, now have no overhead to create. +I find that in practice this often adds up to a 10-30% speed-up in real world code. + +(* It's actually really fast, but it is the kind of thing that rapidly adds up; and it is slow vs. operations that can happen without touching RAM.) + + +## Pkg stdlib and the General Registry +Writing this section is a bit hard as there is no NEWS.md nor HISTORY.md for the Pkg stdlib; and the general registry is more policy than software. +However, some of the biggest changes have been in maturing out Pkg3, and its surroundings. + + +### BinaryBuilder, Artifacts, Yggdasil and JLL packages + +This story was beginning to be told around Julia 1.0 time, +but it wasn't really complete nor built into Pkg until Julia 1.3. +You can read the documentation on [Artifact](https://julialang.github.io/Pkg.jl/v1/artifacts/), [BinaryBuilder](https://github.com/JuliaPackaging/BinaryBuilder.jl), and [Yggdasil](https://github.com/JuliaPackaging/Yggdrasil) for full details. +This is the story about how Julia works with binary dependencies. +As of now, it works really well. + +Sometimes people misunderstand the claims about Julia solving the two language problem as saying everything should be rewritten in Julia. +Which is far from the truth, ain't noone got time for that. +Julia has always had great foreign function interfacing (FFI); like [`ccall`](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/) (as well as [PyCall](https://github.com/JuliaPy/PyCall.jl/), [RCall](https://github.com/JuliaInterop/RCall.jl), [JavaCall](https://github.com/JuliaInterop/JavaCall.jl), and a bunch of others, for non-C-style binaries). +While FFI makes it easy to call binaries, how about actually getting them onto your machine? + +Before Julia 1.0, the standard was to run arbitrary code in the `deps/build.jl` file, which often used [BinDeps.jl](https://github.com/JuliaPackaging/BinDeps.jl/) to download and compile things. +Around 1.0, this changed to having the `deps/build.jl` call [BinaryProvider.jl](https://github.com/JuliaPackaging/BinaryProvider.jl/) to download a compiled binary built with [BinaryBuilder.jl](https://github.com/JuliaPackaging/BinaryBuilder.jl), and store that alongside the package code. +In Julia 1.3+, the Artifacts system basically brought the BinaryProvider part into the package manager. +Now the package manager controls the downloads, stores them in a controlled location to avoid massive duplication, and allows for full compat control. +No more running arbitrary code during installs. + +I feel it is worth really mentioning [BinaryBuilder](https://github.com/JuliaPackaging/BinaryBuilder.jl) here. +While it has existed since before 1.0, it's really grown. +It's a super smooth cross-compilation environment, that can used to build binaries for every platform Julia runs on. +This is an amazing tool, built on [containerization](https://en.wikipedia.org/wiki/OS-level_virtualization). +It's the kind of build tool one dreamed on 10 years ago. +It's far more general than Julia, I know people have at least experimented with using it with Nim. +I hope more things start using it. + +### Temporary Environments +Julia 1.5 added `pkg> activate --temp` which will create and activate a temporary environment. +This environment is deleted when Julia exists. +This is incredibly handy for: + + - Reproducing an issue + - Answering questions on Stack Overflow etc + - Checking the behavior of the last release of a package you currently have `dev`ed. + - Quickly trying out an idea + +Remembering that with Pkg, installing a version of a package you have installed before is incredibly fast. +Pkg doesn't download it again, it basically just points to the existing version on disk. +So using `activate --temp` to quickly try something is indeed quick. + +A more hacky use of it is after loading (via `using`) a package that you have `dev`'ed, you can `activate --temp` and install some of your test-time dependencies to reproduce a test failure without adding them to the main dependencies in the `Project.toml`. +Though there is a better way if you use `test/Project.toml`. + +### Test Dependencies in their own Project.toml +In Julia 1.0 and 1.1, [test specific dependencies are listed in the `[extras]` section](https://julialang.github.io/Pkg.jl/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.0-and-1.1), and under `[targets] test=[...]`; with their compatibility listed in the main `[compat]`. +It seemed like this might be extended for other things in the future, perhaps documentation. +But it was found for documentation in particular that a separate `Project.toml` worked well (as long as you `dev ..` into its `Manifest.toml` so it uses the right version of the package it is documenting). + +The new `test/Project.toml` extends that idea. +One part of that extension is to remove the need to `dev ..`, and other part is to make available all the main dependencies. +It actually works on a different mechanism than the docs. +It relies on stacked environments, which is a feature Julia has had since 1.0 via `LOAD_PATH`, but that is rarely used. + + +One advantage of this is that you can activate that Project.toml, on top of the your existing environment by adding the test directory to your `LOAD_PATH`. +I wrote some more details on exactly how to do that on [Discourse]( +https://discourse.julialang.org/t/activating-test-dependencies/48121/7?u=oxinabox). +It feels kind of hacky, because it is. +At some point there might be a nicer user interface for stacked environments like this. + + +### Full Transition off METADATA.jl and REQUIRE files, and onto the General Registry +Even though Julia 1.0 had Pkg3 and was supposed to use registries, for a long time the old Pkg2 [METADATA.jl](https://github.com/JuliaLang/METADATA.jl) pseudo-registry was used as the the canonical source of truth. +Registering new releases was made against that, and then a script synchronized the [General](https://github.com/JuliaRegistries/General) registry to match it. +This was to allow time for packages to change over to supporting Julia 1.0, while still also making releases that supported julia 0.6. +It wasn't until about a year later that this state ended and the General registry became the one source of truth. + +This was kind of sucky, because a lot of the power of Pkg3 was blocked until then. +In particular, since the registries compat was generated from the REQUIRE file, and the REQUIRE files had kind of [gross requirement specification](https://docs.julialang.org/en/v0.6/manual/packages/#Requirements-Specification-1), and everyone basically just lower-bounded things, or didn't bound compat at all. +Which made sense because with the single global environment that Pkg2 had, you basically needed the whole ecosystem to be compatible, so restricting was bad. +But Pkg3/Project.toml has the much better default [`^`-bounds](https://julialang.github.io/Pkg.jl/v1/compatibility/#Version-specifier-format-1) to accept only non-breaking changes by SemVer. +And per project environments, things don't all have to be compatible -- just things used in a particular project (rather than every project you might ever do). + +#### Automatic Merging +Initially after the transition, all registry PRs to make a new release needed manual review by one of the very small number of General registry maintainers. + +I am a big fan of [releasing after every non-breaking PR](https://www.oxinabox.net/2019/09/28/Continuous-Delivery-For-Julia-Packages.html). +Why would you not want those bug-fixes and new features released? +It is especially rude to contributors who will make a fix or a feature, but then you don't let them use it because you didn't tag a release. +Plus it makes tracking down bugs easier, if it occurs on a precise version you know the PR that caused it. +But with manual merging it feels bad to tag 5 releases in a day. +Now we have automatic merging, so it is fine. +At time of writing, [ChainRules.jl](https://github.com/JuliaDiff/ChainRules.jl/) was up to 67 releases. + +The big advantage of automatic merging is that it comes with automatic enforcement of standards. +In order to be auto-mergeable, some good standards have to be followed. +One of which is that that no unbounded compat specifiers (e.g. `>=1.0`) are permitted; and that everything must have a compat specifier (since unspecified is same as `>=0.0.0`) +That one is particularly great, since if one adds specifiers later that can't be met, then it can trigger downgrades back to incredibly old versions that didn't specify compat and that almost certainly are not compatible. + +To deal with that particular case, retro-capping was done to retroactively add bounds to all things that didn't have them. +This was painful when it was done, since it rewrote compat in the registry, which made it disagree with the `Project.toml` in package repositories, which is always confusing. +But now that it is done, it is good. + +#### Requirement to have a `Project.toml` +Finally the ability to `pkg> dev` packages that had only a REQUIRE and no `Project.toml` was removed in Julia 1.4. +You can still `pkg> add` them if they were registered during the transition period. +But to edit the projects they must a `Project.toml` file, which is fine since all the tools to register releases also require you to have `Project.toml` now. + + + +### Resolver willing to downgrade packages to install new ones (Tiered Resolution) + +Until the [tiered resolver](https://github.com/JuliaLang/Pkg.jl/pull/1330) was added, Julia would not change the version of any currently installed package in order to install a new one. +For example, consider: + + - **Foo.jl** + - v1 + - v2 + - **Bar.jl** + - v1 compatible with Foo v1 + - v2 compatible with Foo v2 + - **Qux.jl** + - v1: compatible only with Foo v1 + +If you did `pkg"add Foo Bar Qux"` you would end up with **Foo** v1, **Bar** v1, and **Qux** v1. +But if you did first: `pkg"add Foo Bar"` (which would install **Foo** v2, and **Bar** v2), +and then did `pkg"add Qux"`, +then on Julia 1.0 you would get an error, as it would refuse to downgrade **Foo** and **Bar** to v1, as is required to allow **Qux** to be installed. +This meant that the package manager is effectively stateful, which turns out to be really counter intuitive. +The way to resolve this in practice was to delete the `Manifest.toml` and do it again as a single action. +This was a significant problem for [test-time dependencies](https://github.com/JuliaLang/Pkg.jl/issues/1352), where if you had a test-time dependency with an indirect dependency shared with a main dependency of the package, but that was only compatible with an older version of the indirect dependency, you would be unable to run tests as resolving the test-time dependency would fail. + +With the new tiered resolved, it will try a number of methods to relax existing versions. +It still wants to avoid changing the versions of currently installed packages if possible, but if that is required then it will do so. +It will first attempt to install without any currently installed package versions being changed, then will try relaxing the constraints to allow changing the versions of indirect dependencies, then to allowing changing the versions of direct dependencies. +This ends up being far more intuitive: if there is a compatible set of package versions then they will be found, regardless of whether new packages are added all at once or one at a time. + +### Precompilation + +Precompilation has been enhanced a lot. +That is not the compilation that runs the first time a function is used in a session, but rather the stuff that runs the first time a package is loaded in an environment. + +For a start, the precompilation cache no longer goes stale every time you swap environments. +This was a massive pain in Julia 1.0, especially if you worked on more than one thing at a time. +This was fixed in 1.3 to have a separate cache for each environment. +It's easy to forget this one, but it is actually one of the biggest usability enhancements since 1.0. + +More dramatic, and less easy to overlook, is the parallelism of precompilation added in 1.6. +Precompiling a package requires precompiling all its dependencies first. +This is now done in parallel, and is automatically triggered when you complete Pkg operations, in contrast to happening serially the first time a package is loaded. +No more waiting 5 minutes the first time you load a package with a lot of dependencies. +Further, the spiffy animated output shows you what is precompiling at any given time, as well as a progress bar, which makes it feel (even) faster. + + + +### Improved conflict messages +This is one of my own contributions. +Julia 1.0 conflict messages are a terrifying wall of text. + +**Julia 1.0:** + +![Julia 1.0 conflict log]({{site.url}}/posts_assets/Julia-1.0-1.6-changes/julia-1.0-conflict.png) + +**Julia 1.6:** + +![Julia 1.6 conflict log]({{site.url}}/posts_assets/Julia-1.0-1.6-changes/julia-1.6-conflict.png) + +The two main changes are the use of colors, and compressing the version number ranges. +No more giant red wall of numbers. + + +Colors were added to make it easier to see which version numbers are referring to which package. +There is a bit of a problem in that it isn't easy to make sure the colors don't get reused, as there are not many in the 16 color list (especially when you skip a few like black, white and error red). +Due to how Pkg constructs its error messages, basically every package in the dependency graph gets some message prepared for it, but not displayed, so just assigning them a color in turn gets it to loop around which still results in colors being reused. +There is a way to fix that, but it is a big change to add structured log messages that are colored only once when they are displayed. +We decided after much debate to assign colors based on the hash of the package name and shortened UUID. +This means things have consistent colors for any packages still listed, even after you have resolves some conflicts. +I think this is going to be a subtle improvement on ease of use. +As a cool hack, you can actually change the color list used. +To get the much larger color list I wanted to use originally you can do: +```julia +append!(Pkg.Resolve.CONFLICT_COLORS, [21:51; 55:119; 124:142; 160:184; 196:220]) +``` + + +The other change was consecutive lists of version numbers were compressed to continuous ranges. +In 1.6, these ranges are split only if there is a version that exists between them that is not compatible. +So normally we get just a single range, since compatibility is typically monotonic. +In contrast, in 1.0, they were split if there was a potential version that could exist that is not compatible (even if that version doesn't currently exist). +This means they split every time the rightmost nonzero part of the version number was incremented. +For something with a lot of pre-1.0 versions, that is a lot of numbers. +I think the new method is much cleaner and easier to read. + + +## Time To First Plot +People often complain about the "Time To First Plot" (TTFP) in Julia. +I personally have never minded it -- by the time I am plotting something, I have done minutes of thinking so 10 seconds of compilation is nothing. +Plotting, it turns out, is basically a really hard thing for a compiler. +It is many, many, small methods, most of which are only called once. +And unlike most Julia code, it doesn't actually benefit all that much from most of the work Julia's JIT is normally doing to specialize things -- plotting itself isn't in the hot-loop. +To make a long-story short, plotting is the postcard example for Julia needing to compile things before it can run them. + +As a really hacky way to time this, consider the following timing. +You can see I did `@time @time ...` and two numbers were displayed. +The first number is how long from the start of the command until the `using Plots` and `plot(...)` command finished. +The second is that, plus how long it took before it could show the output of the first. +It's worth looking at this later because it turns out that invalidations (see below) made the REPL have to recompile a bunch of display code. + +**Julia 1.0:** +```julia +julia> @time @time using Plots; plot(1:0.1:10, sin.(1:0.1:10)) + 7.168878 seconds (15.87 M allocations: 941.114 MiB, 7.36% gc time) + 9.281561 seconds (25.74 M allocations: 1.393 GiB, 7.88% gc time) +``` +**Julia 1.6:** +```julia +julia> @time @time using Plots; plot(1:0.1:10, sin.(1:0.1:10)) + 4.490434 seconds (8.24 M allocations: 583.285 MiB, 7.13% gc time, 32.80% compilation time) + 4.533302 seconds (8.36 M allocations: 590.949 MiB, 7.06% gc time, 33.41% compilation time) +``` +So it's a bit better than 2 times faster on my machine. + +Note also that in the above timing I had already run precompilation, which seems fairest. +Though see below, precompilation caches often would be deleted in 1.0; and conversely are created much faster in 1.6 due to parallelization. + +### Invalidations + +Consider a function `foo` with a method `foo(::Number)`. +If some other function calls it, for example `bar(x::Int) = 2*foo(x)`, the JIT will compile in an instruction for exactly the method instance to call (assuming type inference works) -- a fast static dispatch, possibly even inlined. +If the user then defines a new, more specific method `foo(::Int)`, then the compiled code for `Bar` need to be invalidated so it will call the new one. +It needs to be recompiled -- which means anything that statically dispatches to it needs to be recompiled, and so forth. +This is an invalidation. +It's an important feature of the language. +It is key to extensibility. +It doesn't normally cause too many problems, since generally, basically everything is defined before anything is called, and thus before anything is compiled. + +A notable exception to this is Base an the other standard libraries. +These are compiled into the so-called system image. +Furthermore, methods in these standard libraries are some of the most overloaded, thus most likely to be invalidated. + +A bunch of work has gone into dealing with invalidations better. +Not just point-fixes to remove calls that were likely to be invalidated, but several changes to the compiler. +One particular change was not triggering cascading invalidations for methods that couldn't actually be called due the being ambiguous. +As a result, a lot of user code that triggered invalidations on 1.5 no longer does so on 1.6. +The end result of this is faster compilation after loading packages, since it doesn't have to recompile a ton of invalidated method instances. +i.e. decreased time to first plot. + +This has had a huge effect on [Revise.jl](https://github.com/timholy/Revise.jl) +which started to take a second or so to load when it gained the dependency on [JuliaInterpreter.jl](https://github.com/JuliaDebug/JuliaInterpreter.jl); which isn't much, but when you do it every time you start Julia it is an annoying lack of snappiness. +But thanks to this work, JuliaInterpreter, and thus Revise, now load in a flash. + +A full discussion on the invalidations work can be found in [this blog post](https://julialang.org/blog/2020/08/invalidations/). + +### Per-Module Optimization Flags +>The compiler optimization level can now be set per-module using the experimental macro Base.Experimental.@optlevel n. For code that is not performance-critical, setting this to 0 or 1 can provide significant latency improvements ([#34896](https://github.com/JuliaLang/julia/issues/34896)). + +> Compilation and type inference can now be enabled or disabled at the module level using the experimental macro Base.Experimental.@compiler_options ([#37041](https://github.com/JuliaLang/julia/issues/37041)). + +Recall that I said most of Julia compilation isn't even that useful for plotting, since making it run fast isn't a priority (but loading fast is). + +`@optlevel` only controls which LLVM optimization passes run; which is right at the end of the compilation pipeline. +Turning off compilation and type inference on the other hand turns off a ton more. + + +## Internals +### Manually Created Back-edges for Lowered Code Generated Functions +This is a very niche and not really at all user-facing feature. +To understand why this matters, it's worth understanding how Cassette works. +I wrote a [blog post on this a few years ago](https://invenia.github.io/blog/2019/10/30/julialang-features-part-1#making-cassette). + +In quite the opposite, Julia 1.3 allowed for the creation of more invalidations through allowing back-edges to be manually attached to the `CodeInfo` for `@generated` functions that return lowered code. +Back-edges are the connections from methods back to each method instance that calls them. +This is what allows invalidations to work, as when a method is redefined, it needs to know what things to recompile. +This change allowed those back-edges to be manually specified for `@generated` functions that were working at the lowered code level. +This is useful since this technique is primarily used for generating code based on the (lowered) code of existing methods. +For example, in [Zygote](https://github.com/FluxML/Zygote.jl), generating the gradient code from the code of the primal method. +So you want to be able to trigger the regeneration of this code when that original method changes. + + +Basically, this allows code that uses [Cassette.jl](https://github.com/JuliaLabs/Cassette.jl), and [IRTools.jl](https://github.com/MikeInnes/IRTools.jl) to not suffer from [#265](https://github.com/JuliaLang/julia/issues/265)-like problems. +A particular case of this is for [Zygote](https://github.com/FluxML/Zygote.jl), where redefining a function called by the code that was being differentiated did not result in an updated gradient (unless `Zygote.refresh()`) was run. +This was annoying for + +Other things that this allows are two very weird packages that [Nathan Daly](https://github.com/NHDaly) and I came up with at the JuliaCon 2018 hackathon: [StagedFunctions.jl](https://github.com/NHDaly/StagedFunctions.jl) and [Tricks.jl](https://github.com/oxinabox/Tricks.jl/). +[StagedFunctions.jl](https://github.com/NHDaly/StagedFunctions.jl) relaxes the restrictions on normal `@generated` functions so that they are also safe from [#265](https://github.com/JuliaLang/julia/issues/265)-like problems. +[Tricks.jl](https://github.com/oxinabox/Tricks.jl/) uses this feature to make `hasmethod`, etc. resolve at compile-time, and then get updated if and when new methods are defined. +This can allow for defining traits like _"anything that defines a `iterate` method"_. + + +## Base and Standard Libraries + +### Threading +Julia has full support for threading now. +Not just the limited `@threads for` loops, but full Go-style threads. +They are tightly integrated with the existing Async/Task/Coroutine system. +In effect, threading works by unsetting the sticky flag on a Task, so that it is allowed to run on any thread. +This is normally done via the `Threads.@spawn` macro, rather than the `@async` macro. + +Interestingly, the `@threads for` macro still remains, and doesn't actually use much of the new machinery. It still uses the old way which is a bit tighter if the loop durations are almost identical. +But the new threading stuff is fast, on the order of microseconds to send work off to another thread. +Even for uses of `@threads for`, we get some wins from the improvements. +`IO` is now thread-safe; `ReentrantLock` was added and is the kind of standard lock that you expect to exist. +It has notifications on waiting work, etc.; and a big one: `@threads for` can now be nested without things silently being wrong. + +A lot of this actually landed in Julia 1.2, but Julia 1.3 was the release we think of as being for threading, as it gave us `Threads.@spawn`. + +Also, in Julia 1.6, we now have `julia -t auto` to start Julia with 1 thread per (logical) core. +No more having to remember to set the `JULIA_NUM_THREADS` environment variable before starting it. + +### 5-arg `mul!`, aka in-place generalized multiplication and addition +The 5 arg `mul!(C, A, B, α, β)` performs the operation equivalent to: `C .= A*B*α + C*β`, where `α`, `β` are scalars and `A`, `B` and `C` are compatible mixes of scalars, matrices and vectors. +I am still of the opinion that it should have been called `muladd!`, but this is the human-friendly version of `BLAS.gemm!` (i.e. GEneralized Matrix Multiplication) and it's ilk. +It promises to alway compute the in-place mul-add in the most efficient, correct way for any `AbstractArray` subtype. +In contrast, `BLAS.gemm!` computes the same thing, but with a bunch of conditions. +It must be a strided array containing only BLAS scalars, and if one of the inputs is conjugated/transposed you need to input it in non-conjugated/transposed form, and then tell `BLAS.gemm!` via passing in `'C'` or `T` rather than `N`. +5-arg `mul!` takes care of all that for you dispatching to `BLAS.gemm!` or other suitable methods once it has that all sorted. +Further, the existing 3-arg `mul!(C, A, B)` is a special case of it: +`mul!(C, A, B) = mul!(C, A, B, false, true)` (`true` and `false` being 1, and strong 0). +So in general, one can just implement the 5-arg form and be done with it. + +I personally didn't care about 5-arg `mul!` at all for a long time. +It was yet another in-place function in `LinearAlgebra` that I would never use often enough to remember what it did, and thus wouldn't use. +But I realized that it is a crucial function for my own area: automatic differentiation. +`mul(C, A, B, true, true)` is the in-place accumulation rule for the reverse mode equivalent of the product rule. + + +### You can now print and interpolate `nothing` into strings. + +This is one of mine [#32148](https://github.com/JuliaLang/julia/pull/32148), and I find it is such a usability enhancement. +So many small frustrations in Julia 1.0 related to interpolating a variable containing `nothing` into a string. +Often occurring when you are adding a quick `println` to debug something not being the value you expected; or when building a string for some other error message. +Arguably both of those are better done via other means (`@show`, and values stored in fields in the error type); but we don't always do what is best. +Sometimes it is expedient to just interpolate things into strings without worrying about what type they are. + +### `Base.download` now using libcurl +For a very long time the `download` function which retrieves things over HTTP, was implemented with an amazing hack. +It conditionally shelled out to different programs. +On Windows it ran a mildly scary PowerShell script. +On Unixen it first tried to use `curl`, then if that wasn't installed it tried to use `wget`, and then if that wasn't installed it tried to use `fetch`. +It's low-key amazing that this worked as well as it did -- very few complaints. +But as of 1.6, [it now uses libcurl](https://github.com/JuliaLang/julia/pull/37340). +Using libcurl everywhere gives consistency with proxy settings, and protocol support (beyond HTTP) across all platforms. + +It also has a more extensive API via the new [Downloads.jl](https://github.com/JuliaLang/Downloads.jl) standard library. +It can do things like progress logging, and it can retrieve headers. +I have [tried getting headers via conditional different command-line download functions](https://github.com/oxinabox/DataDeps.jl/pull/22) before, it is low-key nightmare fuel; and I ended up swapping out to HTTP.jl for that. +It wouldn't be too surprising if eventually we see libcurl swapped out for code extracted from HTTP.jl for a pure Julia solution. +HTTP.jl works wonderfully to this, but I suspect untangling the client from the server is just a bit annoying right now, particularly as it is still evolving its API. + + +## Definable Error Hints +[`Experimental.register_error_hint`](https://docs.julialang.org/en/v1.7-dev/base/base/#Base.Experimental.register_error_hint) +allows packages to define extra information to be shown along with a particular type of error. +A really cool use of it was proposed on [Discourse](https://discourse.julialang.org/t/enforce-interface-implementation/52872/15?u=oxinabox): +you could add an error hint to `MethodError` that says to check that an interface has been implemented correctly. +This has the advantage of pointing the user in what is most likely the right direction. +But it doesn't have the problem of really confounding them if you are wrong, since the original `MethodError` information is still shown (See my earlier [blog post on the `NotImplementedException` antipattern](https://www.oxinabox.net/2020/04/19/Julia-Antipatterns.html#notimplemented-exceptions)). + +Right now this is not used anywhere in the standard libraries. +At first, I thought that made sense since they can just edit the original `show_error`. +But now I think since it can be used for a subset of a particular type of exception it could well be useful for some of [`Base`'s interfaces](https://docs.julialang.org/en/v1/manual/interfaces/) exactly as described above. +According to [JuliaHub search](https://juliahub.com/ui/CodeSearch?q=register_error_hint&u=all&t=all), right now there is just one package using it. +ColorTypes.jl is [using it to explain some _consensual type-piracy_](https://github.com/JuliaGraphics/ColorTypes.jl/blob/bd31741d162361ebd44ed05ae532266998d9ce9f/src/error_hints.jl), where for some functions on its types you need to load another package. + +This is an experimental features, so it's not covered by the SemVer guarantees of the rest of the language. +Still it's neat, and I don't expect it to go away, +though I also don't expect it to graduate from experimental status until a bunch of people are using it. +Which they will probably do as people start dropping support for older Julia versions. + + +### A bunch of curried functions +Julia seems to have ended up with a convention of providing curried methods of functions if that would be useful as the first argument for `filter`. +For example, `filter(isequal(2), [1,2,3,2,1])` is the same is `filter(x->isequal(x, 2), [1,2,3,2,1])`. +In particular these are boolean comparison-like functions: +with two arguments, where the thing being compared against is the second. +Julia 1.0 had `isequal`, `==` and `in`. +Since then we have added: +`isapprox`, `<`,`<=`,`>`, `>=`, `!=`, `startswith`, `endswith`, and `contains` (I added the last 3 😁). + +Aside: `contains` is argument-flipped `occursin`. +It was a thing in Julia 0.6, but was removed in 1.0 and now has been added back. +We added it back primarily so we could have the curried form, and to match `startswith` and `endswith`. + +### A ton of other new and improved standard library functions + +I am not going to manage to list all of them here. +But I will list some of my standout favorites. + + +[`@time`])(https://github.com/JuliaLang/julia/pull/37678) now reports how much time was spent on compilation. +This is going to help prevent people new to the language from including compilation time in their benchmarks. +It is still better to use [BenchmarkTools.jl's](https://github.com/JuliaCI/BenchmarkTools.jl) `@btime`, since that does multiple samples. +But now that too can report time spent on compilation. +It's also useful for identifying if certain functions are taking ages to compile. +Which I guess is its theoretical main point, but I think preventing people from benchmarking wrong is going to come up way more often. + +The experimental [`Base.@locals`](https://github.com/JuliaLang/julia/issues/29733) +was added, which returns a dictionary of local variables. +That one surprised me; I though being able to access a dict of local variables would get in the way of the optimizer, since it would prevent it from being able to optimize variables that are used for intermediate values away entirely. +But the compiler folks know better than I do. + +[`splitpath`](https://github.com/JuliaLang/julia/issues/28156) is added, it's the opposite of `joinpath`. +Kind of silly we didn't have that, and had been bugging me at least since 0.6. + +On things that had been bugging me, I had wanted [`eachslice` and its special cases: `eachrow` & `eachcol`](https://github.com/JuliaLang/julia/issues/29749) since Julia 0.3 when I first started using it. +These are super handy, for example when you want to iterate through vectors of the rows of a matrix. + +`redirect_stderr` and `redirect_stdout` now work with `devnull`. +So one can run some code while suppressing output easily as follows: +```julia +julia> redirect_stdout(devnull) do + println("You won't see this") + end +``` +This is just handy, doing it without this is seriously annoying. + +`readdir` now accepts a `join=true|false` keyword argument so that it returns paths with the parent dir. +This is good, almost every time I use `readdir` I used it as: +`joinpath.(x, readdir(x))`. +It is slightly cleaner (and faster) to be able to do `readdir(x; join=true)`. +I think for Julia 2.0 we should consider making it the default. +Also added was a `sort` argument, which I don't see the point of so much, since `sort(readdir(x))` seems cleaner than `readdir(x; sort=true)`; and because I rarely rely on processing files in order. + + +`ccall` is now available as a macro `@ccall` which lets you specify +There was a [short juliacon talk about this](https://www.youtube.com/watch?v=wofq1DdXM3s) +So now one can do `@ccall(sqrt(4.0::Cdouble)::Cdouble)` +rather than `ccall(:sqrt, Cdouble, (Cdouble,), 4.0,)` +This is what `ccall` always should have been. +In Julia 1.7, [`invoke` is getting the same treatment.](https://github.com/JuliaLang/julia/pull/38438). + + +### More "Why didn't it always work that way" than I can count +Since 1.0's release, there have been so many small improvements to functions that I didn't even know happened, because I assumed they always worked that way. +Things like `startswith` supporting regex, [`parse`](https://github.com/JuliaLang/julia/pull/36199) working on `UUID`s, +`accumulate`, `cumsum`, and `cumprod` supporting arbitrary iterators ([#32656](https://github.com/JuliaLang/julia/pull/34656)). +Julia 1.6 is one hell of a more polished language. diff --git a/posts_assets/Julia-1.0-1.6-changes/julia-1.0-code-llvm.png b/posts_assets/Julia-1.0-1.6-changes/julia-1.0-code-llvm.png new file mode 100644 index 00000000..21f2d1c8 Binary files /dev/null and b/posts_assets/Julia-1.0-1.6-changes/julia-1.0-code-llvm.png differ diff --git a/posts_assets/Julia-1.0-1.6-changes/julia-1.0-conflict.png b/posts_assets/Julia-1.0-1.6-changes/julia-1.0-conflict.png new file mode 100644 index 00000000..0b168585 Binary files /dev/null and b/posts_assets/Julia-1.0-1.6-changes/julia-1.0-conflict.png differ diff --git a/posts_assets/Julia-1.0-1.6-changes/julia-1.0-stacktrace.png b/posts_assets/Julia-1.0-1.6-changes/julia-1.0-stacktrace.png new file mode 100644 index 00000000..02212be9 Binary files /dev/null and b/posts_assets/Julia-1.0-1.6-changes/julia-1.0-stacktrace.png differ diff --git a/posts_assets/Julia-1.0-1.6-changes/julia-1.6-code-llvm.png b/posts_assets/Julia-1.0-1.6-changes/julia-1.6-code-llvm.png new file mode 100644 index 00000000..17346d51 Binary files /dev/null and b/posts_assets/Julia-1.0-1.6-changes/julia-1.6-code-llvm.png differ diff --git a/posts_assets/Julia-1.0-1.6-changes/julia-1.6-conflict.png b/posts_assets/Julia-1.0-1.6-changes/julia-1.6-conflict.png new file mode 100644 index 00000000..aab69c37 Binary files /dev/null and b/posts_assets/Julia-1.0-1.6-changes/julia-1.6-conflict.png differ diff --git a/posts_assets/Julia-1.0-1.6-changes/julia-1.6-stacktrace.png b/posts_assets/Julia-1.0-1.6-changes/julia-1.6-stacktrace.png new file mode 100644 index 00000000..5e869912 Binary files /dev/null and b/posts_assets/Julia-1.0-1.6-changes/julia-1.6-stacktrace.png differ diff --git a/stylesheets/styles.css b/stylesheets/styles.css index 5ba7d2bb..e053451e 100644 --- a/stylesheets/styles.css +++ b/stylesheets/styles.css @@ -277,10 +277,6 @@ em, .em { padding: 0 20px; } -section img { - max-width: 100%; -} - blockquote { border-left: 3px solid #ffcc00; margin: 0; @@ -320,8 +316,10 @@ svg { padding: 16px; } -img { /*Some images are not rendering background for some reason */ +img { + /*Some images are not rendering background for some reason */ background-color: white; + max-width: 100%; }