Auto xz

[DeeDeeG]

Pull Request

Problem

Somewhat smaller archives are available, compressed with xz. However, n currently only uses gzip archives. (Edit to add: The N_USE_XZ environment variable is already implemented/respected in n, but this obscure feature is likely not used very heavily. Having it on by default would make more of an impact.)

See: #606

Solution

Determine if using xz is possible for the current user/desired version of node. If using xz seems possible, fetch/decompress an archive compressed with xz, instead of an archive compressed with gzip.

• We deduce support for decompressing xz with tar on the user's system, based on research of likely Linux/macOS configurations.
• We know, at least as it pertains to the official Node.js project, that xz compressed tarballs are available for Node 4.x and greater.

ChangeLog

Added

• Downloads now default to using tarballs compressed by xz, rather than tarballs compressed by gzip. Includes system/feature detection, which is used to guess if xz can be decompressed by the current user. Enabled only for Node 4.x and greater, as this is the first major version for which xz tarballs were provided. ([Consider porting the auto-xz feature from nvh #606] [Auto xz #607])
  • Export any value of N_USE_XZ (other than 0), including setting it to be null or empty, to enable xz-compressed tarballs, overriding whatever the default behavior may be.
  • (Unset N_USE_XZ entirely in order to return n to its default behavior.)

Changed

• Exporting the environment variable N_USE_XZ as 0 will now have the effect of disabling the use of tarballs compressed with xz, rather than enabling it. (Previously, exporting N_USE_XZ as any non-zero-length value, including the number 0, would enable the use of xz-compressed tarballs.)

[DeeDeeG]

The CHANGELOG lines I suggested are already committed on my branch -- it started as an attempt to copy over the documentation changes from nvh's develop branch. But I ended up editing the README.md and CHANGELOG.md entries rather heavily, as the feature there added xz entirely (this PR only makes its usage automatic vs the previous state of things, which was fully manual and done exclusively via an environment variable... Also the documentation is structured slightly differently over at nvh...

I can easily remove the commit that includes these documentation changes, if you'd like to draft them up yourself.

I accidentally left the command-line flags (--use-xz, --no-use-xz) undocumented here. Best to document them somewhere I think...

[DeeDeeG]

I made a branch where I simply copy-paste the xz section from the file docs/environment-variables.md over at nvh. See: https://github.com/DeeDeeG/n/tree/updated-readme-auto-xz#optional-environment-variables

It's more concisely worded, and has the benefit of explaining the command-line flags.

If you'd like that pushed to this branch, I can easily do so.

---

Also: I updated the wording in my suggested CHANGELOG.md lines, to be neater/shorter while getting the same point across.

---

Lastly: I rebased on master branch of this repo, since I noticed master is one commit ahead of develop branch, and I had been missing that commit in my PR's branch.

[DeeDeeG]

(By the way, I tested this PR, and it worked as expected. Here is a branch I put together with echoes that make it more explicit what is going on with the command-line variable N_USE_XZ, and the normalization the script does for this variable, to either true or false: https://github.com/DeeDeeG/n/tree/test-auto-xz)

For ease of review, and demonstrating the soundness of the port.

[shadowspawn]

Plenty here thanks. I was happy with the pattern we worked out for nvh, so other than tweaking wording I don't anticipate making changes.

I'm heading for a big release on another product, so might be a couple of weeks before I work on this.

I made a branch where I simply copy-paste the xz section from the file docs/environment-variables.md over at nvh. See: https://github.com/DeeDeeG/n/tree/updated-readme-auto-xz#optional-environment-variables

It's more concisely worded, and has the benefit of explaining the command-line flags.

If you'd like that pushed to this branch, I can easily do so.

I do like the copy-paste wording better.

[DeeDeeG]

Considering going with the original CHANGELOG.md wording from nvh as well.

Added

• Downloads now default to using tarballs compressed by xz over gzip, if xz support detected. (#ISSUE_NUM #PR_NUM)

Somewhat torn on whether the new situation (N_USE_XZ=0 means "don't use xz") should be mentioned in the CHANGELOG.md.

It's an unexpected behavior change for a subset of a subset of people, so much so that it may be zero actual users: People who have enabled xz in n by setting N_USE_XZ to 0. (Also, I'd note thatn should still work with gzip for these users.)

[DeeDeeG]

I also had some thoughts about the environment variable, and there being a "default on" but also a user-indicated "explicit on."

• The environment variable is more or less obsoleted by the default behavior (most users presumably want smaller network transfers?) and command-line flags are there if the user wants to configure it.
• Relatedly: Having a default "on," and the user being able to specify an explicit "on," is arguably redundant.

Between these two things, much of the code for this feature could be simplified or removed. So I thought I'd bring that up before this is merged, since I feel once the code were to be merged it would probably stick around. (I feel some guilt at influencing n to be larger, since it is a very very small download to start with. Easy to make it grow several % in size with one PR. But most things in coding are trade-offs of some kind or other.)

[shadowspawn]

Somewhat torn on whether the new situation (N_USE_XZ=0 means "don't use xz") should be mentioned in the CHANGELOG.md.

No need to call out the changed behaviour as such, likely affects no one, and non-breaking in the sense it will continue to work with gzip. (Both as you noted.)

I also had some thoughts about the environment variable, and there being a "default on" but also a user-indicated "explicit on."

It is a bit over-engineered, but I'm comfortable that we give people control in case of issues initially. Especially since we can only make an educated guess whether xz supported.

[DeeDeeG]

I updated this PR to use the original CHANGELOG.md entry from nvh.

---

Everything here is now a copy-pasted version of the feature and docs as they appear in nvh.

Here's a list of the changes that were made to the script itself so as to follow the style of n, or to make the feature work:

• Of course, the variable $NVH_USE_XZ was renamed to $N_USE_XZ
• $version in n is the equivalent of $resolved_version in nvh...
  • As such, in install(), when calling to update_xz_settings_for_version(), pass $version, rather than $resolved_version.
• For the in-script command-line flag documentation (printed by e.g. n --help), nvh has the flags alphabetized, but they appear sorted by prominence/frequency of use in n.
  • To preserve existing style in n (i.e. to avoid alphabetizing all the flags just for the sake of an exact copy-paste from nvh), I put the --use-xz and --no-use-xz flags underneath --insecure. (Just to "pick somewhere" to put them).
• Similarly, for the "process options" loop, I put --use-xz and --no-use-xz directly below --insecure
  • I put the flags in this spot just to "pick something," since alphabetical order wasn't an option without reordering the whole case/esac statement. This is theoretically a possible source of behavior difference from nvh; Options will be processed internally in a different order. But I think the practical outcome, the user-apparent behavior of the loop and the script, remains equivalent across nvh and n.
  • (Also: "Not alphabetizing the options in n" predates this PR.)
• Code style: To preserve existing code style in n, I used the following line rather than how it appears in nvh, compare bullet points below:
  • [ "$N_USE_XZ" = "true" ] && ext="xz" (snippet from this PR for n)
  • if [[ "${NVH_USE_XZ}" = "true" ]]; then local ext="xz"; fi (equivalent snippet from nvh)
• Code style: To preserve existing code style in n, the $tarflag/$tarflags are set differently (see below):

if test "$N_USE_XZ" = "true"; then
  local tarflag="-Jx"
else
  local tarflag="-zx"
fi

(The above code snippet is from this PR for n)

local tarflags="-zx"
if [[ "${NVH_USE_XZ}" = "true" ]]; then local tarflags="-Jx"; fi

(The above code snippet is from nvh)

[DeeDeeG]

I'd suggest rewording this:

# tar on linux is likely to support xv if it is available separately

available separately
-->
available on the PATH

Also, a typo:

xv --> xz