An invalid BuildTools.jar gets downloaded - Error 1020

[Electrenator]

A 16byte file gets created when downloading the latest BuildTools.jar file via the GUI. In this file is the string error code: 1020 saved instead of the BuildTools jar file. This probably something to do with Cloudflare.

Running on commit: 4470b92
In the docker container: https://hub.docker.com/r/binhex/arch-mineos-node
Installation age: Fresh, just installed

Note: Don't get this issue with curl in the same container via bash. Doesn't seem to be container-based as far as I can test and it isn't network-based because I can download the file manually without that error. (Used following curl command: curl --output BuildTools.jar https://hub.spigotmc.org/jenkins/job/BuildTools/118/artifact/target/BuildTools.jar)

Edit: A possible fix is to replace BuildTools.jar by hand. Furthermore, this issue may be related to issue #329

[tomdaley92]

Hi, I am running MineOS on top of Ubuntu (not inside a Docker container) and same issue.
Running on commit: 5670b46

Cloudflare DNS does in fact seem to be returning a 403 HTTP response along with the actual file contents being the string error code: 1020 like you said. I can wget the file manually from the same box no problem. My hunch is that this has something to do with insufficient headers being sent with the request being made.. possibly needing to spoof a different agent or enabling CORS or something like that.. I'll take a closer peak within the next couple days.

Here's the relevant output from /var/log/mineos.log after clicking on Download latest BuildTools in the MineOS Web UI

{
  "profile": {
    "id": "BuildTools-latest",
    "time": 1601873010643,
    "releaseTime": 1601873010643,
    "type": "release",
    "group": "spigot",
    "webui_desc": "Latest BuildTools.jar for building Spigot/Craftbukkit",
    "weight": 0,
    "filename": "BuildTools.jar",
    "downloaded": true,
    "version": 0,
    "release_version": "",
    "url": "https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar",
    "$$hashKey": "object:2331"
  },
  "command": "download",
  "level": "info",
  "message": "[WEBUI] Received emit command from 192.168.50.139:tom",
  "timestamp": "2020-10-05T04:44:48.276Z"
}

{
  "level": "error",
  "message": "[WEBUI] Server was unable to download file: https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar",
  "timestamp": "2020-10-05T04:44:48.326Z"
}

{
  "date": "Mon, 05 Oct 2020 04:44:48 GMT",
  "content-type": "text/plain; charset=UTF-8",
  "content-length": "16",
  "connection": "close",
  "x-frame-options": "SAMEORIGIN",
  "cache-control": "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0",
  "expires": "Thu, 01 Jan 1970 00:00:01 GMT",
  "set-cookie": [
    "__cfduid=decb1ede68d548f0207cba25c5d6b3bd61601873088; expires=Wed, 04-Nov-20 04:44:48 GMT; path=/; domain=.spigotmc.org; HttpOnly; SameSite=Lax; Secure"
  ],
  "cf-request-id": "0598aca73b0000f5b9c029c200000001",
  "expect-ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
  "server": "cloudflare",
  "cf-ray": "5dd47d51ff56f5b9-SEA",
  "alt-svc": "h3-27=\":443\"; ma=86400, h3-28=\":443\"; ma=86400, h3-29=\":443\"; ma=86400",
  "level": "error",
  "message": "[WEBUI] Remote server returned status 403 with headers:",
  "timestamp": "2020-10-05T04:44:48.326Z"
}

[Electrenator]

Looks like Cloudflare indeed doesn't like requests without User-Agent. Probably need to add one via a header, so looked into that.

Posible solution

Found the code responsible for the request here at line 376 in server.js and from what I see from my tests with request() it doesn't seem to send a User-Agent. So a possible fix for that line could be the following code line based on the docs of request.

progress(request({url: args.profile.url, headers: {'User-Agent': 'MineOS-node'}}), { throttle: 250, delay: 100 })

Don't really know how to work with nodejs dev environments, therefore can't really test and thus commit this solution. Did however get the wanted response via the node shell, so should work. Used following for node shell test;

var data = {
    "url": "https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar"
};
var request = require('request');
request({ url: data.url, headers: { 'User-Agent': 'MineOS-node' } }, function (error, response, body) {
    console.error('Error:', error);
    console.log('Status code:', response && response.statusCode);
    console.log('Body length:', body.length)
})

Furthermore

It looks like the package in use for downloading (request) is deprecated as of Feb 11th 2020. More info here; request/request#3142
Maybe handy to look into using an alternative like node-fetch.

[hexparrot]

Fixed, thanks for your help with making #414 happen!