Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eng - add checksum validation support to nodejs and extensions #184877

Merged
merged 29 commits into from
Jun 14, 2023
Merged

eng - add checksum validation support to nodejs and extensions #184877

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
aebbee5
eng - add `checksum` validation support to `remote`
bpasero Jun 12, 2023
e7ba203
eng - wire in more `checksum` support
bpasero Jun 12, 2023
c3d2080
eng - renames for remote fetching
bpasero Jun 12, 2023
6c76b57
eng - renames for remote fetching
bpasero Jun 12, 2023
f2d5604
eng - disable verbose
bpasero Jun 12, 2023
e3ac4b1
eng - always fetch verbose in CI
bpasero Jun 12, 2023
d0a7216
eng - :lipstick:
bpasero Jun 12, 2023
1ef4450
eng - add checksums for node
bpasero Jun 12, 2023
4d5b746
eng - report checksum matches
bpasero Jun 12, 2023
acc153f
eng - fix build
bpasero Jun 12, 2023
a6620d4
eng - warn when not being able to check sum
bpasero Jun 12, 2023
2e8d72a
eng - support checksums for built in extensions
bpasero Jun 12, 2023
d362a34
eng - clear todo
bpasero Jun 12, 2023
8229c05
eng - add nodejs metadata to product.json
bpasero Jun 13, 2023
551e33d
:up: version
bpasero Jun 13, 2023
ec7d57f
:up: distro
bpasero Jun 13, 2023
8a6c193
update distro
bpasero Jun 13, 2023
67fb55d
eng - switch to checksum file
bpasero Jun 13, 2023
08be002
cleanup alpine
bpasero Jun 13, 2023
d762c99
fix alpine
bpasero Jun 13, 2023
13b5143
fix bug
bpasero Jun 13, 2023
ed75d29
Merge branch 'main' into ben/remote-checksums
bpasero Jun 13, 2023
e087908
eng - fetch all from remote
bpasero Jun 14, 2023
867f1a1
eng - :lipstick:
bpasero Jun 14, 2023
19c6c8b
eng - print checksums
bpasero Jun 14, 2023
51a8bf4
eng - fix missing import
bpasero Jun 14, 2023
2f727dd
:up: distro
bpasero Jun 14, 2023
ee1b6ae
Merge branch 'main' into ben/remote-checksums
bpasero Jun 14, 2023
96b8182
undo version change
bpasero Jun 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions build/checksums/nodejs.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
dfb37570ef34ac04f34c26d0ec558df60a9665df5961c01c1657c0ca495f2f01 node-v16.17.1-aix-ppc64.tar.gz
bpasero marked this conversation as resolved.
Show resolved Hide resolved
f9f02f7872e2e8ee54320fce13deb9d56904f32bb0615b6e21aa3371d8899150 node-v16.17.1-darwin-arm64.tar.gz
09a45f60bfb9dfbea4f69044dc733ef983945acd92ca89ccccac267f3d71bd44 node-v16.17.1-darwin-arm64.tar.xz
3db26761ad8493b894d42260d7e65094b7af9bc473588739e61bc1c32d6ff955 node-v16.17.1-darwin-x64.tar.gz
8e7089956fa01cf7d0045945c0863d282dc6818fb0476237c1396497e29a4254 node-v16.17.1-darwin-x64.tar.xz
35ccb95caf02cda3bd680da4350a8ae5d666a7a9eae3afe5c2a1b3ef29aef108 node-v16.17.1-headers.tar.gz
554c8d1b4b16e0f4c073b9df7c49c893716a3a533f25ac646f23619f5ccee7df node-v16.17.1-headers.tar.xz
adc7032888d4e672a4aac886baede8c04fccdd1a2e7ab4bcf325e3f336f44a3d node-v16.17.1-linux-arm64.tar.gz
3dfb8fd8f6b97df69cdc56524abc906c50ef1d0bf091188616802e6c7c731389 node-v16.17.1-linux-arm64.tar.xz
aeab05e35f1d2824ecfb88ca321f1408b44d292b2775f2890972c828e00216d0 node-v16.17.1-linux-armv7l.tar.gz
a035ceefb5e16f5fce98c8ddfdf721b96eec20542c72fb8781bcbb6ef20c5550 node-v16.17.1-linux-armv7l.tar.xz
1f48de7bed99e973c4c50f1b7fc99fc9af5144d093fd6d2b50a1e43b5818bf05 node-v16.17.1-linux-ppc64le.tar.gz
70305934661f89fca64053b85317a75f233d5e3fdb2caa6546a19262a519cf20 node-v16.17.1-linux-ppc64le.tar.xz
029dad48018bda07b481213816549b632059fc673c30fdc7a353e04619128344 node-v16.17.1-linux-s390x.tar.gz
1a47f604944c6aff37cb7483503155671cdb34bda9bfb8962007bc440fa04d77 node-v16.17.1-linux-s390x.tar.xz
da5658693243b3ecf6a4cba6751a71df1eb9e9703ca93b42a9404aed85f58ad0 node-v16.17.1-linux-x64.tar.gz
06ba2eb34aa385967f5f58c87a44753f83212f6cccea892b33f80a2e7fda8384 node-v16.17.1-linux-x64.tar.xz
12d10476ea7483298364c810c037b9316d1a73dc8c81cfeff7d794aecadde498 node-v16.17.1.pkg
e423985f6019b2026f9a191adb56a96ae83ecd56cdf839cf94aa980168b7a90f node-v16.17.1.tar.gz
6721feb4152d56d2c6b358ce397abd5a7f1daf09ee2e25c5021b9b4d3f86a330 node-v16.17.1.tar.xz
9777e8c4b2864c5b54a0e4e9400f14887db68560a09b94b4113b560a64d1e680 node-v16.17.1-win-x64.7z
ed290151efb417262b9808a70738d4ab79e9d53653a6a9f4b8dd97912e279dce node-v16.17.1-win-x64.zip
0f8101648d5c9e49e89fee541da9e574f899716c32b7c51a732b1766b9fc4526 node-v16.17.1-win-x86.7z
189b5e8b23226403e7b07a46614de19b444d369e694901e3668e2f549799cbcd node-v16.17.1-win-x86.zip
1bdff65fb7642425c0d6826084d63c4be43520316f0ea0b46e6a51999a0ed7fc node-v16.17.1-x64.msi
b737eb23a2c67c253b9364b5284123faf5220d567615bebd4ec4b81070e4d177 node-v16.17.1-x86.msi
f518a70dcab7c3fac5b2e1ef100b4f628edfb160f4fafa9a94ef222da8a6e9ab win-x64/node.exe
2f459a64647db493da63c790ce368ad54f59f086d9f22f59c5018680420197b3 win-x64/node.lib
23215ce7d1e9de9777c3407239e7cf18d29d60f757b772219421ab361ac67c74 win-x64/node_pdb.7z
8e32ec12028fd3e3147435be79a858ed9c870aaafa1fcb291362307ef3c47547 win-x64/node_pdb.zip
2393aff88be19dbe0205cbde4ff0c1d89911b15de5c99c80f6e5e29604eecd12 win-x86/node.exe
5018c3d42f3fbacbd06cb943b3f2696c8e67ca9bdf6864d0e263d6d6911dffd2 win-x86/node.lib
05a4db56444a60ee70b0d2642d7f2d82a33339894d2d73bd07b1a41d6c869e04 win-x86/node_pdb.7z
8f86eacb7f13a1bf6738cb0819d7854a2abca40fc2e9e1f91421e44ba52cad7e win-x86/node_pdb.zip
94 changes: 68 additions & 26 deletions build/gulpfile.reh.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ const rename = require('gulp-rename');
const replace = require('gulp-replace');
const filter = require('gulp-filter');
const { getProductionDependencies } = require('./lib/dependencies');
const { assetFromGithub } = require('./lib/github');
const vfs = require('vinyl-fs');
const packageJson = require('../package.json');
const flatmap = require('gulp-flatmap');
Expand All @@ -43,7 +42,6 @@ const BUILD_TARGETS = [
{ platform: 'win32', arch: 'x64' },
{ platform: 'darwin', arch: 'x64' },
{ platform: 'darwin', arch: 'arm64' },
{ platform: 'linux', arch: 'ia32' },
{ platform: 'linux', arch: 'x64' },
{ platform: 'linux', arch: 'armhf' },
{ platform: 'linux', arch: 'arm64' },
Expand Down Expand Up @@ -131,6 +129,33 @@ function getNodeVersion() {
return target;
}

function getNodeChecksum(nodeVersion, platform, arch) {
let expectedName;
switch (platform) {
case 'win32':
expectedName = `win-${arch}/node.exe`;
break;

case 'darwin':
case 'linux':
expectedName = `node-v${nodeVersion}-${platform}-${arch}.tar.gz`;
break;

case 'alpine':
expectedName = `${platform}-${arch}/node`;
break;
}

const nodeJsChecksums = fs.readFileSync(path.join(REPO_ROOT, 'build', 'checksums', 'nodejs.txt'), 'utf8');
for (const line of nodeJsChecksums.split('\n')) {
const [checksum, name] = line.split(/\s+/);
if (name === expectedName) {
return checksum;
}
}
return undefined;
}

const nodeVersion = getNodeVersion();

BUILD_TARGETS.forEach(({ platform, arch }) => {
Expand All @@ -155,40 +180,57 @@ if (defaultNodeTask) {
}

function nodejs(platform, arch) {
const { remote } = require('./lib/gulpRemoteSource');
const { fetchUrls, fetchGithub } = require('./lib/fetch');
const untar = require('gulp-untar');
const crypto = require('crypto');

if (arch === 'ia32') {
arch = 'x86';
} else if (arch === 'armhf') {
arch = 'armv7l';
} else if (arch === 'alpine') {
platform = 'alpine';
arch = 'x64';
}

if (platform === 'win32') {
if (product.nodejsRepository) {
log(`Downloading node.js ${nodeVersion} ${platform} ${arch} from ${product.nodejsRepository}...`);
return assetFromGithub(product.nodejsRepository, nodeVersion, name => name === `win-${arch}-node-patched.exe`)
.pipe(rename('node.exe'));
}
log(`Downloading node.js ${nodeVersion} ${platform} ${arch} from https://nodejs.org`);
return remote(`/dist/v${nodeVersion}/win-${arch}/node.exe`, { base: 'https://nodejs.org', verbose: true })
.pipe(rename('node.exe'));
}
log(`Downloading node.js ${nodeVersion} ${platform} ${arch} from ${product.nodejs.repository}...`);

if (arch === 'alpine' || platform === 'alpine') {
const imageName = arch === 'arm64' ? 'arm64v8/node' : 'node';
log(`Downloading node.js ${nodeVersion} ${platform} ${arch} from docker image ${imageName}`);
const contents = cp.execSync(`docker run --rm ${imageName}:${nodeVersion}-alpine /bin/sh -c 'cat \`which node\`'`, { maxBuffer: 100 * 1024 * 1024, encoding: 'buffer' });
return es.readArray([new File({ path: 'node', contents, stat: { mode: parseInt('755', 8) } })]);
const checksumSha256 = getNodeChecksum(nodeVersion, platform, arch);

if (checksumSha256) {
log(`Using SHA256 checksum for checking integrity: ${checksumSha256}`);
} else {
log.warn(`Unable to verify integrity of downloaded node.js binary because no SHA256 checksum was found!`);
}

if (arch === 'armhf') {
arch = 'armv7l';
switch (platform) {
case 'win32':
return (product.nodejs.repository !== 'https://nodejs.org' ?
fetchGithub(product.nodejs.repository, { version: product.nodejs.version, name: `win-${arch}-node.exe`, checksumSha256 }) :
fetchUrls(`/dist/v${nodeVersion}/win-${arch}/node.exe`, { base: 'https://nodejs.org', checksumSha256 }))
.pipe(rename('node.exe'));
case 'darwin':
case 'linux':
return (product.nodejs.repository !== 'https://nodejs.org' ?
fetchGithub(product.nodejs.repository, { version: product.nodejs.version, name: `node-v${nodeVersion}-${platform}-${arch}.tar.gz`, checksumSha256 }) :
fetchUrls(`/dist/v${nodeVersion}/node-v${nodeVersion}-${platform}-${arch}.tar.gz`, { base: 'https://nodejs.org', checksumSha256 })
).pipe(flatmap(stream => stream.pipe(gunzip()).pipe(untar())))
.pipe(filter('**/node'))
.pipe(util.setExecutableBit('**'))
.pipe(rename('node'));
case 'alpine': {
const imageName = arch === 'arm64' ? 'arm64v8/node' : 'node';
log(`Downloading node.js ${nodeVersion} ${platform} ${arch} from docker image ${imageName}`);
const contents = cp.execSync(`docker run --rm ${imageName}:${nodeVersion}-alpine /bin/sh -c 'cat \`which node\`'`, { maxBuffer: 100 * 1024 * 1024, encoding: 'buffer' });
if (checksumSha256) {
const actualSHA256Checksum = crypto.createHash('sha256').update(contents).digest('hex');
if (actualSHA256Checksum !== checksumSha256) {
throw new Error(`Checksum mismatch for node.js from docker image (expected ${options.checksumSha256}, actual ${actualSHA256Checksum}))`);
}
}
return es.readArray([new File({ path: 'node', contents, stat: { mode: parseInt('755', 8) } })]);
}
}
log(`Downloading node.js ${nodeVersion} ${platform} ${arch} from https://nodejs.org`);
return remote(`/dist/v${nodeVersion}/node-v${nodeVersion}-${platform}-${arch}.tar.gz`, { base: 'https://nodejs.org', verbose: true })
.pipe(flatmap(stream => stream.pipe(gunzip()).pipe(untar())))
.pipe(filter('**/node'))
.pipe(util.setExecutableBit('**'))
.pipe(rename('node'));
}

function packageTask(type, platform, arch, sourceFolderName, destinationFolderName) {
Expand Down
Loading