Test SHA1 hash of package to verify integrity of installers

[michaellwest]

While running into an issue related to the download of 7-zip, I realised that the download was only to verify that the zip file is not corrupt.

Would it be good to instead have SHA1 hashes for each of the Sitecore downloads, then test against that value instead? This would serve both the purpose of validating against corruption as well as ensure the expected packages are downloaded.

[pbering]

Yeah I would like to get rid of the 7zip usage for verifying the zips (not always working anyways). We have to maintain the SHA1 of each downloads for our self since Sitecore does not supply any checksums. I think the best why to implement this is to add it to "sitecore-packages.json" like:

  "Sitecore 9.3.0 rev. 003498 (WDP XM1 packages).zip": {
    "url": "https://dev.sitecore.net/~/media/D1F694E4238C438B8464FC1ABFF06C87.ashx",
    "checksum": "7F85E6636B1FA5926259A21FADEC1CF789D306C4"
  },

then during package restore we compare the value with Get-FileHash -Path "S:\Sitecore 9.2.0 rev. 002893 (WDP XM1 packages).zip" -Algorithm SHA1 which works consistently between PowerShell 5 and 6. We should do the compare on every run to make sure the downloaded packages is still valid.

What do you think about this?

[michaellwest]

That's exactly how I imagined it to work. Amazing!

[pbering]

cool! ... did a quick prototype and discovered an issue for users using SMB/UNC for storing the packages, it takes ~1 minute per file (my packages is on Azure File Sharew). I assume that Get-FileHash needs to "download" all bytes into memory to do the calculation :( ... I need to think a bit more...

[michaellwest]

I understand what you mean. For local development it's not an issue. Perhaps it can verify after download and before use in the installation?

[pbering]

the problem with verifying before usage is that what should then happen if it fails? Should it then delete the file in context folder and in the install source folder? or not, but instruct the user to manually do it?

[michaellwest]

Perhaps while running in interactive mode ask to redownload, ignore, or cancel.

[pbering]

There is no such thing today as interactive mode :) ... I think we should download as today and then during build where each file is copied into build context, it will do the verification and if it fails delete the file (only in the build context). Thenfail the build with a message that the file should be manually deleted from the installation source (so it can be redownloaded at next run).

[pbering]

OK I tried to do the validating just before build of each image and it works better, but now the problem is that it has to do it every time and it takes about 1 minute per "asset" image. I will try to defer the check and do it inside the containers instead so that the RUN statements can be cached (which makes since since if should only verify if any source files has been changed)