Checksum verification of aws/aws-cli fails because the expected checksum of aws/aws-cli is wrong

[nikolay]

$ aws version
INFO[0000] download and unarchive the package            aqua_version=2.17.1 env=darwin/arm64 exe_name=aws package_name=aws/aws-cli package_version=2.13.35 program=aqua registry=standard
FATA[0001] aqua failed                                   actual_checksum=36D16A6156150B3940CD7E750FBD7B9C529EE93C8E0EB594283F19A197DE5C42 aqua_version=2.17.1 env=darwin/arm64 error="install the package: checksum is invalid" exe_name=aws expected_checksum=C42812A6F061D931BB21DC12FE9399E8EDD680DA22704CAA2F4836C538C06621 package_name=aws/aws-cli package_version=2.13.35 program=aqua

[suzuki-shunsuke]

Thank you for your report.
I could produce the issue.
I'll look into.

---
# aqua - Declarative CLI Version Manager
# https://aquaproj.github.io/
checksum:
  enabled: true
  require_checksum: true
registries:
- type: standard
  ref: v4.86.0 # renovate: depName=aquaproj/aqua-registry
packages:
- name: aws/[email protected]

{
  "checksums": [
    {
      "id": "http/awscli.amazonaws.com/AWSCLIV2-2.13.35.pkg",
      "checksum": "520E1CB49004ECED7DB1CFE70E6FA73EFC6EDDF1CDB38AF535D126F1DB6574C8",
      "algorithm": "sha256"
    },
    {
      "id": "http/awscli.amazonaws.com/awscli-exe-linux-aarch64-2.13.35.zip",
      "checksum": "4879D5ABD43F2D13C4D7229D889159990081278CFF66748161B29910C8AFC284",
      "algorithm": "sha256"
    },
    {
      "id": "http/awscli.amazonaws.com/awscli-exe-linux-x86_64-2.13.35.zip",
      "checksum": "EB9086823C5850E73F2B5BB1956FEB0CB526E2A7C536419B810E578021E5CDCD",
      "algorithm": "sha256"
    },
    {
      "id": "registries/github_content/github.com/aquaproj/aqua-registry/v4.86.0/registry.yaml",
      "checksum": "9EB5B05029E7821116C7C50F2998261ABDCDFDDE7857C0EBF5FF084497CAD7D4",
      "algorithm": "sha256"
    }
  ]
}

$ aqua i
INFO[0000] download and unarchive the package            aqua_version=2.17.1 env=darwin/amd64 package_name=aws/aws-cli package_version=2.13.35 program=aqua registry=standard
ERRO[0001] install the package                           actual_checksum=36D16A6156150B3940CD7E750FBD7B9C529EE93C8E0EB594283F19A197DE5C42 aqua_version=2.17.1 env=darwin/amd64 error="checksum is invalid" expected_checksum=520E1CB49004ECED7DB1CFE70E6FA73EFC6EDDF1CDB38AF535D126F1DB6574C8 package_name=aws/aws-cli package_version=2.13.35 program=aqua registry=standard
FATA[0001] aqua failed                                   aqua_version=2.17.1 env=darwin/amd64 error="it failed to install some packages" program=aqua

http://awscli.amazonaws.com/AWSCLIV2-2.13.35.pkg

$ sha256sum AWSCLIV2-2.13.35.pkg 
36d16a6156150b3940cd7e750fbd7b9c529ee93c8e0eb594283f19a197de5c42  AWSCLIV2-2.13.35.pkg

expected_checksum is wrong.

$ aqua upc
INFO[0000] updating a package checksum                   aqua_version=2.17.1 env=darwin/amd64 package_name=aws/aws-cli package_registry=standard package_version=2.13.35 program=aqua
INFO[0000] downloading an asset to calculate the checksum  aqua_version=2.17.1 asset_name=awscli-exe-linux-x86_64-2.13.35.zip checksum_env=linux/amd64 env=darwin/amd64 package_name=aws/aws-cli package_registry=standard package_version=2.13.35 program=aqua
INFO[0003] downloading an asset to calculate the checksum  aqua_version=2.17.1 asset_name=awscli-exe-linux-aarch64-2.13.35.zip checksum_env=linux/arm64 env=darwin/amd64 package_name=aws/aws-cli package_registry=standard package_version=2.13.35 program=aqua
INFO[0006] downloading an asset to calculate the checksum  aqua_version=2.17.1 asset_name=AWSCLIV2-2.13.35.pkg checksum_env=darwin/amd64 env=darwin/amd64 package_name=aws/aws-cli package_registry=standard package_version=2.13.35 program=aqua
/Users/shunsuke-suzuki/Documents/test/aqua/17363/aqua-checksums.json

expected_checksum

aqua/pkg/controller/updatechecksum/update.go

Lines 325 to 327 in 428d406

	algorithm := "sha256"
	fields["algorithm"] = algorithm
	chk, err := checksum.CalculateReader(file, algorithm)

actual_checksum

aqua/pkg/installpackage/checksum.go

Lines 147 to 163 in 428d406

	algorithm := "sha256"
	if chksum != nil {
	algorithm = chksum.Algorithm
	}
	calculatedSum, err := is.checksumCalculator.Calculate(is.fs, tempFilePath, algorithm)
	if err != nil {
	return fmt.Errorf("calculate a checksum of downloaded file: %w", logerr.WithFields(err, logrus.Fields{
	"temp_file": tempFilePath,
	}))
	}
	if chksum != nil && !strings.EqualFold(calculatedSum, chksum.Checksum) {
	return logerr.WithFields(errInvalidChecksum, logrus.Fields{ //nolint:wrapcheck
	"actual_checksum": strings.ToUpper(calculatedSum),
	"expected_checksum": strings.ToUpper(chksum.Checksum),
	})
	}

Both of them use the same function internally.

aqua/pkg/checksum/checksum.go

Lines 23 to 47 in 428d406

	func (*Calculator) Calculate(fs afero.Fs, filename, algorithm string) (string, error) {
	f, err := fs.Open(filename)
	if err != nil {
	return "", fmt.Errorf("open a file to calculate the checksum: %w", err)
	}
	defer f.Close()
	return CalculateReader(f, algorithm)
	}
	func CalculateReader(file io.Reader, algorithm string) (string, error) {
	switch algorithm {
	case "md5":
	return checksum.MD5sumReader(file) //nolint:wrapcheck
	case "sha256":
	return checksum.SHA256sumReader(file) //nolint:wrapcheck
	case "sha512":
	return calculateSHA512Reader(file)
	case "sha1":
	return calculateSHA1Reader(file)
	case "":
	return "", errors.New("algorithm is required")
	default:
	return "", errors.New("unsupported algorithm")
	}
	}

The difference of them is that expected_checksum uses the HTTP response body, while actual_checksum uses the temporal file.

[suzuki-shunsuke]

I could reproduce the issue with aqua v2.17.1, v2.16.4, v2.15.0.

[suzuki-shunsuke]

I tried with another laptop, then expected_checksum was changed. A7A2B00B0157A49969AAAD48CBB0F6C03FA86F2427228FA2E43E38B2CECDE1F1
It's interesting.

$ aws --version
INFO[0000] download and unarchive the package            aqua_version=2.17.1 env=darwin/arm64 exe_name=aws package_name=aws/aws-cli package_version=2.13.35 program=aqua registry=standard
Downloading aws/aws-cli 2.13.35 100% |██████████████████████████████████████████████████████████████| (38/38 MB, 47 MB/s)        
FATA[0000] aqua failed                                   actual_checksum=36D16A6156150B3940CD7E750FBD7B9C529EE93C8E0EB594283F19A197DE5C42 aqua_version=2.17.1 env=darwin/arm64 error="install the package: checksum is invalid" exe_name=aws expected_checksum=A7A2B00B0157A49969AAAD48CBB0F6C03FA86F2427228FA2E43E38B2CECDE1F1 package_name=aws/aws-cli package_version=2.13.35 program=aqua

I removed aqua-checksum.json and retried aqua update-checksum, then the checksum was changed.

    {
      "id": "http/awscli.amazonaws.com/AWSCLIV2-2.13.35.pkg",
      "checksum": "3E7B40CF56AB11DB6AD789FA72311DC41EF731EC3665E0213573F6072C8875CF",
      "algorithm": "sha256"
    }

This is weird.

I disabled AQUA_PROGRESS_BAR just in case, but this is unrelated.

[suzuki-shunsuke]

I tried other packages 1xyz/pryrite (package type is github_release) and GoogleCloudPlatform/cloud-sql-proxy (package type is http), but I couldn't reproduce the issue.

So maybe this is an issue specific to aws/aws-cli and almost all packages are unrelated.

I'll try to change how to calculate the checksum of expected_checksum same with actual_checksum.

[suzuki-shunsuke]

As a workaround, please modify aqua-checksum.json manually.
checksum is case insensitive.

{
  "checksums": [
    {
      "id": "http/awscli.amazonaws.com/AWSCLIV2-2.13.35.pkg",
      "checksum": "36D16A6156150B3940CD7E750FBD7B9C529EE93C8E0EB594283F19A197DE5C42", // Please change the checksum to the correct value
      "algorithm": "sha256"
    },
    {
      "id": "http/storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.7.2/cloud-sql-proxy.darwin.arm64",
      "checksum": "D39342A3D724B860EE9840A9DA5F737CFB8C5FC113BF776038D9A7346D5B262D",
      "algorithm": "sha256"
    },
    {
      "id": "registries/github_content/github.com/aquaproj/aqua-registry/v4.86.0/registry.yaml",
      "checksum": "9EB5B05029E7821116C7C50F2998261ABDCDFDDE7857C0EBF5FF084497CAD7D4",
      "algorithm": "sha256"
    }
  ]
}

[suzuki-shunsuke]

I transferred the issue from aquaproj/aqua-registry to aquaproj/aqua because this is the issue of aqua itself.

[suzuki-shunsuke]

I could reproduce the issue with the small code.

main.go

package main

import (
	"errors"
	"fmt"
	"io"
	"log"
	"net/http"

	"github.com/codingsince1985/checksum"
)

func main() {
	if err := core(); err != nil {
		log.Fatal(err)
	}
}

func core() error {
	rc, err := download("http://awscli.amazonaws.com/AWSCLIV2-2.13.35.pkg")
	if err != nil {
		return err
	}
	defer rc.Close()
	s, err := checksum.SHA256sumReader(rc)
	if err != nil {
		return err
	}
	fmt.Println(s)
	return nil
}

var (
	errInvalidHTTPStatusCode = errors.New("status code >= 400")
)

func download(u string) (io.ReadCloser, error) {
	client := http.DefaultClient
	req, err := http.NewRequest(http.MethodGet, u, nil)
	if err != nil {
		return nil, fmt.Errorf("create a http request: %w", err)
	}
	resp, err := client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("send http request: %w", err)
	}
	if resp.StatusCode >= http.StatusBadRequest {
		return resp.Body, errInvalidHTTPStatusCode
	}
	log.Println("status code:", resp.StatusCode)
	return resp.Body, nil
}

The checksum is changed every time.

$ go run .  
2023/11/15 12:25:43 status code: 200
7184789b956d9359e1ac71ebe828470597bd458be918c2e7cac6a5960a512ef8

$ go run .
2023/11/15 12:25:46 status code: 200
e2a81ed5b16c77aa22478e3923c22ea4d3db4668fdfe702aeeb87851b4dd2d84

[suzuki-shunsuke]

The following code resolves the issue.

package main

import (
	"errors"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"

	"github.com/codingsince1985/checksum"
)

func main() {
	if err := core(); err != nil {
		log.Fatal(err)
	}
}

func core() error {
	path, err := downloadAndCopy()
	if err != nil {
		return err
	}
	defer func() {
		if err := os.Remove(path); err != nil {
			log.Println(err)
		}
	}()

	f, err := os.Open(path)
	if err != nil {
		return err
	}
	defer f.Close()

	s, err := checksum.SHA256sumReader(f)
	if err != nil {
		return err
	}
	fmt.Println(s)
	return nil
}

func downloadAndCopy() (string, error) {
	rc, err := download("http://awscli.amazonaws.com/AWSCLIV2-2.13.35.pkg")
	if err != nil {
		return "", err
	}
	defer rc.Close()
	f, err := os.CreateTemp("", "")
	if err != nil {
		return "", err
	}
	defer func() {
		f.Close()
	}()
	if _, err := io.Copy(f, rc); err != nil {
		return "", err
	}
	return f.Name(), nil
}

var (
	errInvalidHTTPStatusCode = errors.New("status code >= 400")
)

func download(u string) (io.ReadCloser, error) {
	client := http.DefaultClient
	req, err := http.NewRequest(http.MethodGet, u, nil)
	if err != nil {
		return nil, fmt.Errorf("create a http request: %w", err)
	}
	resp, err := client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("send http request: %w", err)
	}
	if resp.StatusCode >= http.StatusBadRequest {
		return resp.Body, errInvalidHTTPStatusCode
	}
	log.Println("status code:", resp.StatusCode)
	return resp.Body, nil
}

$ go run .  
2023/11/15 13:00:32 status code: 200
36d16a6156150b3940cd7e750fbd7b9c529ee93c8e0eb594283f19a197de5c42

$ go run .  
2023/11/15 13:01:24 status code: 200
36d16a6156150b3940cd7e750fbd7b9c529ee93c8e0eb594283f19a197de5c42

I'm not sure why, but it seems that we need to do the following things.

1. Create a temporal file
2. Copy response body to the temporal file
3. Close the response body
4. open the temporal file and calculate the checksum with the reader

[suzuki-shunsuke]

• fix(update-checksum): copy an asset to a temporal file to calculate the checksum correctly #2469

[suzuki-shunsuke]

v2.17.2 is out 🎉
https://github.com/aquaproj/aqua/releases/tag/v2.17.2

[suzuki-shunsuke]

If you face the issue already, please remove checksums of aws/aws-cli from aqua-checksums.json and run aqua update-checksum with aqua v2.17.2 or newer.

aqua-checksums.json

    // Delete this element
    {
      "id": "http/awscli.amazonaws.com/AWSCLIV2-2.13.35.pkg",
      "checksum": "520E1CB49004ECED7DB1CFE70E6FA73EFC6EDDF1CDB38AF535D126F1DB6574C8",
      "algorithm": "sha256"
    },