git: fix caching git commit through multiple refs #5215

Job	Run time
build (windows/amd64)	1m 43s
build (freebsd/amd64)	1m 51s
test-freebsd-amd64	2m 47s
test-windows-amd64 (containerd, ./client)	4m 17s
test-windows-amd64 (containerd, ./cmd/buildctl)	3m 11s
test-windows-amd64 (containerd, ./worker/containerd)	2m 50s
test-windows-amd64 (containerd, ./solver)	3m 39s
test-windows-amd64 (containerd, ./frontend)	4m 41s
test-windows-amd64 (containerd, ./frontend/dockerfile)	31m 56s
test-windows-amd64 (./..., 1)	9m 59s
	1h 6m 54s

Provide feedback