v2.3.4 throws context deadline exceeded (Client.Timeout exceeded while awaiting headers) #9478

ccfishk · 2022-05-22T01:48:40Z

Checklist:

I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[x ] I've included steps to reproduce the bug.
I've pasted the output of argocd version.

Describe the bug

command "argocd app create istio --repo https://github.intuit.com/services-mesh/services-mesh-release.git" throws repository not accessible

To Reproduce

On an EKS cluster after verifying the GitHub is reachable

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl port-forward svc/argocd-server -n argocd 8080:443
argocd login localhost:8080
argocd app create istio --repo https://github.intuit.com/services-mesh/services-mesh-release.git --path istio/yamls --dest-server https://kubernetes.default.svc --dest-namespace default

Expected behavior

application is able to get created successfully

Screenshots

Version

argocd version
argocd: v2.3.4+ac8b7df
BuildDate: 2022-05-18T13:00:56Z
GitCommit: ac8b7df
GitTreeState: clean
GoVersion: go1.17.9
Compiler: gc
Platform: darwin/amd64
argocd-server: v2.3.4+ac8b7df
BuildDate: 2022-05-18T11:41:37Z
GitCommit: ac8b7df
GitTreeState: clean
GoVersion: go1.17.10
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
Helm Version: v3.8.0+gd141386
Kubectl Version: v0.23.1
Jsonnet Version: v0.18.0

Logs

Paste any relevant application logs here.

[0015] rpc error: code = InvalidArgument desc = application spec for istio is invalid: InvalidSpecError: repository not accessible: rpc error: code = Unknown desc = error testing repository connectivity: Get "https://github.intuit.com/services-mesh/services-mesh-release.git/info/refs?service=git-upload-pack": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The text was updated successfully, but these errors were encountered:

bluefishforsale · 2022-05-31T20:40:04Z

+1 also having this issue
using :

- name: argo-cd
    version: 4.7.0
    repository: https://argoproj.github.io/argo-helm
    ```
    
I have about 30 apps in Argo, and have increased the reposerver min count to 3 and enabled the HPA scaler

~feels~ like a github.com API rate limit vs argo retry timeout issue

106062 · 2022-06-02T02:21:34Z

+1 also haveing same issue, after i update from version 2.2.x to 2.3.4.
also tried to Update to 2.3.3, have same issue with git repo url using http/https.

But, having install ArgoCD 2.3.3 in a new cluster. sure 2.3.3 's http works fine.

kraj007 · 2022-06-02T19:32:24Z

I am also facing the same issue for 'argocd repo add' command
error : FATA[0162] rpc error: code = DeadlineExceeded desc = context deadline exceeded

argocd: v2.3.3+07ac038
BuildDate: 2022-03-30T01:46:59Z
GitCommit: 07ac038
GitTreeState: clean
GoVersion: go1.17.6
Compiler: gc
Platform: linux/amd64
argocd-server: v2.3.4+ac8b7df
BuildDate: 2022-05-18T11:41:37Z
GitCommit: ac8b7df
GitTreeState: clean
GoVersion: go1.17.10
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
Helm Version: v3.8.0+gd141386
Kubectl Version: v0.23.1
Jsonnet Version: v0.18.0

dngferreira · 2022-06-07T10:18:59Z

Version v2.4.0-rc5 has the same problem.

ClemontX · 2022-06-10T10:30:48Z

same problem here with different versions.

❯ argo repo add https://github.com/louislam/uptime-kuma.git
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
FATA[0019] rpc error: code = Unknown desc = error testing repository connectivity: Get "https://github.com/louislam/uptime-kuma.git/info/refs?service=git-upload-pack": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I'm running on a kubespray one-node "Cluster" with calico and tested the connection to github via a netshoot pod.
A git clone worked inside this netshoot pod.

The interesting part:
I spun up a kind-Cluster and deployed argocd in there and i was able to get a Repository connection and was able to browse branches and directories via the argocd-webinterface.

yanghuijava · 2022-06-11T13:36:50Z

I had the same problem with v2.3.4

AnshumanTripathi · 2022-06-28T18:37:58Z

Seeing similar issues with 2.4.2

{
    "Version": "v2.4.2+c6d0c8b",
    "BuildDate": "2022-06-21T20:42:05Z",
    "GitCommit": "c6d0c8baaa291cd68465acd7ad6bef58b2b6f942",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
    "HelmVersion": "v3.8.1+g5cb9af4",
    "KubectlVersion": "v0.23.1",
    "JsonnetVersion": "v0.18.0"
}

wgzhao · 2022-07-22T02:27:32Z

I had the same promblem with V2.4.6, this problem is making me crack up.

argocd: v2.4.6+a48bca0
  BuildDate: 2022-07-12T22:56:26Z
  GitCommit: a48bca03c79b6d63be0c34d6094831bc6916b3bc
  GitTreeState: clean
  GoVersion: go1.18.3
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.4.6+a48bca0
  BuildDate: 2022-07-12T22:31:17Z
  GitCommit: a48bca03c79b6d63be0c34d6094831bc6916b3bc
  GitTreeState: clean
  GoVersion: go1.18.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
  Helm Version: v3.8.1+g5cb9af4
  Kubectl Version: v0.23.1
  Jsonnet Version: v0.18.0

phuongdkyt · 2022-12-28T11:51:35Z

how to fix?

jarrettprosser · 2023-01-12T00:34:04Z

I'm seeing the same on v2.1.6. Interestingly, I can exec into the repo server pod and manually put the repo with git, but when I try the same in the server pod it times out.

jmc000 · 2023-01-12T16:20:10Z

I had the same promblem with v2.4.7

mdfaizsiddiqui · 2023-02-03T22:50:10Z

I see the same warnings on argocd-repo-server pod's Liveness and Readiness probes -

3 Unhealthy: Liveness probe failed: Get "http://10.0.52.226:8084/healthz?full=true": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
1 Unhealthy: Readiness probe failed: Get "http://10.0.52.226:8084/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

{
    "Version": "v2.5.10+d311fad",
    "BuildDate": "2023-02-02T15:10:21Z",
    "GitCommit": "d311fad538178ef97528edf7f09d1d5b57268f45",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.5.7 2022-08-02T16:35:54Z",
    "HelmVersion": "v3.10.3+g835b733",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.18.0"
}

cevatkerim · 2023-03-14T20:07:49Z

same problem

crenshaw-dev · 2023-03-14T20:55:09Z

@mdfaizsiddiqui context deadline exceeded is a generic error message in golang. It can appear in many contexts. I'm guessing most folks here are getting that error when the repo-server tries to communicate with some git host.

emirot · 2023-03-28T21:46:08Z

@crenshaw-dev that's correct trying to connect to git. (argocd 2.4.11)
Is there a way to increase that timeout, for instance by logging into the pod doing kubectl exec -it argocd-repo-server bash -n argocd
I can clone a repo but it would fail in UI and I would get that in the log and popup.

time="2023-03-28T21:22:56Z" level=error msg="finished unary call with code Unknown" error="Get \"https://private-ghe.com/myorg/myrepo/info/refs?service=git-upload-pack\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2023-03-28T21:22:41Z" grpc.time_ms=15000.81 span.kind=server system=grpc

lukaszkubisiak · 2023-04-04T09:56:56Z

the same problem here.
I'm behind the proxy. Maybe that's the issue? But connection to repository (in the UI under Settings | Repository) is successful.

lukaszkubisiak · 2023-04-04T11:35:56Z

This helped me: #2954

fredcsfaria · 2023-06-05T21:03:01Z

same problem here. Some fix ?

zodiac12k · 2023-06-13T09:25:12Z

I also have the same problem.

behniafb · 2023-10-18T13:37:24Z

Still no one found a solution for this issue?
Then how do you add a private repo to ArgoCD?

myysophia · 2024-02-20T07:33:18Z

I resolved this issue by scale up argocd-server.

shay-ul · 2024-04-02T09:49:33Z

Increasing memory requests and limtis for argocd-repo-server did the trick for me.

bredamatt · 2024-09-19T16:14:36Z

I am also having this same issue from a private EKS cluster, however isn't this just simply a networking issue? My repo has literally nothing in it atm. so not sure why it is not working.

Venture200 · 2024-10-16T13:04:01Z

Hello everyone

I fixed the issue by increasing repo-server replicas to 2 and it started to work. Hopefully helps you.

Have a wonderful day!

EDIT: While adding second repo, got this error again

brian-fogg-fileread · 2025-01-14T00:08:29Z

following

douglasawh · 2025-01-29T22:37:27Z

Maybe there should be some additional hints in the error message? Based on increasing memory requests and limits comment, perhaps something about increasing resources should be suggested?

There is already and note in the documentation to do that, though we have a screenshot above not a link. So, here's the link for anyone struggling with this: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/

ccfishk added the bug Something isn't working label May 22, 2022

CypZ mentioned this issue Jun 29, 2023

[doc] use argocd behind a proxy #12347

Open

u72z mentioned this issue Jun 30, 2023

Proxy env variables on argocd-server breaks repos connectivity #14280

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2.3.4 throws context deadline exceeded (Client.Timeout exceeded while awaiting headers) #9478

v2.3.4 throws context deadline exceeded (Client.Timeout exceeded while awaiting headers) #9478

ccfishk commented May 22, 2022

bluefishforsale commented May 31, 2022

106062 commented Jun 2, 2022

kraj007 commented Jun 2, 2022

dngferreira commented Jun 7, 2022

ClemontX commented Jun 10, 2022 •

edited

Loading

yanghuijava commented Jun 11, 2022

AnshumanTripathi commented Jun 28, 2022

wgzhao commented Jul 22, 2022 •

edited

Loading

phuongdkyt commented Dec 28, 2022

jarrettprosser commented Jan 12, 2023

jmc000 commented Jan 12, 2023

mdfaizsiddiqui commented Feb 3, 2023

cevatkerim commented Mar 14, 2023

crenshaw-dev commented Mar 14, 2023

emirot commented Mar 28, 2023 •

edited

Loading

lukaszkubisiak commented Apr 4, 2023

lukaszkubisiak commented Apr 4, 2023

fredcsfaria commented Jun 5, 2023

zodiac12k commented Jun 13, 2023

behniafb commented Oct 18, 2023

myysophia commented Feb 20, 2024

shay-ul commented Apr 2, 2024

bredamatt commented Sep 19, 2024

Venture200 commented Oct 16, 2024 •

edited

Loading

brian-fogg-fileread commented Jan 14, 2025

douglasawh commented Jan 29, 2025

v2.3.4 throws context deadline exceeded (Client.Timeout exceeded while awaiting headers) #9478

v2.3.4 throws context deadline exceeded (Client.Timeout exceeded while awaiting headers) #9478

Comments

ccfishk commented May 22, 2022

bluefishforsale commented May 31, 2022

106062 commented Jun 2, 2022

kraj007 commented Jun 2, 2022

dngferreira commented Jun 7, 2022

ClemontX commented Jun 10, 2022 • edited Loading

yanghuijava commented Jun 11, 2022

AnshumanTripathi commented Jun 28, 2022

wgzhao commented Jul 22, 2022 • edited Loading

phuongdkyt commented Dec 28, 2022

jarrettprosser commented Jan 12, 2023

jmc000 commented Jan 12, 2023

mdfaizsiddiqui commented Feb 3, 2023

cevatkerim commented Mar 14, 2023

crenshaw-dev commented Mar 14, 2023

emirot commented Mar 28, 2023 • edited Loading

lukaszkubisiak commented Apr 4, 2023

lukaszkubisiak commented Apr 4, 2023

fredcsfaria commented Jun 5, 2023

zodiac12k commented Jun 13, 2023

behniafb commented Oct 18, 2023

myysophia commented Feb 20, 2024

shay-ul commented Apr 2, 2024

bredamatt commented Sep 19, 2024

Venture200 commented Oct 16, 2024 • edited Loading

brian-fogg-fileread commented Jan 14, 2025

douglasawh commented Jan 29, 2025

ClemontX commented Jun 10, 2022 •

edited

Loading

wgzhao commented Jul 22, 2022 •

edited

Loading

emirot commented Mar 28, 2023 •

edited

Loading

Venture200 commented Oct 16, 2024 •

edited

Loading