DevOps example not working correctly #971

FrankHoogmans · 2020-11-10T08:18:44Z

Which Renovate are you using?

Renovate Open Source CLI

Which platform are you using?

Azure DevOps

Have you checked the logs? Don't forget to include them if relevant

Every single repo gives the following error (redacted repo names):

       "renovateVersion": "23.74.1"
ERROR: Repository has unknown error (repository=*******)
       "err": {
         "statusCode": 401,
         "result": {
           "$id": "1",
           "innerException": null,
           "message": "TF400813: The user '' is not authorized to access this resource.",
           "typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server",
           "typeKey": "UnauthorizedRequestException",
           "errorCode": 0,
           "eventId": 3000
         },
         "message": "TF400813: The user '' is not authorized to access this resource.",
         "stack": "Error: TF400813: The user '' is not authorized to access this resource.\n    at RestClient.<anonymous> (/home/vsts/.npm/_npx/2374/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:202:31)\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/vsts/.npm/_npx/2374/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:6:58)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"
       }
 INFO: Repository finished (repository=*******)
       "durationMs": 402

What would you like to do?

We are using Renovate in an Azure DevOps pipeline, based on the starter template available at https://docs.renovatebot.com/setup-azure-devops/. This runs multiple times a day, to update dependencies for all of our modules (repositories).

Last friday (6th of november) everything worked great, but yesterday I started getting 401's on the DevOps repository access. I have not changed the configuration in any way, and even tried the template in another devops account to be sure it wasn't account related. Downgrading to the version that last worked did also not provide a working solution (same errors).

I have checked, and do not have any limitations on the System.AccessToken about using it in scripts or across repositories.

I am unsure if this is somehow a bug in renovate, or if i'm doing anything wrong with the configuration, but given that the example doesn't work for me anymore, gives me reason to think something changed, but I seem to be unable to figure out what.

The text was updated successfully, but these errors were encountered:

rarkins · 2020-11-10T09:10:18Z

@JamieMagee do you think this could be related to the transitive dependency updates yesterday?

I also recall previously problems of expired tokens but this error message doesn't warn about that.

FrankHoogmans · 2020-11-10T09:16:20Z

I also recall previously problems of expired tokens but this error message doesn't warn about that.

I have already tried swapping tokens (switching to a PAT for my own account instead of System.AccessToken). Creating a new pipeline with the same yaml, and trying the template on a different devops account. So expired tokens don't sound likely.

JamieMagee · 2020-11-10T10:12:30Z

@JamieMagee do you think this could be related to the transitive dependency updates yesterday?

I don't think so. The lockfile updates went out yesterday, but this issue started happening on Friday.

I am also seeing this on my own Renovate jobs running against Azure DevOps (I should check in on those more often!). They started failing on Friday between 19:00 and 20:00 CET, but the Renovate version was 23.73.2 for both. It's still failing right now.

This might be something on the Azure DevOps side. Let me investigate it.

FrankHoogmans · 2020-11-11T07:49:22Z

@JamieMagee do you think this could be related to the transitive dependency updates yesterday?

I don't think so. The lockfile updates went out yesterday, but this issue started happening on Friday.

I am also seeing this on my own Renovate jobs running against Azure DevOps (I should check in on those more often!). They started failing on Friday between 19:00 and 20:00 CET, but the Renovate version was 23.73.2 for both. It's still failing right now.

This might be something on the Azure DevOps side. Let me investigate it.

I also did some more digging, but have not found anything so far. I'll just share what I checked.

The Azure DevOps release notes seem to indicate nothing has changed on Friday the 6th of November:
https://docs.microsoft.com/en-us/azure/devops/release-notes/2020/sprint-177-update
Last release of anything seems to have been October 21st.

I checked the run logs between a run that worked on friday, and one that failed on monday.
Both runs are using exactly the same software on the build agent (ubuntu image, devops agent software, bash and npmauthenticate task versions). This is the 'Initialize job' logging:

Agent name: 'Hosted Agent'
Agent machine name: 'fv-az101-655'
Current agent version: '2.175.2'
Operating System
Ubuntu
18.04.5
LTS
Virtual Environment
Environment: ubuntu-18.04
Version: 20201102.0
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20201102.0/images/linux/Ubuntu1804-README.md
Current image version: '20201102.0'
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: Bash (3.177.2)
Downloading task: npmAuthenticate (0.174.0)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Finished checking job knob settings.
Start tracking orphan processes.

rarkins · 2020-11-11T08:44:26Z

So same version of Renovate and (apparent) same version of Azure DevOps, but it suddenly stopped working?

rarkins · 2020-11-11T08:48:13Z

Can somebody log at debug level in case it gives additional info? I'm guessing it's failing at one of these lines: https://github.com/renovatebot/renovate/blob/ad50398fb3981a7c5d04b96bddce3dbad45a7d21/lib/platform/azure/index.ts#L124-L125

rarkins · 2020-11-11T08:53:23Z

This question was coincidentally asked yesterday: https://stackoverflow.com/questions/64759524/tf400813-the-user-is-not-authorized-to-access-this-resource-when-using-syste

Someone asked the same question 4 months ago, here was some suggestion from a MSFT employee:

Note that the OP said that the problem resolved itself though.

FrankHoogmans · 2020-11-11T09:03:43Z

So same version of Renovate and (apparent) same version of Azure DevOps, but it suddenly stopped working?

Correct. Although at first the renovate version was different, but reverting doesn't fix it.

This question was coincidentally asked yesterday: https://stackoverflow.com/questions/64759524/tf400813-the-user-is-not-authorized-to-access-this-resource-when-using-syste

Someone asked the same question 4 months ago, here was some suggestion from a MSFT employee:

Note that the OP said that the problem resolved itself though.

I agree it seems to be a DevOps (authentication) issue... We use the Microsoft provided build agents and images, no proxy settings, and (I guess) 0 debugging on network issues there for us. I can't seem to find any other reports regarding these issues...

I still hope it will resolve itself automagically...

FrankHoogmans · 2020-11-11T10:19:55Z

Can somebody log at debug level in case it gives additional info? I'm guessing it's failing at one of these lines: https://github.com/renovatebot/renovate/blob/ad50398fb3981a7c5d04b96bddce3dbad45a7d21/lib/platform/azure/index.ts#L124-L125

Ok, a debug run seems to confirm that it fails somewhere after 'initRepo'.
This is the debug-logging of a single repo (redacted repo name)

2020-11-11T10:12:20.2543975Z DEBUG: Using default gitAuthor: Renovate Bot <[email protected]>
2020-11-11T10:12:20.2544209Z DEBUG: Using baseDir: /tmp/renovate
2020-11-11T10:12:20.2544385Z DEBUG: Using cacheDir: /tmp/renovate/cache
2020-11-11T10:12:20.2544787Z DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
2020-11-11T10:12:20.2545019Z DEBUG: Commits limit = null
2020-11-11T10:12:20.2545204Z  INFO: Repository started (repository=***)
2020-11-11T10:12:20.2545405Z        "renovateVersion": "23.77.0"
2020-11-11T10:12:20.2545663Z DEBUG: Using localDir: /tmp/renovate/repos/azure/*** (repository=***)
2020-11-11T10:12:20.2545983Z DEBUG: initRepo("***") (repository=***)
2020-11-11T10:12:20.6909588Z ERROR: Repository has unknown error (repository=***)
2020-11-11T10:12:20.6909994Z        "err": {
2020-11-11T10:12:20.6910525Z          "statusCode": 401,
2020-11-11T10:12:20.6910718Z          "result": {
2020-11-11T10:12:20.6910898Z            "$id": "1",
2020-11-11T10:12:20.6911119Z            "innerException": null,
2020-11-11T10:12:20.6912043Z            "message": "TF400813: The user '' is not authorized to access this resource.",
2020-11-11T10:12:20.6912504Z            "typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server",
2020-11-11T10:12:20.6912984Z            "typeKey": "UnauthorizedRequestException",
2020-11-11T10:12:20.6913193Z            "errorCode": 0,
2020-11-11T10:12:20.6913365Z            "eventId": 3000
2020-11-11T10:12:20.6913509Z          },
2020-11-11T10:12:20.6913925Z          "message": "TF400813: The user '' is not authorized to access this resource.",
2020-11-11T10:12:20.6915288Z          "stack": "Error: TF400813: The user '' is not authorized to access this resource.\n    at RestClient.<anonymous> (/home/vsts/.npm/_npx/2588/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:202:31)\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/vsts/.npm/_npx/2588/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:6:58)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"
2020-11-11T10:12:20.6916019Z        }
2020-11-11T10:12:20.6919136Z DEBUG: Unknown res (repository=***)
2020-11-11T10:12:20.6919594Z        "res": "unknown-error"
2020-11-11T10:12:20.6931906Z DEBUG: Repository timing splits (milliseconds) (repository=***)
2020-11-11T10:12:20.6932204Z        "splits": {},
2020-11-11T10:12:20.6932363Z        "total": 463
2020-11-11T10:12:20.6936525Z  INFO: Repository finished (repository=***)
2020-11-11T10:12:20.6936775Z        "durationMs": 463

JamieMagee · 2020-11-11T10:20:45Z

I figured out the issue. It's a little bit convoluted but bear with me.

On Friday 6th November, version 1.8.0 of typed-rest-client was published. It included this commit about disabling authentication on redirects.

Looking at the changes on that commit, the authentication is not added to every request, so we're authenticating as an anonymous user for some requests (where have I seen that error 🤔)

azure-devops-node-api has a dependency on typed-rest-client@^1.7.3. So, even though Renovate specifies a different version in our yarn.lock file, because our guide use npx the yarn.lock isn't used. The ^1.7.3 range resolves to 1.8.0, which is currently broken.

Therefore, the fix for this should either be in azure-devops-node-api to set allowCrossOriginAuthentication, or in typed-rest-client to add authentication to all requests.

JamieMagee · 2020-11-11T10:30:43Z

I continued the discussion here

rarkins · 2020-11-13T19:36:46Z

@JamieMagee do we need to make any code modifications now?

JamieMagee · 2020-11-13T19:46:01Z

Yes, small code change required.

I can send a PR out later this evening, or tomorrow. I'm on mobile right now.

FrankHoogmans · 2020-11-16T06:17:23Z

I can confirm version 23.83.3 of renovatebot does work as expected again. Thank you so much for the help 👍

rarkins mentioned this issue Nov 12, 2020

Getting value of null error with Azure Devops #977

Closed

rarkins added the pinned label Nov 12, 2020

JamieMagee mentioned this issue Nov 14, 2020

fix(azure): allow cross origin auth for azure devops renovatebot/renovate#7728

Merged

5 tasks

FrankHoogmans closed this as completed Nov 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DevOps example not working correctly #971

DevOps example not working correctly #971

FrankHoogmans commented Nov 10, 2020 •

edited

Loading

rarkins commented Nov 10, 2020

FrankHoogmans commented Nov 10, 2020 •

edited

Loading

JamieMagee commented Nov 10, 2020

FrankHoogmans commented Nov 11, 2020

rarkins commented Nov 11, 2020

rarkins commented Nov 11, 2020

rarkins commented Nov 11, 2020

FrankHoogmans commented Nov 11, 2020

FrankHoogmans commented Nov 11, 2020 •

edited

Loading

JamieMagee commented Nov 11, 2020

JamieMagee commented Nov 11, 2020

rarkins commented Nov 13, 2020

JamieMagee commented Nov 13, 2020

FrankHoogmans commented Nov 16, 2020

DevOps example not working correctly #971

DevOps example not working correctly #971

Comments

FrankHoogmans commented Nov 10, 2020 • edited Loading

Which Renovate are you using?

Which platform are you using?

Have you checked the logs? Don't forget to include them if relevant

What would you like to do?

rarkins commented Nov 10, 2020

FrankHoogmans commented Nov 10, 2020 • edited Loading

JamieMagee commented Nov 10, 2020

FrankHoogmans commented Nov 11, 2020

rarkins commented Nov 11, 2020

rarkins commented Nov 11, 2020

rarkins commented Nov 11, 2020

FrankHoogmans commented Nov 11, 2020

FrankHoogmans commented Nov 11, 2020 • edited Loading

JamieMagee commented Nov 11, 2020

JamieMagee commented Nov 11, 2020

rarkins commented Nov 13, 2020

JamieMagee commented Nov 13, 2020

FrankHoogmans commented Nov 16, 2020

FrankHoogmans commented Nov 10, 2020 •

edited

Loading

FrankHoogmans commented Nov 10, 2020 •

edited

Loading

FrankHoogmans commented Nov 11, 2020 •

edited

Loading