Preview 2.0 Panic error

[red-erik]

Hello,
just tried and received this
Last line was: INF Scanning subscriptions for Microsoft.SignalRService/webPubSub

panic: interface conversion: interface {} is map[string]interface {}, not string

goroutine 80 [running]:
github.com/Azure/azqr/internal.AprlScanner.graphScan({}, {0x2431e68, 0xc0000916d0}, 0xc00040e168, {0xc000352b40, 0xc, 0x0?}, 0xc000279350)
D:/a/azqr/azqr/internal/aprl_scanner.go:198 +0xb94
github.com/Azure/azqr/internal.AprlScanner.Scan.func2({0xc000352b40?, 0xc?, 0x8f?}, 0xa9aa0a9aa0a9aa0?)
D:/a/azqr/azqr/internal/aprl_scanner.go:154 +0x125
created by github.com/Azure/azqr/internal.AprlScanner.Scan in goroutine 1
D:/a/azqr/azqr/internal/aprl_scanner.go:146 +0x254

Regards,
Red.

[cmendible]

Hi thanks for reporting.

Can you please set the AZURE_SDK_GO_LOGGING environment variable to all and run the tool with the --debug flag and share the details?

[red-erik]

Hello,
it seems to be quite the same:

[Aug 20 11:36:11.379389] Retry: response 200
[Aug 20 11:36:11.380006] Retry: exit due to non-retriable status code
panic: runtime error: index out of range [2] with length 1

goroutine 58 [running]:
github.com/Azure/azqr/internal/azqr.GetSubsctiptionFromResourceID(...)
D:/a/azqr/azqr/internal/azqr/azqr.go:274
github.com/Azure/azqr/internal.AprlScanner.graphScan({}, {0x2431e68, 0xc000156050}, 0xc00010c2e0, {0xc000438b40, 0xc, 0x0?}, 0xc0002f8870)
D:/a/azqr/azqr/internal/aprl_scanner.go:223 +0xb37
github.com/Azure/azqr/internal.AprlScanner.Scan.func2({0xc000438b40?, 0xc?, 0x8f?}, 0xffffffffffffffff?)
D:/a/azqr/azqr/internal/aprl_scanner.go:154 +0x125
created by github.com/Azure/azqr/internal.AprlScanner.Scan in goroutine 1
D:/a/azqr/azqr/internal/aprl_scanner.go:146 +0x254
Regards,
Red.

[red-erik]

We have
RESPONSE Status: 429 Too Many Requests so I suppose it's related to that
https://learn.microsoft.com/en-us/graph/throttling
https://learn.microsoft.com/en-us/azure/architecture/patterns/throttling

Regards,
Red.

[cmendible]

Hello, it seems to be quite the same:

[Aug 20 11:36:11.379389] Retry: response 200 [Aug 20 11:36:11.380006] Retry: exit due to non-retriable status code panic: runtime error: index out of range [2] with length 1

goroutine 58 [running]: github.com/Azure/azqr/internal/azqr.GetSubsctiptionFromResourceID(...) D:/a/azqr/azqr/internal/azqr/azqr.go:274 github.com/Azure/azqr/internal.AprlScanner.graphScan({}, {0x2431e68, 0xc000156050}, 0xc00010c2e0, {0xc000438b40, 0xc, 0x0?}, 0xc0002f8870) D:/a/azqr/azqr/internal/aprl_scanner.go:223 +0xb37 github.com/Azure/azqr/internal.AprlScanner.Scan.func2({0xc000438b40?, 0xc?, 0x8f?}, 0xffffffffffffffff?) D:/a/azqr/azqr/internal/aprl_scanner.go:154 +0x125 created by github.com/Azure/azqr/internal.AprlScanner.Scan in goroutine 1 D:/a/azqr/azqr/internal/aprl_scanner.go:146 +0x254 Regards, Red.

This seems to be yet another issue. If you run with the env variable and the --debug flag you should see the ARG query that causes the issues. Can you share please share it?

[cmendible]

We have RESPONSE Status: 429 Too Many Requests so I suppose it's related to that https://learn.microsoft.com/en-us/graph/throttling https://learn.microsoft.com/en-us/azure/architecture/patterns/throttling

Regards, Red.

Throttling is been handled and is not the cause of these Panic errors

[red-erik]

Hello,
don't know what you exactly mean (sorry, I'm only a user and not a developer) but, last lines before the error are:

[Aug 20 11:36:10.816275] Retry: response 429
[Aug 20 11:36:10.816275] Retry: End Try #1, Delay=1s
[Aug 20 11:36:11.378879] Response: ==> REQUEST/RESPONSE (Try=1/829.39ms, OpTime=829.39ms) -- RESPONSE RECEIVED
POST https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2021-06-01-preview
Accept: application/json
Authorization: REDACTED
Content-Length: 2242
Content-Type: application/json
User-Agent: azsdk-go-armresourcegraph/v0.9.0 (go1.22.0; Windows_NT)

RESPONSE Status: 200 OK
Cache-Control: no-cache
Content-Length: 275362
Content-Type: application/json; charset=utf-8
Date: Tue, 20 Aug 2024 09:36:10 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: REDACTED
X-Cache: REDACTED
X-Content-Type-Options: REDACTED
X-Ms-Correlation-Request-Id: REDACTED
X-Ms-Ratelimit-Remaining-Tenant-Reads: REDACTED
X-Ms-Ratelimit-Remaining-Tenant-Resource-Requests: REDACTED
X-Ms-Request-Id: 116da4f0-70aa-4944-8074-8129e12ce11d
X-Ms-Resource-Graph-Request-Duration: REDACTED
X-Ms-Routing-Request-Id: REDACTED
X-Ms-User-Quota-Remaining: REDACTED
X-Ms-User-Quota-Resets-After: REDACTED
X-Msedge-Ref: REDACTED

Regards,
Red.

[cmendible]

Hmm sorry about that. If you use the --debug flag (i.e. azqr scan --debug) you should see messages like:

DBG // Azure Resource Graph Query
// This resource graph query will return all storage accounts that does not have a Private Endpoint Connection or where a private endpoint exists but public access is enabled
resources
| where type =~ "Microsoft.Storage/StorageAccounts"
| where isnull(properties.privateEndpointConnections) or properties.privateEndpointConnections[0].properties.provisioningState != ("Succeeded") or (isnull(properties.networkAcls) and properties.publicNetworkAccess == 'Enabled')
| extend param1 = strcat('Private Endpoint: ', iif(isnotnull(properties.privateEndpointConnections),split(properties.privateEndpointConnections[0].properties.privateEndpoint.id,'/')[8],'No Private Endpoint'))
| extend param2 = strcat('Access: ', iif(properties.publicNetworkAccess == 'Disabled', 'Public Access Disabled', iif(isnotnull(properties.networkAcls), 'NetworkACLs in place','Public Access Enabled')))
| project recommendationId = "dc55be60-6f8c-461e-a9d5-a3c7686ed94e", name, id, tags, param1, param2

If possible share with me the Azure Resource Graph Query that shows right before the panic error.

Thanks for your patience and help!

[red-erik]

Hello,
it should be this one:

[90m2024-08-20T12:40:03+02:00�[0m DBG // Azure Resource Graph Query
// Find Subnets with Service Endpoint enabled for services that offer Private Link
resources
| where type =~ 'Microsoft.Network/virtualnetworks'
| mv-expand subnets = properties.subnets
| extend se = array_length(subnets.properties.serviceEndpoints)
| where se >= 1
| project name, id, tags, subnets, serviceEndpoints=todynamic(subnets.properties.serviceEndpoints)
| mv-expand serviceEndpoints
| project name, id, tags, subnetName=subnets.name, serviceName=tostring(serviceEndpoints.service)
| where serviceName in (parse_json('["Microsoft.CognitiveServices","Microsoft.AzureCosmosDB","Microsoft.DBforMariaDB","Microsoft.DBforMySQL","Microsoft.DBforPostgreSQL","Microsoft.EventHub","Microsoft.KeyVault","Microsoft.ServiceBus","Microsoft.Sql", "Microsoft.Storage","Microsoft.StorageSync","Microsoft.Synapse","Microsoft.Web"]'))
| project recommendationId = "24ae3773-cc2c-3649-88de-c9788e25b463", name, id, tags, param1 = strcat("subnet=", subnetName), param2=strcat("serviceName=",serviceName), param3="ServiceEndpoints=true"

but not sure since I don't see it on session screen but only using > debug.txt (where I don't see the panic error)

The command line is:
.\azqr2_0.exe scan --mask=false -f --debug > debug.txt

The env is quite huge

Regards,
Red.

[cmendible]

Just released: v.2.0.0-preview.4

Please check if it fixes the issue for you.

[red-erik]

seems to be running, but very slow

Keep you updated.

2024-08-20T14:41:07+02:00 FTL Failed to get diagnostic settings error="Post "https://management.azure.com/batch?api-version=2020-06-01\": dial tcp [2603:1030:a0c::10]:443: bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full."

Running again with debug option

Red.

[cmendible]

Now it seems that due to the huge size of your environment, azqr is attempting more than 13.000 diagnostics settings queries in parallel and failing.

Let me see what I can do to fix this.

In the mean time please use the -s flag to set a subscription Id and if needed the -g flag to specify a resource group in order to reduce the number of services.

[cmendible]

@red-erik closing this one since the panic error was fixed.

Diagnostic Settings issue for large environments will be tracked here: #249