403 forbidden error when createShareIfNotExists

[yungezz]

I have a vscode extension, which call createShareIfNotExists to createFileShare. While I found that on one of my laptop, this call always got 403 error, but this scenario is not reproachable on other machines.

I debugged the code, its' using correct account name and key(reason why it passed on other machine). I did some search found, storage auth might fail with 403 if time between service and request time stamp is out of sync. And I did see some time service error on that laptop.

The time provider 'VMICTimeProvider' has indicated that the current hardware and operating environment is not supported and has stopped. This behavior is expected for VMICTimeProvider on non-HyperV-guest environments. This may be the expected behavior for the current provider in the current operating environment as well.

actually my laptop date & time setting is in right time zone, right time. No idea why this 403 happens even with right accoutn name and key. below is sample call and response

activating extension
Starting express server on port: 1863
[2018-07-26T06:41:20.624Z]debug : FINAL REQUEST OPTIONS:
{ uri: 'https://csxxxxxxx.file.core.windows.net:443/xxxxxxxxxxxxxx?restype=share',
  method: 'HEAD',
  headers: 
   { 'x-ms-client-request-id': 'e7fd9bb0-909e-11e8-871f-919af9f38dfd',
     'user-agent': 'Azure-Storage/2.10.0 (NODE-VERSION v7.9.0; Windows_NT 10.0.16299)',
     'x-ms-version': '2018-03-28',
     'x-ms-date': 'Thu, 26 Jul 2018 06:41:20 GMT',
     accept: 'application/atom+xml,application/xml',
     'Accept-Charset': 'UTF-8',
     'content-type': '',
     'content-length': 0,
     authorization: 'SharedKey csxxxxx:xxxxx' },
  mode: 'disable-fetch',
  encoding: undefined,
  timeout: 120000,
  forever: true }
[2018-07-26T06:41:21.230Z]debug : RESPONSE:
{ error: 
   { Error: 
     at Function.StorageServiceClient._normalizeError (C:\work\vscode-ansible\node_modules\azure-storage\lib\common\services\storageserviceclient.js:1198:23)
     at FileService.StorageServiceClient._processResponse (C:\work\vscode-ansible\node_modules\azure-storage\lib\common\services\storageserviceclient.js:744:50)
     at Request.processResponseCallback [as _callback] (C:\work\vscode-ansible\node_modules\azure-storage\lib\common\services\storageserviceclient.js:317:37)
     at Request.init.self.callback (C:\work\vscode-ansible\node_modules\request\request.js:185:22)
     at emitTwo (events.js:106:13)
     at Request.emit (events.js:194:7)
     at Request.<anonymous> (C:\work\vscode-ansible\node_modules\request\request.js:1157:10)
     at emitOne (events.js:96:13)
     at Request.emit (events.js:191:7)
     at IncomingMessage.<anonymous> (C:\work\vscode-ansible\node_modules\request\request.js:1079:12)
     at Object.onceWrapper (events.js:293:19)
     at emitNone (events.js:91:20)
     at IncomingMessage.emit (events.js:188:7)
     at endReadableNT (_stream_readable.js:975:12)
     at _combinedTickCallback (internal/process/next_tick.js:80:11)
     at process._tickCallback (internal/process/next_tick.js:104:9)
     name: 'StorageError',
     message: 'Forbidden',
     code: 'Forbidden',
     statusCode: 403,
     requestId: 'd8c8e114-201a-0073-4cab-24def7000000' },
  response: 
   { isSuccessful: false,
     statusCode: 403,
     body: '',
     headers: 
      { 'transfer-encoding': 'chunked',
        server: 'Microsoft-HTTPAPI/2.0',
        'x-ms-request-id': 'd8c8e114-201a-0073-4cab-24def7000000',
        'x-ms-error-code': 'AuthenticationFailed',
        date: 'Thu, 26 Jul 2018 06:41:20 GMT' },
     md5: undefined } }

[XiaoningLiu]

There is a header x-ms-request-source-ikey received in Storage server side, but not in your attached request headers log. Can you check why this header is there?

Following are server side log for this request. Removed header value of x-ms-request-source-ikey in following log.

info: NephosFile.exe: Info: Processing exception: Microsoft.Cis.Services.Nephos.Common.Authentication.AuthenticationFailureException : The MAC signature found in the HTTP request 'GEiI8aunitHfD+m13DS92yrv7/gnXTaoNaVuiVJIRGU=' is not the same as any computed signature. Server used following string to sign: 'HEAD............x-ms-client-request-id:e7fd9bb0-909e-11e8-871f-919af9f38dfd.x-ms-date:Thu, 26 Jul 2018 06:41:20 GMT.x-ms-request-source-ikey:[removed value in this GitHub Issue].x-ms-version:2018-03-28./cs1685ba005af8dx4b04x8f1/[removed value in this GitHub Issue].restype:share'.

[yungezz]

no idea of why the header was there. here's my code to make storage call,

https://github.com/VSChina/vscode-ansible/blob/f832e5d6e1d87982506ea7159dbe1320233d507e/src/azureStorageHelper.ts#L10

https://github.com/VSChina/vscode-ansible/blob/f832e5d6e1d87982506ea7159dbe1320233d507e/src/azureStorageHelper.ts#L24

azure storage version:

"azure-storage": "^2.8.2",

[XiaoningLiu]

@yungezz A similar issue microsoft/ApplicationInsights-node.js#143

[yungezz]

thanks for investigation. Just curious, why the issue only reproducible on some machines, on majority cases, it works well.

[XiaoningLiu]

@yungezz sorry for the late response. It's hard to say without investigation, there maybe some plugins or other environments difference.

[yungezz]

@XiaoningLiu thanks for investigation. We've got more and more customer escalation on this issue. Our extension code didn't specify the header x-ms-request-source-ikey, but directly call sdk methods. Is there way to find out where the header was added? meanwhile, is there way to workaround this issue? Thanks.

[yungezz]

@XiaoningLiu any update on this?

[XiaoningLiu]

@yungezz azure-storage-node doesn't set the header. I cannot tell which VSC plugin or extension modify the request before sending out. You need some debugging to find out who modify this.

Using shared key for authentication will force Azure Storage server check headers integrity, e.g. injecting headers will lead auth error.

But one workaround is that you can use Azure Storage SAS token or OAuth token for authentication.

[yungezz]

@XiaoningLiu I did debug the extension, the extension has no way to add header to request to azure storage since we're calling sdk to make the call, you can see it from code snippet:

const client = storage.createFileService(storageAccountName, storageAccountKey);

...
function createFileShare(client: FileService, fileShareName: string): Promise<void> {
    return new Promise<void>((resolve, reject) => {
        client.createShareIfNotExists(fileShareName, (err, result, response) => {
            if (err) {
                reject(err);
            } else {
                resolve();
            }
        });
    })
}

we'll try using SAS token to see if the error persistents.

[yungezz]

close this since sas workaround the issue.