Proxy settings not applied to all outgoing requests

[dlouzan]

Describe the bug

While using spectral on a corporate network, our lints are taking a long time (hitting a network timeout) because not all outgoing connections are honouring the proxy settings.

After initiating a lint, spectral seemed to hang and we tracked this to an outgoing connection to http://json-schema.org on startup:

NET 338: createConnection [
  {
    protocol: 'http:',
    slashes: true,
    auth: null,
    host: 'json-schema.org',
    port: 80,
    hostname: 'json-schema.org',
    hash: null,
    search: null,
    query: null,
    pathname: '/draft-04/schema',
    path: null,
    href: 'http://json-schema.org/draft-04/schema',
    method: 'GET',
    headers: [Object: null prototype] {
      Accept: [Array],
      'User-Agent': [Array],
      'Accept-Encoding': [Array],
      Connection: [Array]
    },
    agent: undefined,
    servername: 'json-schema.org',
    _agentKey: 'json-schema.org:80:',
    encoding: null
  },
  [Function: oncreate],
  [Symbol(normalizedArgs)]: true
]

...

TIMER 304: process timer lists 136415
TIMER 304: timeout callback 1000
TIMER 304: 1000 list wait because diff is 0
TIMER 304: process timer lists 137416
TIMER 304: timeout callback 1000
TIMER 304: 1000 list wait because diff is 0
TIMER 304: process timer lists 138417
TIMER 304: timeout callback 1000
TIMER 304: 1000 list wait because diff is 0
TIMER 304: process timer lists 139418
TIMER 304: timeout callback 1000
TIMER 304: 1000 list wait because diff is 0
TIMER 304: process timer lists 140419
TIMER 304: timeout callback 1000

...

Looks like this is triggered from one of the libraries (ajv?), and it is not respecting the proxy settings. The net effect is that on startup spectral will hang for a big while until the timeout hits and then the lint continues.

On top of this, the standard environment variables HTTP(S)_PROXY are not used to configure the proxy settings, the CLI will only recognize PROXY. This is documented in https://meta.stoplight.io/docs/spectral/docs/guides/2-cli.md#proxying, but it is non-standard behaviour.

Environment

• Spectral: 5.5.0 via CLI
• OS: Observed both on macOS 10.15.6 with node 12.10.0 and gitlab-ci running latest image node:erbium

[philsturgeon]

Want to see if you can take a swing at a PR on this one? It’s a little odd that AJV is making that request, it doesn’t need to be a resolvable URI just a URI, so I’m not sure why it would do that.

[nulltoken]

@dlouzan FWIW, I'm also working behind a corporate proxy at $DAYJOB and never encountered this behavior. So we may indeed need your help to get this weird behavior fixed.

There are two places where that url appears in Spectral:

• spectral/src/functions/schema.ts

  Line 71 in 627a24d

  ajv._refs['http://json-schema.org/schema'] = 'http://json-schema.org/draft-04/schema';

• spectral/src/rulesets/oas/schemas/schema.oas3.json

  Line 3 in 84a8238

  "$schema": "http://json-schema.org/draft-04/schema#",

One way to start troubleshooting the origin of this issue would be to check the code out, run yarn, and tweak the urls in those places with different replacements (eg. http://json-schema.org/draft-04/schemaOne and http://json-schema.org/draft-04/schemaTwo#

then run yarn cli lint {here goes the parameters that trigger the weird behavior you're seeing} while monitoring the outgoing requests, and see which one pops up on the radar.

Note: Would ajv be in cause, it looks like it exposes a loadSchema option. This would require to make the schema function asynchronous (Thanks to @P0lip's #1058 this shouldn't be a major pain) and use the provided resolver to implement the loadSchema core resolving.

[samcrutt9900]

I am experiencing the same issue behind a corporate proxy. Interestingly I checked the code out and attempted to build it. I get a similar issue during the build process where it appears to be attempting to resolve http://json-schema.org/draft-04/schema. The error is:

./scripts/generate-assets.ts
(node:4209) UnhandledPromiseRejectionWarning: ResolverError: Error downloading http://json-schema.org/draft-04/schema 
connect ETIMEDOUT *******:80
    at /home//spectral/node_modules/@apidevtools/json-schema-ref-parser/lib/resolvers/http.js:127:16
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

[PawelKolat]

We are behind the proxy as well also having the same issue. After around 90 seconds delay Lint continues.

[P0lip]

Yup, I can confirm the issue.
I already pushed the fix onto fix/ruleset-proxy however prior to opening a PR, I'll add a few tests making sure everything's in tact.

[PawelKolat]

@P0lip works like a charm just installed 5.9.0. Will run in docker later but am I right to assume the stoplight/spectral image is pointing to the latest? Many thanks for this!!!

[P0lip]

@PawelKolat glad to hear it resolved the issue!
I'm afraid to tell you that Docker images deployment process is currently broken, and therefore no new images have been created since 5.7 or 5.6.
That said, it's not pointing to the latest, but some older version instead.

[PawelKolat]

@P0lip Thanks for that. I'm not clear if the bump after your comment has changed the situation? This will bite us when we go into the local CICD. Is there anything we could do to help to fix the deployment? I guess we can always host our own spectral image with the latest Spectral version

[P0lip]

I'm not clear if the bump after your comment has changed the situation?

Nope, it's the same.
At the moment we're not actively maintaining our docker image.
This is to change in future, but we have no bandwith to address that issue.
You're not the only one that would like to use that docker image, so be sure that we'll get it fixed at some point.

Is there anything we could do to help to fix the deployment

If I am not mistaken dockercloud has introduced some changes, and right now our CI step keeps failing.
I'm not sure what exactly got broken.

Hosting your own spectral image will be probably the right thing to do for now until we get it fixed.