wish: add unix domain socket support

[nahuel]

Add the option to connect to a unix socket, like https://github.com/request/request#unix-domain-sockets

[TimothyGu]

Do browsers support Unix domain sockets? We try to align with browsers, not always other Node.js libraries.

[nahuel]

I think browsers doesn't support them, but node supports serving HTTP from a unix domain socket (https://nodejs.org/api/http.html#http_server_listen_path_callback ), making requests to them (https://nodejs.org/api/http.html#http_http_request_options_callback ) and many utilities like curl also support them.

[TimothyGu]

I'm going to close this for the same reason we don't support file: URLs. Please search through other issues for the rationale on why that is the case, though we should perhaps document that somewhere more permanent.

[masaeedu]

@TimothyGu This is unfortunate, because it makes it impossible to use node-fetch to communicate with a local docker instance (without doing some portmap gymnastics that permissions on a CI server may or may not allow).

[caub]

in theory this could work:

const http = require('http');
const fetch = require('node-fetch');

const agent = new http.Agent();
agent.createConnection({ path: '/var/run/docker.sock' });
fetch('http:/v1.37/containers/json', { agent }).then(console.log, console.error)

We get TypeError: Only absolute URLs are supported

and with

fetch('http:/v1.37/containers/json', { agent }).then(console.log, console.error)

FetchError: request to http://v1.37/containers/json failed, reason: getaddrinfo ENOTFOUND v1.37 v1.37:80

even if curl works fine (from the doc https://docs.docker.com/develop/sdk/examples/#list-all-images)

> curl --unix-socket /var/run/docker.sock http:/v1.37/containers/json
[]

Sharing this nice solution using http.get (thanks @jaawerth)

const http = require('http');

const get = o => new Promise((resolve, reject) => {
  const req = http.get(o);
  req.once('error', reject);
  req.once('response', async res => {
    const bufs = [];
    for await (const buf of res) bufs.push(buf);
    resolve(JSON.parse(Buffer.concat(bufs)));
  });
});

get({ path: '/v1.37/containers/json', socketPath: '/var/run/docker.sock' }).then(console.log, console.error);

[masaeedu]

@caub We can do some stuff with an HTTP client directly, but the whole point of having the fetch API is to have a higher level abstraction. If we have to write get/post/etc ourselves, and deal with feeding things buffer by buffer to and fro, there's no point to using node-fetch at all.

[Qix-]

Please reconsider this. Not having this functionality in node-fetch means having two implementations for HTTP handling.

Nobody is expecting a complete 1:1 parity with the w3 module - meaning, I don't think anyone would care or even notice if an extra option was available to specify the connection should be made over a domain socket.

Even if there was require('node-fetch').unixSocket or something like that - that would help immensely.

---

Please note that file:// and unix domain socket handling are different cases and I don't think it's really accurate to compare them.

file:// is a URI scheme that doesn't dictate transport at all. Unix domain sockets are a form of transportation, and govern an entirely separate part of the request pipeline.

Saying you won't support domain sockets for the same reasons as file:// doesn't make much sense.

[caub]

I use this small http/https.request wrapper https://github.com/caub/fetchu/blob/master/fetchu-node.js

[bitinn]

While we have no plan to fix this, can people try creating an Agent with path option and pass it to node-fetch? It's not in the nodejs doc, but Agent should pass path to net.createConnection and cause the request to use unix socket.

ref:

• https://nodejs.org/api/http.html#http_agent_createconnection_options_callback
• Doc: Agent ctor options are not fully documented, such as lookup nodejs/node#24098

[cdoublev]

EDIT: it was already tested in a comment above and it doesn't work. Sorry for the duplicate.

@bitinn, Could you please give me some hints about how to test your providden solution? I might not have understand how to test it the right way...

My test setup is a NodeJS app listening on /var/run/nodejs/nodejs.sock proxyied by Nginx listening on port 8000 inside a Vagrant guest. Port 8000 and Vagrant are non important details but I'm mentioning them just in case.

// test.js (inside the guest)
const fetch = require('node-fetch')
const { Agent } = require('http')

const agent = new Agent({ path: '/var/run/nodejs/nodejs.sock' })

fetch('http://0.0.0.0:8000/api/categories/index.json', { agent })
    .then(res => res.json())
    .then(console.log)
    .catch(console.log)
   // Output : { ...parsed JSON } and an entry is appended to Nginx access log.

fetch('/api/categories/index.json', { agent })
    .then(res => res.json())
    .then(console.log)
    .catch(console.log)
   // Output : Error: only absolute urls are supported

[bitinn]

Your first example appear to work (network-wise). The error you got seems to suggest the JSON response was invalid JSON, you should post the full error. (From my phone)

…

On Nov 19, 2018, at 18:14, Guillaume ***@***.***> wrote: @bitinn, Could you please give me some hints about how to test your providden solution? I might not have understand how to test it the right way... My test setup is a NodeJS app listening on /var/run/nodejs/nodejs.sock proxyied by Nginx listening on port 8000 inside a Vagrant guest. Port 8000 and Vagrant are non important details but I'm mentioning them just in case. // test.js (inside the guest) const fetch = require('node-fetch') const { Agent } = require('http') const agent = new Agent({ path: '/var/run/nodejs/nodejs.sock' }) fetch('http://0.0.0.0:8000/api/categories/index.json', { agent }) .then(res => res.json()) .then(console.log) .catch(console.log) // Output : { ...parsed JSON } and an entry is appended to Nginx access log. fetch('/api/categories/index.json', { agent }) .then(res => res.json()) .then(console.log) .catch(console.log) // Output : Error: only absolute urls are supported — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[cdoublev]

EDIT: it was already tested in a comment above and it doesn't work. Sorry for the duplicate.

Sorry, I was very brief on my last comment and I realize that I could simplify the test. But just to not be confusing and to answer you, the first example is working if we mean by that that the expected data is returned but it's returned from NodeJS via Nginx, not directly from NodeJS. And the second example outputs an error from node-fetch.

Simplified test:

// server.js

require('http').createServer((_, response) => {
    response.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' })
    response.end('Hello')
}).listen('/var/run/nodejs/nodejs.sock')

// Just to prove that the server created above is correctly handling request via unix socket
require('http').get({ socketPath: '/var/run/nodejs/nodejs.sock' }, response => {
    response.setEncoding('utf8')
    let data = ''
    response.on('data', chunk => { data += chunk })
    response.on('end', () => console.log(data))
})

When running node server.js, Hello is correctly outputed.
Now the tests with node-fetch (Nginx service has been stopped):

const fetch = require('node-fetch')
const { Agent } = require('http')

const agent = new Agent({ path: '/var/run/nodejs/nodejs.sock' })

fetch('http://0.0.0.0', { agent })
    .then(res => res.text())
    .then(console.log)
    .catch(console.log)
   // Output : { FetchError: request to http://0.0.0.0 failed, reason: connect ECONNREFUSED 0.0.0.0:80

fetch('/', { agent })
    .then(res => res.text())
    .then(console.log)
    .catch(console.log)
   // Output : Error: only absolute urls are supported at /var/www/node-fetch-socket/node_modules/node-fetch/index.js:54:10

[bitinn]

@cdoublev I guess the problem comes down to whether you can construct a "valid" absolute url that can go pass url.parse while satisfying the http(s) scheme requirement.

Due to lack of context and lack of Fetch Spec definition on non-http protocol, we only support http(s) scheme and absolute url.

I originally thought it might work if you construct the url in a way that can resolve properly. But maybe I am too optimistic about this.

Seeing how non-standard unix domain socket url are (I still don't know if http+unix:// or unix: or http://unix: are valid, every tool only seem to support a fraction of them), I really don't think we will invest more time to support this feature.

[cdoublev]

Ok, I'm fine with that. Thank you for your quick feed back.

[caub]

just a note, I tried that with docker a few months ago, and it doesn't work:

var fetch = require('node-fetch');
var agent = new http.Agent({ path: '/var/run/docker.sock' });
fetch('http:/v1.37/containers/json', {agent}).then(console.log); // .then is never invoked

[zkochan]

Seems like this fork of node-fetch supports unix domain sockets.

I am switching pnpm from request/got to node-fetch. Unfortunatly, pnpm needs also the unix socket support, so I'll have to use a fork of node-fetch or create my own 😢

[garkin]

This was a major blow for my homebrew docker util. 🤷‍♂️
Any hints for an alternative mainstream promise-based ajax library with unix://?

[ericvicenti]

I thought node-fetch was a high-level http library, but now I need an alternative because of the transport layer?? I'm still talking HTTP!

For a bit of context, I want to use unix sockets for tests so they can run in parallel without port conflicts. My http server is happily capable of attaching to a unix socket.. why can't the client??

I can switch to got for my node.js usage, but this seems like a silly reason to ditch node-fetch...

[Qix-]

@ericvicenti I commented above a few years ago about this, and since they have a hard stance against it, I switched all of my projects to got and haven't looked back. I recommend you do, too.

[joshbalfour]

looks like i'll have to do the same, i found this which looks like it'll make the transition easier https://www.npmjs.com/package/got-fetch

[gwn]

Do browsers support Unix domain sockets? We try to align with browsers, not always other Node.js libraries.

Unix domain sockets are not a "Node.js library". It's a feature of the unix platform.

I think the proper goal ought to be aligning with the fetch interface provided by the browsers; not blindly align with "browser the platform". The point of this library is being platform-agnostic; but that doesn't necessarily justify leaving out support for platform specific features. What developers are looking for is one-fetch-library-to-rule-them-all. If I have to switch libraries whenever I need a platform specific feature, what's the point?

[gwn]

I mistook node-fetch with platform agnostic fetch libraries so my comments are a bit misaligned. But the argument still holds.

Let's make a correction:

The point of this library is having the same interface as the browsers' fetch; but that doesn't necessarily justify leaving out support for server-side features.

[elmerbulthuis]

Do browsers support Unix domain sockets? We try to align with browsers, not always other Node.js libraries.

Supporting unix sockets is not aligning with other libraries, it's aligning with the environment. And for node-fetch that is node.js.

I think not supporting unix socket via the agent is unexpected (btw also unnecessary) behavior and can therefore be considered a bug.

It's not that hard to fix. I would love to do it, but i guess there has been some effort already that will not be merged.

[elmerbulthuis]

So in the end it seems that this is a bug (as in unexpected behavior). The problem is, however not with node-fetch!

Here we can read about the http.Agent. The documentation says

options in socket.connect() are also supported.

Check out those options here. It says:

For IPC connections, available options are:

    path <string> Required. Path the client should connect to. See Identifying paths for IPC connections. If provided, the TCP-specific options above are ignored.

And yes, an IPC socket is a unix socket

So we can configure the agent to talk to a unix socket, just like explained earlier in this post. Only problem is that this does not work. But why?

Well, one reason is that the path option is set to null in the constructor of the Agent. The path option will always be null!

At some point, while making the actual request, the agent will call the createConnection function, via it's createSocket method.

Now in this method look at this piece of code:

  options = { ...options, ...this.options };
  if (options.socketPath)
    options.path = options.socketPath;

Yes, the path property that was so brutally set to null in the constructor is set again! But with the value of the socketPath property of the options that we pass to the Agent (this.options).

So when we pass the unix path as a socketPath option to the agent, everything works as expected, so something like:

        const agent = new http.Agent({
            socketPath: "/var/run/docker.sock",
        });

After all this was a bug! But not in node-fetch. The node.js documentation is a bit off here.

I am very happy that can keep using node-fetch for everything http!

[dalechyn]

So when we pass the unix path as a socketPath option to the agent, everything works as expected, so something like

I'm trying to connect to snap socket located in /run/snapd.socket, and that solution doesn't really work. I get Failed to fetch net::ERR:REFUSED_CONNECTION error, tried different variants such as http://localhost/ and http://0.0.0.0.
Also I'm using TypeScript and looks like it lacks all http.Agent methods.
I'd love seeing support for unix sockets since Node.JS is slowly getting to PC appications.

[Bessonov]

Not tested yet, but the native node's fetch could support unix socket and there are some pointers in the code. And no documentation ATM.

[AndASM]

Not tested yet, but the native node's fetch could support unix socket and there are some pointers in the code. And no documentation ATM.

Here, have an unofficial example. It's still fetch, so it's still sending http requests. I'm using it for Apollo GraphQL over a unix socket.

// Node 18 uses undici fetch, so you don't have to import fetch. But you can.
import { Agent } from "undici";

// The URL must parse as a valid url, but the hostname doesn't matter for the connection.
// I've included a dummy value of... dummy. But you could use real host names if proxying, or route off the host name.
fetch("http://dummy/path/in/http/request/over/socket", {
  dispatcher: new Agent({
    connect: {
      socketPath: "/tmp/test-unix-socket"
    }
  }) 
});