Webpack unable to load binary "sdk-napi.win32-x64-msvc.node" when using "@bitwarden/sdk-napi" in a NextJS 13 App Dir API route ( Edge runtime & Node.js runtimes )

[jules-sommer]

Steps To Reproduce

• Step 1: Clone a Next JS 13 App Dir project using create-next-app
  npx create-next-app@latest
• Step 2: Install the SDK and project deps using your package manager of choice, in my case pnpm add @bitwarden/sdk-napi, this should also install the package-lock.json deps.
• Step 3: Create an api route by creating a new api folder in the src/app directory and a subsequent bitwarden folder containing a route.ts file such that our API can be hit at http://localhost:3000/api/bitwarden with a GET request
• Use the following boilerplate code taken from the SDK docs to instantiate our API route with a GET handler that returns a JSON response of our secrets ( created and saved ahead of time in the Secrets Manager dashboard ).

import { BitwardenClient, ClientSettings, DeviceType, LogLevel } from "@bitwarden/sdk-napi";
import { NextResponse, type NextRequest } from 'next/server'

// Note this error will occur if you omit this config to run the API route in a NodeJs runtime
export const config = {
    runtime: 'edge',
}

export default async function GET(req: NextRequest) {

    // Optional settings
    const settings: ClientSettings = {
        apiUrl: "https://api.bitwarden.com",
        identityUrl: "https://identity.bitwarden.com",
        userAgent: "Bitwarden SDK",
        deviceType: DeviceType.SDK,
    };

    const accessToken = process.env.BW_ACCESS_TOKEN;

    const client = new BitwardenClient(settings, LogLevel.Info);

    // Authenticating using a service accounts access token
    const result = await client.loginWithAccessToken(accessToken);
        if (!result.success) {
            throw Error("Authentication failed");
    }

    // List secrets
    const secrets = await client.secrets().list();

    return NextResponse.json(secrets, { status: 200 });

}

• Step 4: Run the dev server by running pnpm run dev ( or npm / yarn, etc depending on your local environment )
• Step 5: Open a web browser and navigate to http://localhost:3000/api/bitwarden thus sending the GET request to the server we've just created to be greeted by the error I am receiving on my local reproduction which pasted below:

./node_modules/.pnpm/@bitwarden+sdk-napi-win32-x64-msvc@0.2.1/node_modules/@bitwarden/sdk-napi-win32-x64-msvc/sdk-napi.win32-x64-msvc.node
Module parse failed: Unexpected character '�' (1:2)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)

Expected Result

So obviously the secrets manager is in beta and I was excited to test it out in a production environment, in this case a NextJs 13 web app, I wanted to created an API route ( which would obviously be behind authentication in production, however is not in this reproduction for brevity ) that would return a JSON serialized list of the secrets in my BW Secrets Manager project.

Curious to hear what you have to say about this, I am aware I might be using the SDK out of its intended environment, though NextJS api routes are NodeJS runtime and I would expect this to work, though obviously my webpack bundler is having some trouble with the SDK module.

Thanks for taking the time, and I appreciate any feedback!

Actual Result

Again, I am getting this build error when pulling that SDK into my NextJS application.

./node_modules/.pnpm/@bitwarden+sdk-napi-win32-x64-msvc@0.2.1/node_modules/@bitwarden/sdk-napi-win32-x64-msvc/sdk-napi.win32-x64-msvc.node
Module parse failed: Unexpected character '�' (1:2)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)

Screenshots or Videos

Additional Context

No response

Operating System

Windows

Operating System Version

No response

Build Version

@bitwarden/[email protected]

Issue Tracking Info

• I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

[Hinton]

Hi @jules-sommer,

The Napi SDK (Node-API) is only compatible with node.js. By default Webpack attempts to bundle external dependencies, I'm not well versed with how next.js works, but I've previously used externals https://webpack.js.org/configuration/externals/ to inform webpack to not bundle it, and instead depend on the node_modules version.

vercel/next.js#17807 suggests using __non_webpack_require__ which also seems to avoid the webpack bundling.

[jules-sommer]

Hi @jules-sommer,

The Napi SDK (Node-API) is only compatible with node.js. By default Webpack attempts to bundle external dependencies, I'm not well versed with how next.js works, but I've previously used externals https://webpack.js.org/configuration/externals/ to inform webpack to not bundle it, and instead depend on the node_modules version.

vercel/next.js#17807 suggests using __non_webpack_require__ which also seems to avoid the webpack bundling.

Oh, nice - thank you; that's a fairly clean solution to avoid having webpack bundle it leaving me dependent on the NPM module alone - still will be interesting to see if it runs well in a NextJs API route, but without that bundling I see no reason why it wouldn't run as API routes are NodeJs runtime by default. I will play around with that and get back to you, thanks for the quick response!

[jules-sommer]

object
{
  BitwardenClient: [Getter],
  SecretsClient: [Getter],
  DeviceType: [Getter],
  Convert: [Getter]
}
- error Error: Authentication failed
    at BwHandler (webpack-internal:///(sc_server)/./app/api/bitwarden/route.ts:23:15)
    at async eval (webpack-internal:///(sc_server)/./node_modules/.pnpm/[email protected]_@[email protected][email protected][email protected]/node_modules/next/dist/server/future/route-modules/app-route/module.js:253:37

I managed to get to this point with __non_webpack_require__ so it seems that the module does run in NextJs API endpoints it might just require some more fiddling to get the client running.

I am going to be away from my PC but I will do that fiddling later today and update this issue in case anyone else has the same use case and wants to get the SDK running, it's possible my environment variable was incorrectly set.

[jules-sommer]

{
  data: null,
  errorMessage: 'Internal error: Failed to parse IdentityTokenResponse',
  success: false
}

I am now getting this error with a valid access token ( triple-checked ) and the error throws at client.loginWithAccessToken funtion. Any thoughts?

[Hinton]

That typically means the response from the bitwarden server is malformed, it could be due to quite a few reasons. We recently merged a change in #109 that improves the error message for this and it will be included in the next release.

In the meanwhile I believe you should be able to download the latest CLI build from the master branch and attempt to execute the command through it to receive a better error message. Last build is https://github.com/bitwarden/sdk/actions/runs/5588705919.

[Hinton]

@jules-sommer Version 0.3.0 which just got released to npm should provide better error handling for the IdentityTokenResponse error.

[jules-sommer]

@jules-sommer Version 0.3.0 which just got released to npm should provide better error handling for the IdentityTokenResponse error.

Hey, thanks for keeping me updated, below is the stack trace that I am getting when I hit my API route. I think the __non_webpack_require__ is still not properly importing the requested modules, it's obviously getting to the point of making the client.loginWithAccessToken call, as we see the error that is caught there output to the console, but that was only after I manually set LogLevel.Info to int(2) since that was throwing a type error of undefined, indicating that it's not getting either the types or something despite my attempted import of them ( I can import types using ES module syntax without getting that .node loader issue with webpack ).

I tried a few other methods of getting webpack to load the .node binary and or ignore it to no avail. Very frustrating.

Anyways this is the error after doing a pnpm update on your SDK to 0.3.0.

- error Error: Authentication failed
    at getSecrets (webpack-internal:///(rsc)/./app/api/bitwarden/route.ts:22:15)
    at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/[email protected]_@[email protected][email protected]_react@18._us5fmivjwwbv462g44f5nimhai/node_modules/next/dist/server/future/route-modules/app-route/module.js:253:37)
thread '<unnamed>' panicked at 'Builder::init should not be called after logger initialized: SetLoggerError(())', C:\Users\runneradmin\.cargo\registry\src\jackfan.us.kg-1ecc6299db9ec823\env_logger-0.10.0\src\lib.rs:816:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceTypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11576:11)
    at async invokeRequest (C:\Users\User\Documents\GitHub\rcsrc-canada\node_modules\.pnpm\[email protected]_@[email protected][email protected]_react@18._us5fmivjwwbv462g44f5nimhai\node_modules\next\dist\server\lib\server-ipc\invoke-request.js:34:23)

[Hinton]

That definitely sounds like a bug on our end. I believe it's caused by our NAPI integration initializing the logger multiple times. Could you try creating a single instance of the BitwardenClient and persist it between requests?

I'll break out a new issue for this.

[jules-sommer]

That definitely sounds like a bug on our end. I believe it's caused by our NAPI integration initializing the logger multiple times. Could you try creating a single instance of the BitwardenClient and persist it between requests?

I'll break out a new issue for this.

Thanks so much for helping me out with this, I appreciate it - and yes, I suppose the way my code is written currently, a new instance of the client is created every time the API is hit, I hadn't thought of that being an issue. Come to think of it though, I already solve this problem elsewhere in my codebase for the database, I just have a utility function check if an instance has been initialized and if not create one otherwise return the existing so I don't have thousands of open connections to my DB in prod. I will try to apply that same logic to the BW client and see what happens!

[jules-sommer]

So with a global client instance this is the error I am getting now, it seems to have changed the behaviour slightly in the direction of functionality but I've not quite got it yet. It looks like that was the issue with the logger though, so something to look into for sure, I can post the code I used but basically I just conditionally initialized a global instance of the BW client if one had not already been created and exported it to be consumed by the API route.

{
  data: null,
  errorMessage: 'Internal error: Failed to parse IdentityTokenResponse',     
  success: false
}
- error Error: Invalid value for key "secrets" on Command. Expected an optional object but got {"list":{}}

[Hinton]

The last error message indicates that the command is missing an argument. list expects an organization id to be provided. After modifying your example slightly and providing an org id it seems to work on my machine.

Can you try updating your code with:

const secrets = await client.secrets().list("org-id");

[jules-sommer]

The last error message indicates that the command is missing an argument. list expects an organization id to be provided. After modifying your example slightly and providing an org id it seems to work on my machine.

Can you try updating your code with:

const secrets = await client.secrets().list("org-id");

Okay, I didn't catch that in the docs, but for the life of me I can't figure out what organization id you're referring to - again, searching through the docs I only see an OID referenced for SSO, but nothing with regards to this SDK or the secrets manager in general, this is totally just my ignorance in using this SDK and is probably why I can't authenticate; but I've searched around my API keys and such in the BW web vault dashboard, as well as the docs, for about an hour and couldn't find the specific token you're referencing. When I setup the SDK the only .env variable I set was my access token.

I've been working on another project so haven't had a minute to check in here, hence my late reply, so sorry about that as well, and forgive my ignorance and lack of familiarity with your SDK.

Hopefully this is all that's preventing my code from running and I will be sorted once I find this other identifier I am after.

[Hinton]

We haven't yet properly documented the node api unfortunately.

The organization id is visible in the url. When going to the secrets manager dashboard you have a url like https://vault.bitwarden.com/#/sm/00000000-0000-0000-0000-000000000000 the number after sm is the organization id.

[jules-sommer]

We haven't yet properly documented the node api unfortunately.

The organization id is visible in the url. When going to the secrets manager dashboard you have a url like https://vault.bitwarden.com/#/sm/00000000-0000-0000-0000-000000000000 the number after sm is the organization id.

I would love to help with that if you're allowing open-source contributors, given I now have some experience, but thank you, I will try it and update the issue if that finally gets me going with the SDK!!

[Hinton]

Hey @jules-sommer,

I'm closing this issue since I believe we resolved the root cause of your issues. Hopefully the secret manager team can get some documentation published soon.

Feel free to reach out to support in case you run into any other issues.