Docusaurus Build/Deploy Broken

[bnewbold]

Building this site on main branch is very slow and not working at all in our deploy environment (render.com).

There are at least two problems, both related to the auto-generated API reference docs:

1. some of the auto-generated MDX files have gotten very large (eg, over 5MB), which makes the build step very slow and frequently hits out of memory (OOM)
2. there is a build error with a specific API endpoint (tools.ozone.setting.removeOptions)
3. build process sometimes fails with:

[ERROR] Client bundle compiled with errors therefore further build is impossible.
Unexpected end of JSON input

I have a branch #236 which fixes the second issue (simple patch), and desperately tries to work around the first issue by removing many of the MDX files (just rips API docs out of the site). This results in a working and faster build locally, but still fails to deploy due to the third issue, which I can't reproduce locally.

I suspect that the first issue might be caused by recursive/nested Lexicon references getting pulled in to generated docs, and an acceptable fix/mitigation might be to truncate the depth of nesting.

The second is probably fixed by just cherry-picking my commit.

The third I have no idea. Might need to create a temporary render.com account to trigger/debug.

As some high-level context, we have a plan to move generated API reference docs out of this repo and website and in to a generic Lexicon documentation site/service for all Lexicions in the ecosystem. So don't want to invest too much time in to that aspect. But will probably continue to use docusaurus for the dev blog, tutorials, and other docs.

Clarifications

I merged #247 which hopefully fixed problem two.

This repo / build process has multiple stages:

• pull Lexicons from another repo and generate an OpenAPI JSON file. this seems to be working fine
• compile OpenAPI JSON to MDX files. this is running fine, but results in huge output files and maybe needs tweaks
• there is a command to "check" the MDX syntax. this basically always succeeds, even if later build fails
• build the actual site. this is the part that fails

[bnewbold]

I just merged #247, which hopefully resolves problem two

[sugyan]

I attempted to reproduce and investigate the problem in an attempt to resolve this. As it turns out, the large mdx file seemed to be correct and the build failure was not reproduced.

Perhaps thanks to the fix in #247, the build succeeded fine in the local environment and on render.com. However, it is slow! It takes about 10 minutes.

I looked into the mdx files generated. Indeed, 5 of them, including app.bsky.feed.getFeed, are huge, over 5MB.
However, I checked the contents and there are no recursive references, only because feedViewPost is deeply nested and large. To be more specific, the postView is quite large, and the feedViewPost is 3 times larger because it references the postView in .post, .reply.root, and .reply.parent, respectively.

I was concerned that docusaurus was slightly out of date. There seem to have been some performance improvements up to the latest 3.6, so I updated it to the latest version and related docusaurus-plugin-openapi-docs and so on.
sugyan@3a9380c
The mdx files are now smaller due to the new data generation format, but the build process still seems to take as long as the current one, or even longer. The results can be viewed without any problem.
https://bsky-docs-246.onrender.com/docs/category/http-reference

I have tried several times to build & deploy using render.com so far and never reproduced the build failures you described in the third one.

In the end, it seems that the current bsky lexicon inevitably slows down the build because of the large postView. On the other hand, if you just want to speed up the build, replacing this would be very effective (although it would lose some detail in the documentation)

--- a/atproto-openapi-types/main.ts
+++ b/atproto-openapi-types/main.ts
@@ -70,6 +70,10 @@ for await (const entry of entries) {
         components.schemas![identifier] = convertArray(id, name, def);
         break;
       case "object":
+        if (id == "app.bsky.feed.defs" && name == "postView") {
+          components.schemas![identifier] = {};
+          break;
+        }
         components.schemas![identifier] = convertObject(id, name, def);
         break;
       case "procedure": {

[bnewbold]

@sugyan thanks! Giving those tips a try.

Also got a tip from render.com support to lock the Node version, specifically to avoid version 23.2: nodejs/node#55826

[bnewbold]

Ok! Progress on this...

• I think the Node v23.2 issue is what was causing the "Unexpected end of JSON input" issue, though I never confirmed with a successful build of just that. added a .node-version to lock to v22.11.0 based on render.com docs
• docusaurus upgrades are great, so merged that. dropped MDX file size a lot, great!
• builds were still really slow (I waited 15min for a render.com build, and 10min on my laptop, before giving up)
• so I merged a modified version of the postView truncation above, to include a warning text. this doesn't seem to come through in the web interface? and isn't great in theory? but It Works for now, with only a 3-4min build time
• also did some other small tidying tweaks

Horray! I'm going to call this 50/50 render.com support and @sugyan. We already (happily) pay for render, so i'll offer the full bounty (https://bsky.app/profile/bnewbold.net/post/3lbhsy3uncs2r) to sugyan.