New Self-hosted Version Progress & issues

[EYHN]

UPDATED: selfhost has been released! Please refer to our selfhost tutorial. https://docs.affine.pro/docs/self-host-affine

If you have tried our docker before, don't forget to pull the image again.

If you encounter any problems, please open a new issue.

Many people are interested in the development of our new self-hosted version, so I created this issue to keep everyone updated on its progress.

The current version of docker image has lagged behind our stable branch for a long time. We need to update the new docker-image.

Since we ship many features like payment and account on the current backend server, it is not suitable for common self-hosted use cases. So we need to make these features optional in the new self-hosted version, which requires some work.

Tasks

Preview Give feedback

1. Make Payment Module/Frontend optional #5003
   +0
   
2. make server/client modularity #5057
   app:server mod:payment +2
   
3. Update Docker Image
4. Self-hosted tutorial
5. AFFiNE client supports connecting selfhosted server #2646
   roadmap story type:feat +3
6. feat: support self-host docker build #5506
   app:core app:server mod:dev mod:workspace-impl +4
   
7. fix: improve self-host convenience #5582
   app:server +1
   
8. feat(core): make password sigin default if user has one #5577
   app:core mod:component mod:i18n test +4
   

Archived issues

• Revive and update the Self-Hosted Docker Image + tutorial #4914
• Update the Self-Hosted Docker Images​ OctoBase#457
• Why Selfhosted Version Different Than The Local Version? #3962 (comment)
• Can we get a self-hosted docker image with beta tag? AFFiNE-Community#25

[omarcostahamido]

🏆 a self-host tutorial on synology would be golden 🙌

[PhilippMolitor]

@omarcostahamido I think there is endless potential for integration into other managed systems.
A few integrations that would enable more non-tech-people (or those who know about local systems, but nothing about application development/deployment) to use AFFiNE:

• CasaOS
• Synology DSM
• QNAP QTS
• Unraid

[omarcostahamido]

@PhilippMolitor

Yes synology DSM, that's what I meant. I'm happy to help with it!

[bphenriques]

I have a synology and I prefer have a docker compose setup

[mmospanenko]

Waiting for official Docker image (ARM64). This is required for any server-side projects nowadays :)

[thelazyoxymoron]

Looks like the docker build for selfhost support got merged recently. Excited to try this!

[TheNexter]

Waiting for official Docker image (ARM64). This is required for any server-side projects nowadays :)

The new version drop few hours ago can be run on ARM64 using a raspberry pi 4 😄

[TheNexter]

The new version for self hosted is running but i have this message :

My server not gonna save my note ? or it's a UI bug ?

[Exitialis]

I tried to run this docker-compose on my server too, but i faced with several issues:

1. Problem with docker-compose version - in the compose.yaml version tag didnt mentionted and because of that i faced with error "Unsupported config option for services: 'postgres'", when i added version: "3" at the top of the file, this error gone.
2. The next problem is that in the latest docker-compose, depends-on tag have different type and should be described differently
3. After all fixes for docker-compose, everything was up, but i found, that only frontend version was launched and didnt save docs to the database and in no option in desktop version to configurate my server url in order to use it as cloud, only one option - activate affine cloud
4. The last problem - i didnt find how to setup OAuth to provide login through google, and button from interface just redirect me back on frontend part login form with token, but login was not completed

I can provide any help that you need with self-hosted version, even write instructions on how to run affine on cloud-ocean droplet or drop docker image on TrueNas Scale.

[omarcostahamido]

@Exitialis Thank you for sharing your experience and the workarounds that you found!

no option in desktop version to configurate my server url in order to use it as cloud

I guess that's why #2646 is open?

[fourdim]

The new version for self hosted is running but i have this message :

My server not gonna save my note ? or it's a UI bug ?

The server is not gonna save you note unless the cloud is enabled. All data will be saved in the browser's local storage.
It can be easily erased because of cleaning browsing data.

[benjick]

I tried running the self-host docker-compose myself but whatever should run on port 5555 isn't starting

[Brooooooklyn]

I tried running the self-host docker-compose myself but whatever should run on port 5555 isn't starting

5555 is reserved for Primsa Studio, you can run yarn prisma studio to start Primsa Studio.

[benjick]

I see, thank you @Brooooooklyn!

Then I have some other issue, if I go to http://123.123.123.123:3010 I just see this:

{"compatibility":"0.12.0","message":"AFFiNE 0.12.0 Server","flavor":{"type":"allinone","main":true,"sync":true,"selfhosted":false}}

I can see the logs saying this:

And the public server should be recognized as https://affine.example.com

But If browse to https://affine.example.com I get a cloudflare timeout. What are the steps to troubleshoot here?

[simonkaiser9]

@benjick I had the same issue, which stemmed from me using the docker-compose.yaml from the canary-branch instead of 0.12.0. On canary SERVER_FLAVOR is not set, compared to 0.12.0 where the following is added in the docker-compose file:

• SERVER_FLAVOR=selfhosted

Although my service now comes up as expected and my redis and graph db are both up and running and configured, I still get the prompt, that my data is only stored in the browser instead of in the database:
I cannot find an option to active the db storage explicitly and wonder, what may go wrong during the start up of the service...

[acranglesim]

I see the same issue as above, when I try to sign in - the frontend says "GraphQL empty body"

And the docker log:

[Nest] 113  - 02/23/2024, 10:37:17 AM    WARN [NextAuth] NEXTAUTH_URL

[Nest] 113  - 02/23/2024, 10:37:46 AM   ERROR [ExceptionLogger] Error: requestId-HajWAl7Na5GhwDcaPdLD-: Cannot POST /graphql

[PhilippMolitor]

i can confirm the register.js file is missing. when changing the import in docker-compose env to the dist/prelude.js file, I get the same error as @PovilasID . when fully removing the node options, i get the same error with more stuff to it:

affine_1    | Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import '/app/dist/fundamentals/config' is not supported resolving ES modules imported from /app/dist/prelude.js
affine_1    |     at new NodeError (node:internal/errors:405:5)
affine_1    |     at finalizeResolution (node:internal/modules/esm/resolve:321:11)
affine_1    |     at moduleResolve (node:internal/modules/esm/resolve:980:10)
affine_1    |     at defaultResolve (node:internal/modules/esm/resolve:1193:11)
affine_1    |     at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
affine_1    |     at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
affine_1    |     at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
affine_1    |     at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:39)
affine_1    |     at link (node:internal/modules/esm/module_job:75:36) {
affine_1    |   url: 'file:///app/dist/fundamentals/config',
affine_1    |   code: 'ERR_UNSUPPORTED_DIR_IMPORT'
affine_1    | }
affine_1    |
affine_1    | Node.js v18.19.1
affine_1    |
affine_1    |     at checkExecSyncError (node:child_process:890:11)
affine_1    |     at execSync (node:child_process:962:15)
affine_1    |     at runPredeployScript (file:///app/scripts/self-host-predeploy.js:41:3)
affine_1    |     at file:///app/scripts/self-host-predeploy.js:51:1
affine_1    |     at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
affine_1    |     at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
affine_1    |     at async loadESM (node:internal/process/esm_loader:34:7)
affine_1    |     at async handleMainPromise (node:internal/modules/run_main:106:12) {
affine_1    |   status: 1,
affine_1    |   signal: null,
affine_1    |   output: [
affine_1    |     null,
affine_1    |     Buffer(0) [Uint8Array] [],
affine_1    |     Buffer(932) [Uint8Array] [
affine_1    |       110, 111, 100, 101,  58, 105, 110, 116, 101, 114, 110,  97,
affine_1    |       108,  47, 101, 114, 114, 111, 114, 115,  58,  52,  57,  54,
affine_1    |        10,  32,  32,  32,  32,  69, 114, 114, 111, 114,  67,  97,
affine_1    |       112, 116, 117, 114, 101,  83, 116,  97,  99, 107,  84, 114,
affine_1    |        97,  99, 101,  40, 101, 114, 114,  41,  59,  10,  32,  32,
affine_1    |        32,  32,  94,  10,  10,  69, 114, 114, 111, 114,  32,  91,
affine_1    |        69,  82,  82,  95,  85,  78,  83,  85,  80,  80,  79,  82,
affine_1    |        84,  69,  68,  95,  68,  73,  82,  95,  73,  77,  80,  79,
affine_1    |        82,  84,  93,  58,
affine_1    |       ... 832 more items
affine_1    |     ]
affine_1    |   ],
affine_1    |   pid: 19,
affine_1    |   stdout: Buffer(0) [Uint8Array] [],
affine_1    |   stderr: Buffer(932) [Uint8Array] [
affine_1    |     110, 111, 100, 101,  58, 105, 110, 116, 101, 114, 110,  97,
affine_1    |     108,  47, 101, 114, 114, 111, 114, 115,  58,  52,  57,  54,
affine_1    |      10,  32,  32,  32,  32,  69, 114, 114, 111, 114,  67,  97,
affine_1    |     112, 116, 117, 114, 101,  83, 116,  97,  99, 107,  84, 114,
affine_1    |      97,  99, 101,  40, 101, 114, 114,  41,  59,  10,  32,  32,
affine_1    |      32,  32,  94,  10,  10,  69, 114, 114, 111, 114,  32,  91,
affine_1    |      69,  82,  82,  95,  85,  78,  83,  85,  80,  80,  79,  82,
affine_1    |      84,  69,  68,  95,  68,  73,  82,  95,  73,  77,  80,  79,
affine_1    |      82,  84,  93,  58,
affine_1    |     ... 832 more items
affine_1    |   ]
affine_1    | }

[xuqingkuang]

FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-

I get node errors using the provided unmodified self-hosted compose.yaml

affine_selfhosted  | node:internal/errors:496
affine_selfhosted  |     ErrorCaptureStackTrace(err);
affine_selfhosted  |     ^
affine_selfhosted  |
affine_selfhosted  | Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/
affine_selfhosted  |     at new NodeError (node:internal/errors:405:5)
affine_selfhosted  |     at finalizeResolution (node:internal/modules/esm/resolve:327:11)
affine_selfhosted  |     at moduleResolve (node:internal/modules/esm/resolve:980:10)
affine_selfhosted  |     at defaultResolve (node:internal/modules/esm/resolve:1193:11)
affine_selfhosted  |     at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
affine_selfhosted  |     at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
affine_selfhosted  |     at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
affine_selfhosted  |     at ModuleLoader.import (node:internal/modules/esm/loader:335:34)
affine_selfhosted  |     at node:internal/process/esm_loader:26:84
affine_selfhosted  |     at node:internal/per_context/primordials:544:39 {
affine_selfhosted  |   url: 'file:///app/scripts/register.js',
affine_selfhosted  |   code: 'ERR_MODULE_NOT_FOUND'
affine_selfhosted  | }
affine_selfhosted  |
affine_selfhosted  | Node.js v18.19.1

The PR #5920 was recently merged. The documentation has been updated, but obviously, the new version of stable tag hasn't been published yet.

You just need to restore the environment.

- - NODE_OPTIONS="--import=./scripts/register.js"
+ - NODE_OPTIONS="--es-module-specifier-resolution=node"

[xuqingkuang]

Thanks to the tip from the person above, I managed to get Affine running locally. However, please don't have too high expectations as it is still a fair way off from replacing Notion at this point.

1. The web UI cannot sync data with self-hosted server yet, it still working with local (browser) data only.
2. The newly added GraphQL API seems to haven't been implemented in the self-hosted server yet, it currently returns a 404 error. - It maybe Affine Cloud only.

[peterbuga]

@xuqingkuang thanks for the reply, I managed to get it up'n'running but there's a bit of inconsistency I'm missing.

by default, it spins up the graphql server on port :3010. using comment above #5069 (comment) it drops the graphql server and spins up the web UI on the same port but when I try to login is looking for :3010/graphql that's not available anymore.

in theory i can spin up 2 instances and route the requests to /graphql to server instance using traefik for example, i'll try it later on and see if this a solution 🤞

[PovilasID]

@peterbuga looks like graphql cloud be on secondary port... or two containers? But honestly it sounds like classic case of two services trying claiming the same port. Fore somebody with knowledge of the architecture should not be a very hard fix.
In mean time how do I prevent the container from trying to star web UI? I can probably work out traefik config from there....

[peterbuga]

@PovilasID I did some testing, using 2 containers and routing the /graphql requests to backend using traefik as mentioned above.
I know it's a lot of work from the devs but I do believe it's now yet ready for us to play around, for example the migrations are never ran, the admin user it's not getting created (the db is empty), there's more to it and we should just wait for a while until they get sorted out.

thanks for all the hard work! i'll keep an eye on the project

[PhilippMolitor]

For me, the app under port 3010 responds with the following at /:

{"compatibility":"0.12.0","message":"AFFiNE 0.12.0 Server","flavor":{"type":"allinone","main":true,"sync":true,"selfhosted":false}}

The /graphql path responds with the GraphQL Playground UI.

[acranglesim]

@PhilippMolitor you need to add:
SERVER_FLAVOR=selfhosted to your docker-compile file in the environment section of affine

[fish-uncle]

/graphql is 404
👋 I encountered the same problem, how did you handle it？

[forehalo]

we are recommending you start the selfhost server with docker image distribution(canary/beta/stable) the same as branch(canary/beta/stable) where your compose file located at.

there always some difference in between the compose file and docker image for different branches.

for example, if you are in affine repo at stable branch, you should make sure the compose file are using the image with stable tag. and the same for canary or beta branch.

we've already updated the docs to tell cloning with stable branch:

git clone https://github.com/toeverything/AFFiNE.git --branch stable

BTW, don't forget to pull the latest images

[fabiencharrasse]

Hello, with the last update i can't login to the "connect to AFFINE Cloud" form, i'm redirect every time to xxx:3010/signin?callbackUrl=%2Fauth%2FsignUp
I'm suppose to be able to use AFFINE_ADMIN_EMAIL and AFFINE_ADMIN_PASSWORD in this form, right ?

[omarcostahamido]

[yguo]

I encounter the following errors when using docker compose to do the self-hosting deployment.

"Error: Command failed: yarn predeploy
affine_selfhosted | at checkExecSyncError (node:child_process:890:11)
affine_selfhosted | at execSync (node:child_process:962:15)
affine_selfhosted | at runPredeployScript (file:///app/scripts/self-host-predeploy.js:44:3)
affine_selfhosted | at file:///app/scripts/self-host-predeploy.js:52:1
affine_selfhosted | at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
affine_selfhosted | at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
affine_selfhosted | at async loadESM (node:internal/process/esm_loader:34:7)
affine_selfhosted | at async handleMainPromise (node:internal/modules/run_main:106:12) {
affine_selfhosted | status: 139,"

the branch I clone is 'stable' and the image is also tagged 'stable'

[WarlockFish]

image loading fail

[endquote]

I was able to get it running using the provided instructions.

Is it limited to just the admin user for now? Can I open it to registration for other users, and share pages between users?

[doldremus]

@endquote
I think the best option now is to connect the smtp service.
Example compose.yml environment parameters:

- MAILER_HOST=smtp.gmail.com
- MAILER_PORT=465
- [email protected]
- MAILER_PASSWORD=54745trurt5ttu
- MAILER_SENDER=AFFiNE <[email protected]>
- MAILER_SECURE=true

[PhilippMolitor]

@acranglesim the env setting SERVER_FLAVOR=selfhosted is no longer valid. Omitting it now yields the correct configuration.
@fabiencharrasse I am facing the same issue as you. Each time I enter the email address for login, I get redirected to the same login form over and over again.

My Affine server logs show the following:

affine_1    | [Nest] 121  - 03/03/2024, 11:12:31 AM    WARN [NextAuth] NEXTAUTH_URL

Maybe NextAuth doesn't pick up the correct url from the affine server env config?

[Kerber0ss]

@PhilippMolitor same issue with login via email-address.

[omarcostahamido]

@EYHN consider reopening and help users with their issue?

[markoott]

@PhilippMolitor @Kerber0ss
I initially encountered the same error. The reason for this was that I used a different e-mail address + password for my first configuration. When I recreated the Docker stack, the database and app were probably not recreated. I therefore deleted all folders first and rebuilt them. I was then able to log in with the admin account without any problems.

[Kerber0ss]

@markoott
I recreated stack several times. I try stable, beta, canary, removed old containers, images, install in other instance, but nothing, when type email - and press button - page reload and email again.

[endquote]

@endquote I think the best option now is to connect the smtp service. Example compose.yml environment parameters:

- MAILER_HOST=smtp.gmail.com
- MAILER_PORT=465
- [email protected]
- MAILER_PASSWORD=54745trurt5ttu
- MAILER_SENDER=AFFiNE <[email protected]>
- MAILER_SECURE=true

Thanks @doldremus. I set up these env vars and tried to invite someone else to a workspace. I got this error:

[Nest] 117  - 03/04/2024, 9:01:37 PM    WARN ;5;3m[WorkspaceResolver] failed to send 4eb2131e-e310-4afd-8081-1e1053d34326 invite email to [removed], but successfully revoked permission: Error: Mailer service is not configured.

Here are my settings:

MAILER_HOST=smtp.mycompany.com
MAILER_PASSWORD= # my server does not require a password
MAILER_PORT=587
MAILER_SECURE=true
MAILER_SENDER=AFFiNE <[email protected]>
[email protected]

I've had issues with this no-password thing in other apps, I wonder if that's the issue here.

I tried entering an email into the registration form, but I did not see the same error in the log. Maybe it only happens once.

I could make this a new issue somewhere else since this thread is getting mixed.

[EYHN]

Please open a new issue if you need help

[fabiencharrasse]

For those who have problems with the login form, I posted my solution here: #6007 (comment)