v0.23.0 image tag available on Docker but not on project

[guyguy333]

Bug description

I'm not sure it's a bug but I've been notified that a new image tag has been released on Docker Hub : 0.23.0
However, latest version is always an alpha on GitHub project. Is it expected to have this image published as a stable tag ?

https://hub.docker.com/layers/headscale/headscale/0.23.0/images/sha256-9e95e62d1c59a51bb5688737dd38b5d149cea44502a7c76ff668c07d9e8ff4c9?context=explore

[CoolnsX]

The problem is it's not working, my watchtower updated my headscale, and it started going in restart loop, so I read logs and log says

• "headscale serve" not found try headscale --help

so I figured that they set the entrypoint directly to headscale so I removed "headscale" from "headscale serve" in "command"
So now it's giving error -- 2024-02-15T20:52:14Z FTL invalid database type "", must be sqlite, sqlite3 or postgres

I have stopped the headscale container for now, I will do it later some time

[epichub]

struggled with the same sudden change in entrypoint, this post helped, found out the db config part had changed for the next problem:

database:
  type: sqlite

  # SQLite config
  sqlite:
    path: /var/lib/headscale/db.sqlite

vs the old

db_type: sqlite3
db_path: /etc/headscale/db.sqlite

(from https://raw.githubusercontent.com/juanfont/headscale/main/config-example.yaml)

[muchachagrande]

The "latest" tag is now pointing to 0.23 alpha 4

[nikolas-digitalBabylon]

The "latest" tag is now pointing to 0.23 alpha 4

Is this really how it should be? According to docker hub the tags 0.23, 0.23-alpha4, and unstable all have the same digest. Is that what is expected, or is it some issue with a ci/cd pipeline?

[CoolnsX]

I think CI/CD pipeline issue here

[TehNomad]

The problem is it's not working, my watchtower updated my headscale, and it started going in restart loop, so I read logs and log says

* "headscale serve" not found try headscale --help

so I figured that they set the entrypoint directly to headscale so I removed "headscale" from "headscale serve" in "command" So now it's giving error -- 2024-02-15T20:52:14Z FTL invalid database type "", must be sqlite, sqlite3 or postgres

I have stopped the headscale container for now, I will do it later some time

struggled with the same sudden change in entrypoint, this post helped, found out the db config part had changed for the next problem:

database:
  type: sqlite

  # SQLite config
  sqlite:
    path: /var/lib/headscale/db.sqlite

vs the old

db_type: sqlite3
db_path: /etc/headscale/db.sqlite

(from https://raw.githubusercontent.com/juanfont/headscale/main/config-example.yaml)

These two changes solved almost all the issues, but I also had the following error:

headscale | 2024-02-16T08:11:05Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/server.go:26 > Error starting server error="failed to set up gRPC socket: listen unix /var/run/headscale/headscale.sock: bind: no such file or directory"

Which I solved by adding a new volume to my docker-compose.yml file:

volumes:
  - ...
  - /var/run/headscale

Now, headscale is running fine in a docker container.

[kradalby]

Sorry about this, the tags was an issue and #1763 should address it for the future.

Will make amendments to the Changelog for other things.

[troycarpenter]

This bit me yesterday as well. Docker compose pulled the alpha release and nothing worked after that. I did try by making the config.yaml changes, but the database changes were enough to have all my machines nodes disappear and request re-registration.

I changed the image tag to 0.22.3, reverted my config changes and everything is back again.

A little more warning next time, please.

[epichub]

I would like to add that i love headscale - and tailscale. I think its a great, and more importantly very useful, piece of software. And I am absolutely sure that this was a good old mistake. And as a result, that if the developers knew it was going to happen - they would have given us that heads up :)

[kradalby]

Thanks.

This was an accident, we did review the new tags as part of the pipeline, but not all the impact occured to us. As we often state, we do not use docker ourselves so we dont grasp all the ways people use it.

In this case there were several problems:

• Database changes that was suppose to be aliased did not work (fixed in make database configuration change breaking #1766)
• The entrypoint change of the docker container was not understood and documented (fixed in rework docker tags #1763)
• latest was replaced by alpha (fixed in rework docker tags #1763)
• major, minor and patch was pushed as part of prerelease (fixed in rework docker tags #1763)

The last one is very unfortunate as it looks safe to upgrade, and I am extra sorry for that.

When it comes to latest, I would personally recommend everyone to avoid that for all their containers as you loose a lot of control of your own infrastructure and are very vulnerable to both mistakes like this, but also supply chain attacks.

[artis3n]

Thanks for this update and quick resolution! I wanted to confirm - have the docker hub tags been fixed? #1763 (comment)

https://hub.docker.com/r/headscale/headscale/tags still seems to show the 0.23 and latest tags pointing to the bad SHAs

[kradalby]

We have released alpha5 now, the tags are no longer written incorrectly, we have also done the following:

Deleted 0, 0.23, 0.23.0.
Untagged latest, we will not replace this until we do another stable to avoid accidents.

[CoolnsX]

so, I think I am going to stick with alpha5 since I have updated my config.yml file and made changes in my docker-compose too, and I will wait until 0.23 is released.

[nikolas-digitalBabylon]

We have released alpha5 now, the tags are no longer written incorrectly, we have also done the following:

Deleted 0, 0.23, 0.23.0. Untagged latest, we will not replace this until we do another stable to avoid accidents.

Thanks for all this. If I get it right, previously deployed containers will still be deployed on an "alpha" version.
Is a downgrade possible to the last stable release? I assume the alpha5 still might have bugs / security issues and a downgrade makes sense.

[kradalby]

Downgrade would require rolling back to a database backup, and changing back your config. If you have a recent database then yes.

[nikolas-digitalBabylon]

I cannot tell what kind of issues the alpha5 might present to those running it, but I would suggest that a downgrade path to the actual stable release is communicated.

Thanks again!

[guyguy333]

We have released alpha5 now, the tags are no longer written incorrectly, we have also done the following:

Deleted 0, 0.23, 0.23.0. Untagged latest, we will not replace this until we do another stable to avoid accidents.

Thanks @kradalby

On my side, I can still see 0.23.0 tag on Docker Hub https://hub.docker.com/layers/headscale/headscale/0.23.0/images/sha256-9e95e62d1c59a51bb5688737dd38b5d149cea44502a7c76ff668c07d9e8ff4c9?context=explore

Also https://hub.docker.com/layers/headscale/headscale/0.23.0-debug/images/sha256-9084af941716122990f3c43ad944b6d0cac95d1e7ae48f5b7e425ff92f2a6fbf?context=explore

[juanfont]

@guyguy333 should be now fixed :)