Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for Global Mode #7

Closed
rjchicago opened this issue May 29, 2019 · 14 comments
Closed

Support for Global Mode #7

rjchicago opened this issue May 29, 2019 · 14 comments
Labels
kind/enhancement

Comments

@rjchicago
Copy link

rjchicago commented May 29, 2019 •
edited
Loading

Hi,
I'd like support for global mode services in swarm.
Our use-case is to run a prune service globally.

Prune Service

version: "3.6"
services:
  prune:
    image: docker:18.06
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: ["docker", "system", "prune", "-f"]
    deploy:
      mode: global
      restart_policy:
        condition: none
      update_config:
        parallelism: 1
        delay: 10s
        failure_action: continue
      labels:
        - "swarm.cronjob.enable=true"
        - "swarm.cronjob.schedule=0 */5 * * * *"
        - "swarm.cronjob.skip-running=false"

Swarm-Cronjob Log

Wed, 29 May 2019 14:42:38 CDT INF Starting swarm-cronjob 1.2.0
Wed, 29 May 2019 14:42:38 CDT INF Add cronjob with schedule 0 */5 * * * * service=swarm-cronjob_prune
Wed, 29 May 2019 14:45:00 CDT INF Start job last_status=failed service=swarm-cronjob_prune
2019/05/29 14:45:00 cron: panic running job: runtime error: invalid memory address or nil pointer dereference
goroutine 49 [running]:
github.com/crazy-max/cron.(*Cron).runWithRecovery.func1(0xc00035c150)
	/go/pkg/mod/github.com/crazy-max/[email protected]/cron.go:260 +0x9e
panic(0x8c89c0, 0xd58ad0)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/crazy-max/swarm-cronjob/internal/worker.(*Client).Run(0xc000324340)
	/app/internal/worker/worker.go:35 +0x20d
github.com/crazy-max/cron.(*Cron).runWithRecovery(0xc00035c150, 0x9f2ae0, 0xc000324340)
	/go/pkg/mod/github.com/crazy-max/[email protected]/cron.go:264 +0x57
created by github.com/crazy-max/cron.(*Cron).run
	/go/pkg/mod/github.com/crazy-max/[email protected]/cron.go:298 +0x90a
@crazy-max
Copy link
Owner

Hi @rjchicago,

For now I will fix this nil pointer and I will look to bring support to global mode.

crazy-max added a commit that referenced this issue May 30, 2019
@crazy-max
Fix nil pointer (Issue #7)
e321103
@pascalandy
Copy link

@rjchicago example to execute command: ["docker", "system", "prune", "-f"] should be in the examples.

@DnaMes
Copy link

DnaMes commented Jul 11, 2019

Hello.

Any update supporting global mode?

@crazy-max
Copy link
Owner

Working on it

@crazy-max
Copy link
Owner

crazy-max commented Jul 17, 2019 •
edited
Loading

Ok I think I have something pretty stable :

version: "3.6"
services:
  # Swarm cronjob
  swarm-cronjob:
    image: swarm-cronjob
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - LOG_LEVEL=debug
      - LOG_JSON=false
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

  global:
    image: docker:18.09.7
    command: ["docker", "info", "--format=NodeID: {{ json .Swarm.NodeID }}"]
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      mode: global
      labels:
        - "swarm.cronjob.enable=true"
        - "swarm.cronjob.schedule=0/10 * * * * *"
        - "swarm.cronjob.skip-running=false"
      restart_policy:
        condition: none
root@swarm1:~# docker service logs -f global_swarm-cronjob
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:37:53 UTC INF Starting swarm-cronjob 0.1.2
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:37:53 UTC DBG Creating Docker API client
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:37:53 UTC DBG 1 scheduled services found through labels
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:37:53 UTC INF Add cronjob with schedule 0/10 * * * * * service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:37:53 UTC DBG Starting the cron scheduler
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:37:53 UTC DBG Listening docker events...
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Service task node=swarm3 service_id=nhyhorihg8q9yvhi06i04l965 status_message=finished status_state=complete task_id=kmojt33heev2jqwt8s86fg3h1
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Service task node=swarm2 service_id=nhyhorihg8q9yvhi06i04l965 status_message=finished status_state=complete task_id=za7oam08xzg580cs3x7bfxz61
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Service task node=swarm1 service_id=nhyhorihg8q9yvhi06i04l965 status_message=finished status_state=complete task_id=s8nxgawmyjgjswnuwz1i3r69f
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC INF Start job service=global_global status=
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Event triggered newstate= oldstate= service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Update cronjob with schedule 0/10 * * * * * service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Number of cronjob tasks: 1
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Event triggered newstate=updating oldstate= service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Update cronjob with schedule 0/10 * * * * * service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Number of cronjob tasks: 1
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Event triggered newstate=paused oldstate=updating service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Update cronjob with schedule 0/10 * * * * * service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:00 UTC DBG Number of cronjob tasks: 1
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Service task node=swarm2 service_id=nhyhorihg8q9yvhi06i04l965 status_message=finished status_state=complete task_id=nygpiw9btm47piume821173ox
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Service task node=swarm1 service_id=nhyhorihg8q9yvhi06i04l965 status_message=finished status_state=complete task_id=whar3kafvev2ooq88b1b547ge
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Service task node=swarm3 service_id=nhyhorihg8q9yvhi06i04l965 status_message=finished status_state=complete task_id=kmojt33heev2jqwt8s86fg3h1
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Service task node=swarm2 service_id=nhyhorihg8q9yvhi06i04l965 status_message=finished status_state=complete task_id=za7oam08xzg580cs3x7bfxz61
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Service task node=swarm1 service_id=nhyhorihg8q9yvhi06i04l965 status_message=finished status_state=complete task_id=s8nxgawmyjgjswnuwz1i3r69f
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC INF Start job service=global_global status=paused
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Event triggered newstate=paused oldstate=updating service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Update cronjob with schedule 0/10 * * * * * service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Number of cronjob tasks: 1
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Event triggered newstate=updating oldstate=updating service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Update cronjob with schedule 0/10 * * * * * service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Number of cronjob tasks: 1
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Event triggered newstate=paused oldstate=updating service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Update cronjob with schedule 0/10 * * * * * service=global_global
global_swarm-cronjob.1.qnm16bifh5sn@swarm1    | Wed, 17 Jul 2019 22:38:10 UTC DBG Number of cronjob tasks: 1
^C
root@swarm1:~# docker service logs global_global
global_global.0.s8nxgawmyjgj@swarm1    | NodeID: "xo7d9q27erel2c5z13vkfnr41"
global_global.0.whar3kafvev2@swarm1    | NodeID: "xo7d9q27erel2c5z13vkfnr41"
global_global.0.0of1t1af54rd@swarm1    | NodeID: "xo7d9q27erel2c5z13vkfnr41"
global_global.0.y0cijb25ffa0@swarm2    | NodeID: "crqtogoa7whwjekjzik3vjw7m"
global_global.0.kehkwkyum10o@swarm2    | NodeID: "crqtogoa7whwjekjzik3vjw7m"
global_global.0.m3ef6lyw3i47@swarm2    | NodeID: "crqtogoa7whwjekjzik3vjw7m"
global_global.0.uq2dw8o92s0k@swarm2    | NodeID: "crqtogoa7whwjekjzik3vjw7m"
global_global.0.kmojt33heev2@swarm3    | NodeID: "x5ei52hxxh7ldp7egmblycd0q"
global_global.0.rnajt23swggm@swarm3    | NodeID: "x5ei52hxxh7ldp7egmblycd0q"
global_global.0.sa3r1sbd9xmj@swarm3    | NodeID: "x5ei52hxxh7ldp7egmblycd0q"
global_global.0.20fiz5825sab@swarm3    | NodeID: "x5ei52hxxh7ldp7egmblycd0q"
global_global.0.l5g7cmq3ha0p@swarm3    | NodeID: "x5ei52hxxh7ldp7egmblycd0q"

I keep you in touch for the next iteration.

crazy-max added a commit that referenced this issue Jul 18, 2019
@crazy-max
Add support for global mode (Issue #7)
dd81322
@crazy-max
Copy link
Owner

crazy-max commented Jul 18, 2019 •
edited
Loading

You can try out the latest beta release.

@rjchicago
Copy link
Author

I tested this out and cron jobs work with Global mode correctly. Thanks again, @crazy-max!

For reference, here is the test service I used:

  demo:
    image: docker:18.06
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: ["docker", "info", "--format", "\"{{ .Name }}\""]
    deploy:
      mode: global
      restart_policy:
        condition: none
      update_config:
        failure_action: continue
      labels:
        - "swarm.cronjob.enable=true"
        - "swarm.cronjob.schedule=0 */10 * * * *"
        - "swarm.cronjob.skip-running=false"

@DnaMes
Copy link

DnaMes commented Jul 22, 2019

great job @crazy-max :)

@djmaze
Copy link

djmaze commented Aug 10, 2019 •
edited
Loading

Just to understand the drawback: there seems to be no way to prevent a global service from running once at the moment it is deployed. Right?

@crazy-max
Copy link
Owner

@djmaze Yes that's a drawback of global mode.

@djmaze
Copy link

djmaze commented Sep 23, 2019

Also, I am experiencing strange restart issues with global mode. Disabled that service on my swarm for now. Anyone else experiencing problems?

@crazy-max
Copy link
Owner

@djmaze I have not encountered this kind of problem. Can you give us more explanations? Maybe trough a new issue?

@djmaze
Copy link

djmaze commented Oct 26, 2019

@crazy-max I just created #20.

Apart from that one, I am experiencing a different problem as well, where swarm-cronjob triggers new tasks again and again in a short amount of time. Will see if I can reproduce.

@crazy-max
Copy link
Owner

@djmaze

@crazy-max I just created #20.

Thanks will take a look asap.

Apart from that one, I am experiencing a different problem as well, where swarm-cronjob triggers new tasks again and again in a short amount of time. Will see if I can reproduce.

I also experienced this issue if triggers < 10sec. I will take a look on this too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@djmaze @rjchicago @crazy-max @pascalandy @DnaMes