Skip to content

Commit

Permalink
refactor: call system gc once instead of after each job cleanup itera…
Browse files Browse the repository at this point in the history
…tion

When cleaning up a namespace, we previously:

- purged the ignorable jobs
- called system gc
- called namespace deletion

This was performed for each namespace, causing more load on Nomad as it called GC potentially dozens of times.

Instead, we now purged all the ignorable jobs in each namespace, _then_ call system gc, and finally delete all the underlying namespaces. This should cause less stress on the nomad cluster.

Additionally, if no jobs are found for a deletable namespace, skip the deletion logic completely (which makes the log output a bit cleaner).
  • Loading branch information
josegonzalez committed Sep 30, 2022
1 parent 0af84f4 commit ca7a445
Showing 1 changed file with 15 additions and 6 deletions.
21 changes: 15 additions & 6 deletions command/namespace/gc.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ func GC(c *cli.Context, logger *log.Logger) error {
return err
}

if len(jobs) == 0 {
continue
}

if c.Bool("dry") {
logger.Infof("Skipping deletion of %d jobs in region/namespace %s/%s because dry flag was provided", len(jobs), region, namespace.Name)
continue
Expand All @@ -75,16 +79,21 @@ func GC(c *cli.Context, logger *log.Logger) error {
}
logger.Infof("Job '%s' in region/namespace '%s/%s' successfully deleted", job.ID, region, namespace.Name)
}
}
}

if err := nomadClient.System().GarbageCollect(); err != nil {
return fmt.Errorf("error running garbage collection: %w", err)
}
if !c.Bool("dry") {
logger.Infof("executing garbage collection")
if err := nomadClient.System().GarbageCollect(); err != nil {
return fmt.Errorf("error running garbage collection: %w", err)
}

if err := nomadClient.System().ReconcileSummaries(); err != nil {
return fmt.Errorf("error reconciling summaries: %w", err)
}
if err := nomadClient.System().ReconcileSummaries(); err != nil {
return fmt.Errorf("error reconciling summaries: %w", err)
}
}

for _, namespace := range deletableNamespaces {
if c.Bool("dry") {
logger.Infof("Skipping deletion of namespace %s because dry flag was provided", namespace.Name)
continue
Expand Down

0 comments on commit ca7a445

Please sign in to comment.