Hocus Agent stuck on restarting because "/run/netns/vms": File exists

[ajh123]

🐛 Bug description

The Hocus Agent docker container looks like it is stuck on restarting. I don't know why or how. This is probably why I can't create a project.

sudo docker ps says

CONTAINER ID   IMAGE                           COMMAND                  CREATED        STATUS                          PORTS
                          NAMES
d6592b3c7030   hocus-local-hocus-agent         "docker-entrypoint.s…"   42 hours ago   Restarting (1) 26 seconds ago
                          hocus-local-hocus-agent-1
7ca69898d314   hocus-local-hocus-ui            "docker-entrypoint.s…"   42 hours ago   Up 42 minutes (healthy)         0.0.0.0:3000->3000/tcp, :::3000->3000/tcp             hocus-local-hocus-ui-1
c2f9644557e6   hocus-local-keycloak            "/opt/keycloak/bin/k…"   42 hours ago   Up 42 minutes (healthy)         8443/tcp, 0.0.0.0:4200->8080/tcp, :::4200->8080/tcp   hocus-local-keycloak-1
d726836c7758   temporalio/auto-setup:1.20.0    "/etc/temporal/entry…"   42 hours ago   Up 42 minutes (healthy)         6933-6935/tcp, 6939/tcp, 7233-7235/tcp, 7239/tcp      hocus-local-temporal-1
888680bf811c   postgres:15.2-alpine            "docker-entrypoint.s…"   42 hours ago   Up 42 minutes (healthy)         0.0.0.0:5432->5432/tcp, :::5432->5432/tcp             hocus-local-db-1
2fc3ebf54eae   cloudflare/cloudflared:latest   "cloudflared --no-au…"   42 hours ago   Up 42 minutes
                          cloudflare

sudo docker logs -f hocus-local-hocus-agent-1 says

Starting OpenBSD Secure Shell server: sshd.
net.ipv4.conf.vpeer-vms.proxy_arp = 1
net.ipv6.conf.vpeer-vms.disable_ipv6 = 1
net.ipv4.conf.vpeer-ssh-vms.proxy_arp = 1
net.ipv6.conf.vpeer-ssh-vms.disable_ipv6 = 1
net.ipv4.conf.veth-vms.proxy_arp = 1
net.ipv6.conf.veth-vms.disable_ipv6 = 1
2023-09-28T17:54:41.748Z [INFO] Found TCMU: Target Engine Core ConfigFS Infrastructure v5.0 on Linux/x86_64 on 5.15.0-84-generic

2023-09-28T17:54:41.750Z [INFO] Found TCM LOOP: TCM Loopback Fabric module v2.1-rc2

Not setting up any projects
2023-09-28T17:54:41.783Z [INFO] Creating worker {
  options: {
    namespace: 'default',
    identity: '1@d6592b3c7030',
    shutdownGraceTime: 0,
    maxConcurrentActivityTaskExecutions: 100,
    maxConcurrentLocalActivityExecutions: 100,
    enableNonLocalActivities: true,
    maxConcurrentWorkflowTaskExecutions: 100,
    stickyQueueScheduleToStartTimeout: '10s',
    maxHeartbeatThrottleInterval: '60s',
    defaultHeartbeatThrottleInterval: '30s',
    isolateExecutionTimeout: '5s',
    workflowThreadPoolSize: 8,
    maxCachedWorkflows: 261,
    enableSDKTracing: false,
    showStackTraceSources: false,
    reuseV8Context: false,
    debugMode: false,
    interceptors: { activityInbound: [Array], workflowModules: [Array] },
    sinks: { defaultWorkerLogger: [Object] },
    connection: NativeConnection {
      nativeClient: [External: 5ccede0],
      referenceHolders: Set(0) {}
    },
    workflowBundle: { codePath: '/app/workflow-bundle.js' },
    activities: {
      fetchRepository: [AsyncFunction (anonymous)],
      buildfs: [Function (anonymous)],
      checkoutAndInspect: [Function (anonymous)],
      prebuild: [Function (anonymous)],
      changePrebuildEventStatus: [AsyncFunction (anonymous)],
      createWorkspace: [Function (anonymous)],
      startWorkspace: [AsyncFunction (anonymous)],
      stopWorkspace: [AsyncFunction (anonymous)],
      getOrCreateBuildfsEvents: [AsyncFunction (anonymous)],
      createPrebuildImages: [Function (anonymous)],
      getWorkspaceInstanceStatus: [AsyncFunction (anonymous)],
      addProjectAndRepository: [AsyncFunction (anonymous)],
      getRepositoryProjects: [AsyncFunction (anonymous)],
      updateGitBranchesAndObjects: [AsyncFunction (anonymous)],
      getDefaultBranch: [AsyncFunction (anonymous)],
      deleteWorkspace: [AsyncFunction (anonymous)],
      getOrCreatePrebuildEvents: [AsyncFunction (anonymous)],
      initPrebuildEvents: [AsyncFunction (anonymous)],
      getPrebuildEvents: [AsyncFunction (anonymous)],
      reservePrebuildEvent: [AsyncFunction (anonymous)],
      removePrebuildEventReservation: [AsyncFunction (anonymous)],
      waitForPrebuildEventReservations: [Function (anonymous)],
      markPrebuildEventAsArchived: [AsyncFunction (anonymous)],
      deleteLocalPrebuildEventFiles: [Function (anonymous)],
      deleteRemovablePrebuildEvents: [AsyncFunction (anonymous)],
      getArchivablePrebuildEvents: [AsyncFunction (anonymous)],
      saveGitRepoConnectionStatus: [AsyncFunction (anonymous)],
      cleanUpAfterPrebuildError: [AsyncFunction (anonymous)],
      cleanUpWorkspaceInstanceLocal: [Function (anonymous)],
      signalWithStartLockWorkflow: [AsyncFunction (anonymous)],
      getWorkflowStatus: [AsyncFunction (anonymous)],
      cleanUpWorkspaceInstanceDb: [Function (anonymous)],
      createPrebuildEvent: [Function (anonymous)],
      signalWithStartWaitWorkflow: [AsyncFunction (anonymous)],
      getProjectsRepository: [AsyncFunction (anonymous)],
      removeContentWithPrefix: [Function (anonymous)]
    },
    taskQueue: 'main',
    dataConverter: { payloadConverterPath: '/app/data-converter.js' },
    shutdownGraceTimeMs: 0,
    shutdownForceTimeMs: undefined,
    stickyQueueScheduleToStartTimeoutMs: 10000,
    isolateExecutionTimeoutMs: 5000,
    maxHeartbeatThrottleIntervalMs: 60000,
    defaultHeartbeatThrottleIntervalMs: 30000,
    loadedDataConverter: {
      payloadConverter: [CompositePayloadConverter2],
      failureConverter: [DefaultFailureConverter],
      payloadCodecs: []
    }
  }
}
2023-09-28T17:54:42.530510Z  INFO temporal_sdk_core::worker: Initializing worker task_queue=main namespace=default
Starting worker...
2023-09-28T17:54:42.531Z [INFO] Worker state changed { state: 'RUNNING' }
Trying to load kernel module target_core_user
Trying to load kernel module tcm_loop
SCSI scan mode not in sync mode, enabling sync mode
Cannot create namespace file "/run/netns/vms": File exists
Cannot create namespace file "/run/netns/vms": File exists
# the last line continues for like another 50 times.

Environment

• Hocus version (commit hash) git log says f04d83f9592a0cf719701b6a9e990b5e59616c9c
• OS and its version
  OS is an Ubuntu VM inside Hyper V running on Windows 11 Pro (Version: 21H2) (Build: 22000.2416)
  • uname -a says Linux ubuntu-server 5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • cat /etc/os-release says

    PRETTY_NAME="Ubuntu 22.04.3 LTS"
    NAME="Ubuntu"
    VERSION_ID="22.04"
    VERSION="22.04.3 LTS (Jammy Jellyfish)"
    VERSION_CODENAME=jammy
    ID=ubuntu
    ID_LIKE=debian
    HOME_URL="https://www.ubuntu.com/"
    SUPPORT_URL="https://help.ubuntu.com/"
    BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
    PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
    UBUNTU_CODENAME=jammy
    

• Any other environment information that may help
  sudo docker --version says Docker version 20.10.24, build 297e128

[RDG88]

I have the same problem.