Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] orc mysql cluster all pod role are secondary after delete primary pod #8841

Open
JashBook opened this issue Jan 21, 2025 · 0 comments
Open

[BUG] orc mysql cluster all pod role are secondary after delete primary pod #8841

JashBook opened this issue Jan 21, 2025 · 0 comments
Assignees
@kubeJocker
Labels
kind/bug Something isn't working
Milestone
Release 1.0.0

Comments

@JashBook
Copy link
Collaborator

JashBook commented Jan 21, 2025 •
edited
Loading

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. create orc cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: orcm-tvmcmu
  namespace: default
  annotations:
    "kubeblocks.io/extra-env": '{"ORC_TOPOLOGY_PASSWORD":"orchestrator","ORC_META_PASSWORD":"orchestrator","ORC_META_DATABASE":"orchestrator"}'
spec:
  clusterDef: orchestrator
  topology: raft
  terminationPolicy: WipeOut
  services:
    - name: orchestrator
      componentSelector: orchestrator
      spec:
        ports:
          - name: orc-http
            port: 80
  componentSpecs:
    - name: orchestrator
      serviceVersion: 3.2.6
      env:
        - name: ORC_TOPOLOGY_PASSWORD
          value: orchestrator
        - name: ORC_META_PASSWORD
          value: orchestrator
        - name: ORC_META_DATABASE
          value: orchestrator
      replicas: 3
      resources:
        requests:
          cpu: 500m
          memory: 0.5Gi
        limits:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. create mysql cluster with orc
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: asmysql-tvmcmu
  namespace: default
  
spec:
  clusterDef: mysql
  topology: orc
  terminationPolicy: WipeOut
  componentSpecs:
    - name: mysql
      serviceVersion: 8.0.39
      disableExporter: true
      replicas: 2      
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi      
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
      serviceRefs:
      - name: orchestrator
        namespace: default
        clusterServiceSelector:
          cluster: orcm-tvmcmu
          service:
            component: orchestrator
            service: orchestrator
            port: orc-http

kbcli cluster list-instances asmysql-tvmcmu --namespace default 
    
NAME                     NAMESPACE   CLUSTER          COMPONENT   STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                      CREATED-TIME                 
asmysql-tvmcmu-mysql-0   default     asmysql-tvmcmu   mysql       Running   primary     ReadWrite    cn-shanghai-a   500m / 500m          512Mi / 512Mi           data:20Gi   172.16.0.20/172.16.0.20   Jan 21,2025 17:00 UTC+0800   
asmysql-tvmcmu-mysql-1   default     asmysql-tvmcmu   mysql       Running   secondary   Readonly     cn-shanghai-a   500m / 500m          512Mi / 512Mi           data:20Gi   172.16.0.36/172.16.0.36   Jan 21,2025 17:00 UTC+0800
  1. delete primary pod
kubectl delete pod asmysql-tvmcmu-mysql-0  --namespace default
pod "asmysql-tvmcmu-mysql-0" deleted

kbcli cluster list-instances asmysql-tvmcmu --namespace default
    
NAME                     NAMESPACE   CLUSTER          COMPONENT   STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                      CREATED-TIME                 
asmysql-tvmcmu-mysql-0   default     asmysql-tvmcmu   mysql       Running   secondary   Readonly     cn-shanghai-a   500m / 500m          512Mi / 512Mi           data:20Gi   172.16.0.36/172.16.0.36   Jan 21,2025 17:02 UTC+0800   
asmysql-tvmcmu-mysql-1   default     asmysql-tvmcmu   mysql       Running   primary     ReadWrite    cn-shanghai-a   500m / 500m          512Mi / 512Mi           data:20Gi   172.16.0.36/172.16.0.36   Jan 21,2025 17:00 UTC+0800
  1. delete primary pod again
kubectl delete pod asmysql-tvmcmu-mysql-1 --namespace default
pod "asmysql-tvmcmu-mysql-1" deleted
  1. See error
kubectl get cluster asmysql-tvmcmu 
NAME             CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
asmysql-tvmcmu   mysql                WipeOut              Updating   15m

➜  ~ kbcli cluster list-instances asmysql-tvmcmu --namespace default
NAME                     NAMESPACE   CLUSTER          COMPONENT   STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                      CREATED-TIME                 
asmysql-tvmcmu-mysql-0   default     asmysql-tvmcmu   mysql       Running   secondary   Readonly     cn-shanghai-a   500m / 500m          512Mi / 512Mi           data:21Gi   172.16.0.36/172.16.0.36   Jan 21,2025 17:02 UTC+0800   
asmysql-tvmcmu-mysql-1   default     asmysql-tvmcmu   mysql       Running   secondary   Readonly     cn-shanghai-a   500m / 500m          512Mi / 512Mi           data:21Gi   172.16.0.36/172.16.0.36   Jan 21,2025 17:06 UTC+0800

describe cluster

kubectl describe cluster asmysql-tvmcmu
Name:         asmysql-tvmcmu
Namespace:    default
Labels:       app.kubernetes.io/instance=asmysql-tvmcmu
              clusterdefinition.kubeblocks.io/name=mysql
Annotations:  kubeblocks.io/crd-api-version: apps.kubeblocks.io/v1
API Version:  apps.kubeblocks.io/v1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2025-01-21T09:00:05Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:        5
  Resource Version:  33558711
  UID:               8b7d78b8-3cae-41d9-b414-1ba173e58ce3
Spec:
  Cluster Def:  mysql
  Component Specs:
    Component Def:     mysql-orc-8.0-1.0.0-alpha.0
    Disable Exporter:  true
    Name:              mysql
    Replicas:          2
    Resources:
      Limits:
        Cpu:     500m
        Memory:  512Mi
      Requests:
        Cpu:     500m
        Memory:  512Mi
    Service Refs:
      Cluster Service Selector:
        Cluster:  orcm-tvmcmu
        Service:
          Component:  orchestrator
          Port:       orc-http
          Service:    orchestrator
      Name:           orchestrator
      Namespace:      default
    Service Version:  8.0.39
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  21Gi
  Services:
    Component Selector:  mysql
    Name:                mysql-intranet
    Role Selector:       primary
    Service Name:        mysql-intranet
    Spec:
      Ports:
        Name:          mysql
        Node Port:     30890
        Port:          3306
        Protocol:      TCP
        Target Port:   mysql
      Type:            LoadBalancer
  Termination Policy:  WipeOut
  Topology:            orc
Status:
  Components:
    Mysql:
      Phase:  Updating
  Conditions:
    Last Transition Time:  2025-01-21T09:00:05Z
    Message:               The operator has started the provisioning of Cluster: asmysql-tvmcmu
    Observed Generation:   5
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2025-01-21T09:00:05Z
    Message:               Successfully applied for resources
    Observed Generation:   5
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2025-01-21T09:01:33Z
    Message:               cluster asmysql-tvmcmu is ready
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     5
  Phase:                   Updating
Events:
  Type    Reason                    Age                 From                Message
  ----    ------                    ----                ----                -------
  Normal  ComponentPhaseTransition  15m (x2 over 15m)   cluster-controller  cluster component mysql is Creating
  Normal  ClusterReady              14m                 cluster-controller  cluster asmysql-tvmcmu is ready
  Normal  Running                   14m                 cluster-controller  Cluster: asmysql-tvmcmu is ready, current phase is Running
  Normal  PreCheckSucceed           14m (x4 over 16m)   cluster-controller  The operator has started the provisioning of Cluster: asmysql-tvmcmu
  Normal  ApplyResourcesSucceed     14m (x4 over 16m)   cluster-controller  Successfully applied for resources
  Normal  ComponentPhaseTransition  14m (x13 over 14m)  cluster-controller  cluster component mysql is Running
  Normal  ComponentPhaseTransition  10m (x7 over 14m)   cluster-controller  cluster component mysql is Updating

logs pod mysql

➜  ~ kubectl logs asmysql-tvmcmu-mysql-0 mysql 
2025-01-21 09:02:49+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.39-1.el9 started.
2025-01-21 09:02:49+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2025-01-21 09:02:49+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.39-1.el9 started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
➜  ~ 
➜  ~ kubectl logs asmysql-tvmcmu-mysql-1 mysql
2025-01-21 09:06:28+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.39-1.el9 started.
2025-01-21 09:06:29+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2025-01-21 09:06:29+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.39-1.el9 started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'

logs pod kbagent

➜  ~ kubectl logs asmysql-tvmcmu-mysql-0 kbagent
2025-01-21T09:02:55Z	INFO	create service Action	{"actions": "roleProbe,postProvision,preTerminate,memberLeave,accountProvision"}
2025-01-21T09:02:55Z	INFO	create service Probe	{"probes": "roleProbe"}
2025-01-21T09:02:55Z	INFO	create service Streaming	{"actions": ""}
2025-01-21T09:02:55Z	INFO	service Action started...
2025-01-21T09:02:55Z	INFO	service Probe started...
2025-01-21T09:02:55Z	INFO	service Streaming started...
2025-01-21T09:02:55Z	INFO	starting the HTTP server
2025-01-21T09:02:55Z	INFO	register service to server	{"service": "Action", "method": "POST", "uri": "/v1.0/action"}
2025-01-21T09:02:55Z	INFO	register service to server	{"service": "Probe", "method": "POST", "uri": "/v1.0/probe"}
2025-01-21T09:02:55Z	INFO	register service to server	{"service": "Streaming", "method": "POST", "uri": "/v1.0/streaming"}
2025-01-21T09:02:55Z	INFO	probe started	{"probe": "roleProbe", "config": {"instance":"asmysql-tvmcmu-mysql","action":"roleProbe"}}
2025-01-21T09:02:55Z	INFO	starting the streaming server
2025-01-21T09:03:55Z	INFO	send probe event	{"probe": "roleProbe", "probe": "roleProbe", "code": 0, "output": "secondary", "message": ""}
2025-01-21T09:06:55Z	INFO	send probe event	{"probe": "roleProbe", "probe": "roleProbe", "code": 0, "output": "", "message": ""}
2025-01-21T09:08:55Z	INFO	send probe event	{"probe": "roleProbe", "probe": "roleProbe", "code": 0, "output": "secondary", "message": ""}

➜  ~ kubectl logs asmysql-tvmcmu-mysql-1 kbagent
2025-01-21T09:07:24Z	INFO	create service Action	{"actions": "memberLeave,accountProvision,roleProbe,postProvision,preTerminate"}
2025-01-21T09:07:24Z	INFO	create service Probe	{"probes": "roleProbe"}
2025-01-21T09:07:24Z	INFO	create service Streaming	{"actions": ""}
2025-01-21T09:07:24Z	INFO	service Action started...
2025-01-21T09:07:24Z	INFO	service Probe started...
2025-01-21T09:07:24Z	INFO	service Streaming started...
2025-01-21T09:07:24Z	INFO	starting the HTTP server
2025-01-21T09:07:24Z	INFO	register service to server	{"service": "Action", "method": "POST", "uri": "/v1.0/action"}
2025-01-21T09:07:24Z	INFO	register service to server	{"service": "Probe", "method": "POST", "uri": "/v1.0/probe"}
2025-01-21T09:07:24Z	INFO	register service to server	{"service": "Streaming", "method": "POST", "uri": "/v1.0/streaming"}
2025-01-21T09:07:24Z	INFO	probe started	{"probe": "roleProbe", "config": {"instance":"asmysql-tvmcmu-mysql","action":"roleProbe"}}
2025-01-21T09:07:24Z	INFO	starting the streaming server
2025-01-21T09:08:24Z	INFO	send probe event	{"probe": "roleProbe", "probe": "roleProbe", "code": 0, "output": "secondary", "message": ""}
  1. connect mysql
kubectl exec -it asmysql-tvmcmu-mysql-0 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init)
bash-5.1# mysql -h127.0.0.1 -uroot -P3306 -p'1HzD9V7w78'
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 47
Server version: 8.0.39 MySQL Community Server - GPL

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show master status\G;
*************************** 1. row ***************************
             File: asmysql-tvmcmu-mysql-0-bin.000001
         Position: 157
     Binlog_Do_DB: 
 Binlog_Ignore_DB: 
Executed_Gtid_Set: 
1 row in set (0.00 sec)

ERROR: 
No query specified

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: asmysql-tvmcmu-mysql-mysql-1
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 1
              Master_Log_File: asmysql-tvmcmu-mysql-1-bin.000001
          Read_Master_Log_Pos: 157
               Relay_Log_File: relay-bin.000003
                Relay_Log_Pos: 407
        Relay_Master_Log_File: asmysql-tvmcmu-mysql-1-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 157
              Relay_Log_Space: 861
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2
                  Master_UUID: 250ff968-d7d6-11ef-9963-00163e66f7f8
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 1
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 1
            Network_Namespace: 
1 row in set, 1 warning (0.00 sec)

ERROR: 
No query specified

mysql> 


kubectl exec -it asmysql-tvmcmu-mysql-1 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init)
bash-5.1# mysql -h127.0.0.1 -uroot -P3306 -p'1HzD9V7w78'
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 35
Server version: 8.0.39 MySQL Community Server - GPL

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show master status\G;
*************************** 1. row ***************************
             File: asmysql-tvmcmu-mysql-1-bin.000001
         Position: 157
     Binlog_Do_DB: 
 Binlog_Ignore_DB: 
Executed_Gtid_Set: 
1 row in set (0.00 sec)

ERROR: 
No query specified

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: asmysql-tvmcmu-mysql-mysql-0
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 1
              Master_Log_File: asmysql-tvmcmu-mysql-0-bin.000001
          Read_Master_Log_Pos: 157
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 407
        Relay_Master_Log_File: asmysql-tvmcmu-mysql-0-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 157
              Relay_Log_Space: 611
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
                  Master_UUID: 259cc774-d7d6-11ef-9a0b-00163e6d1c74
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 1
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 1
            Network_Namespace: 
1 row in set, 1 warning (0.00 sec)

ERROR: 
No query specified

mysql>

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Jan 21, 2025
@JashBook JashBook added this to the Release 1.0.0 milestone Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kubeJocker kubeJocker

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
Release 1.0.0
Development

No branches or pull requests
2 participants
@JashBook and others