Chapter 3. Restarting the cluster gracefully

Procedure

1. Turn on the control plane nodes.

   • If you are using the admin.kubeconfig from the cluster installation and the API virtual IP address (VIP) is up, complete the following steps:

     1. Set the KUBECONFIG environment variable to the admin.kubeconfig path.

     2. For each control plane node in the cluster, run the following command:

        Copy to Clipboard Toggle word wrap

        $ oc adm uncordon <node>

   • If you do not have access to your admin.kubeconfig credentials, complete the following steps:

     1. Use SSH to connect to a control plane node.
     2. Copy the localhost-recovery.kubeconfig file to the /root directory.

     3. Use that file to run the following command for each control plane node in the cluster:

        Copy to Clipboard Toggle word wrap

        $ oc adm uncordon <node>

2. Power on any cluster dependencies, such as external storage or an LDAP server.

3. Start all cluster machines.

   Use the appropriate method for your cloud environment to start the machines, for example, from your cloud provider’s web console.

   Wait approximately 10 minutes before continuing to check the status of control plane nodes.

4. Verify that all control plane nodes are ready.

   Copy to Clipboard Toggle word wrap

   $ oc get nodes -l node-role.kubernetes.io/master

   The control plane nodes are ready if the status is Ready, as shown in the following output:

   Copy to Clipboard Toggle word wrap

   NAME                           STATUS   ROLES                  AGE   VERSION
   ip-10-0-168-251.ec2.internal   Ready    control-plane,master   75m   v1.30.3
   ip-10-0-170-223.ec2.internal   Ready    control-plane,master   75m   v1.30.3
   ip-10-0-211-16.ec2.internal    Ready    control-plane,master   75m   v1.30.3

5. If the control plane nodes are not ready, then check whether there are any pending certificate signing requests (CSRs) that must be approved.

   1. Get the list of current CSRs:

      Copy to Clipboard Toggle word wrap

      $ oc get csr

   2. Review the details of a CSR to verify that it is valid:

      Copy to Clipboard Toggle word wrap

      $ oc describe csr <csr_name> 

      1

      1

      <csr_name> is the name of a CSR from the list of current CSRs.

   3. Approve each valid CSR:

      Copy to Clipboard Toggle word wrap

      $ oc adm certificate approve <csr_name>

6. After the control plane nodes are ready, verify that all worker nodes are ready.

   Copy to Clipboard Toggle word wrap

   $ oc get nodes -l node-role.kubernetes.io/worker

   The worker nodes are ready if the status is Ready, as shown in the following output:

   Copy to Clipboard Toggle word wrap

   NAME                           STATUS   ROLES    AGE   VERSION
   ip-10-0-179-95.ec2.internal    Ready    worker   64m   v1.30.3
   ip-10-0-182-134.ec2.internal   Ready    worker   64m   v1.30.3
   ip-10-0-250-100.ec2.internal   Ready    worker   64m   v1.30.3

7. If the worker nodes are not ready, then check whether there are any pending certificate signing requests (CSRs) that must be approved.

   1. Get the list of current CSRs:

      Copy to Clipboard Toggle word wrap

      $ oc get csr

   2. Review the details of a CSR to verify that it is valid:

      Copy to Clipboard Toggle word wrap

      $ oc describe csr <csr_name> 

      1

      1

      <csr_name> is the name of a CSR from the list of current CSRs.

   3. Approve each valid CSR:

      Copy to Clipboard Toggle word wrap

      $ oc adm certificate approve <csr_name>

8. After the control plane and compute nodes are ready, mark all the nodes in the cluster as schedulable by running the following command:

   Copy to Clipboard Toggle word wrap

   $ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm uncordon ${node} ; done

9. Verify that the cluster started properly.

   1. Check that there are no degraded cluster Operators.

      Copy to Clipboard Toggle word wrap

      $ oc get clusteroperators

      Check that there are no cluster Operators with the DEGRADED condition set to True.

      Copy to Clipboard Toggle word wrap

      NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
      authentication                             4.17.0    True        False         False      59m
      cloud-credential                           4.17.0    True        False         False      85m
      cluster-autoscaler                         4.17.0    True        False         False      73m
      config-operator                            4.17.0    True        False         False      73m
      console                                    4.17.0    True        False         False      62m
      csi-snapshot-controller                    4.17.0    True        False         False      66m
      dns                                        4.17.0    True        False         False      76m
      etcd                                       4.17.0    True        False         False      76m
      ...

   2. Check that all nodes are in the Ready state:

      Copy to Clipboard Toggle word wrap

      $ oc get nodes

      Check that the status for all nodes is Ready.

      Copy to Clipboard Toggle word wrap

      NAME                           STATUS   ROLES                  AGE   VERSION
      ip-10-0-168-251.ec2.internal   Ready    control-plane,master   82m   v1.30.3
      ip-10-0-170-223.ec2.internal   Ready    control-plane,master   82m   v1.30.3
      ip-10-0-179-95.ec2.internal    Ready    worker                 70m   v1.30.3
      ip-10-0-182-134.ec2.internal   Ready    worker                 70m   v1.30.3
      ip-10-0-211-16.ec2.internal    Ready    control-plane,master   82m   v1.30.3
      ip-10-0-250-100.ec2.internal   Ready    worker                 69m   v1.30.3

      If the cluster did not start properly, you might need to restore your cluster using an etcd backup.