High availability with three nodes and behaviour on node failure

What it is for

With three nodes the cluster control plane survives the failure of one node, and the volumes exist as copies on several nodes. If a node fails, Kubernetes restarts the affected services on another node, and the storage attaches a copy of the volume there.

The application services and the data services (MariaDB, Elasticsearch, RabbitMQ) each run as one instance. Until a service is ready again on another node, it is interrupted. Expect a few minutes. This chapter describes how to plan a three-node cluster and what to do after a failure.

Prerequisites

Three EasyUC appliances in the same network segment, without filtering between them.
A common internal time source for all three nodes.
Enough storage per node for the copies of the volumes.
DNS names for the web interfaces and the CUCM targets that can be moved to another node. There is no shared virtual IP address.
For Teams: a public address or 1:1 NAT to the node the SBC runs on, and a plan for changing that mapping.

Step by step: build a three-node cluster

Commission the first appliance and install EasyUC.
Add the second and the third node one after the other with Knoten hinzufügen. Wait each time until the node is "Bereit".
On each node, set the same time source on the console under NTP Server, one node after the other.
On the Speicher card, check that it reports "alle gesund".
Enter the addresses in DNS, CUCM and the firewall in such a way that they can be moved to another node.
Set up the transfer of backups to an external server. A cluster does not replace a backup.

Step by step: after a node failure

Open the Manager through the address of a running node. If necessary, the Manager restarts on another node.
On the Knoten card, check which node is "Nicht bereit".
On the EasyUC card, check whether all components are ready again.
Change DNS records, the targets in CUCM and, for Teams, the NAT mapping if they point to the failed node.
Fix the cause and start the node again. When it returns, the storage resynchronises the copies.
If the node is lost for good, remove it with Knoten entfernen and add a new appliance.

Fields and options

Fields and options
Field / action	Meaning and use
1 node	No redundancy in the cluster. Protection comes from virtualisation (hypervisor high availability, snapshots) and from backups.
2 nodes	Does not tolerate a failure of the control plane: etcd needs a majority of the nodes.
3 nodes	The control plane tolerates the failure of one node. This is the recommended size for redundancy.
Control plane	Every node is a K3s server with etcd. Pure worker nodes are not offered.
Storage	Longhorn keeps copies of the volumes on several nodes. If a node is missing, the Speicher card shows the affected volumes as "eingeschränkt".
Application and data services	One instance each. Restart on another node after a failure, with an interruption.
Updates	An EasyUC update restarts all services together. The application is unavailable for several minutes, also in a cluster.
Not available	Database replication, several instances per service, failover without interruption and updates without interruption.

Check the result and troubleshoot

The Knoten card shows three nodes with "Bereit", and the Speicher card reports "alle gesund".
If a component stays not ready after a failure, the EasyUC card shows the reason on the component. The Reconcile cluster button triggers a new reconciliation.
If users cannot reach the interface after a failure, the DNS name usually still points to the failed node.
If no Teams calls arrive after a failure, the NAT mapping still points to the old node. The SBC runs on the node where the ACD media service runs.
In the console menu the time source can only be activated with one node or exactly three nodes. The signed offline package on the System-Updates page requires exactly three nodes.
Active calls through the ACD media service or the Teams SBC of the failed node cannot be saved.