Redundancy and failure behaviour

Overview

Redundancy in EasyUC exists on two levels. With three nodes, the Kubernetes control plane and etcd survive the loss of one node. Longhorn keeps copies of volumes on several nodes.

The application and the data services, on the other hand, each run as a single instance. If a node fails, Kubernetes restarts the affected services on another node. This is restart-based recovery, not seamless failover.

Control plane and etcd quorum

etcd stores the state of the Kubernetes cluster. Every EasyUC node is a server node and therefore an etcd member. The control plane stays operational as long as a majority of these members is available.

With three nodes that majority is two. If one node fails, the other two keep working. Two nodes give you no fault tolerance, because after one failure the majority is gone. The table is under Why three nodes.

Longhorn replicas

Longhorn keeps copies of volumes on several nodes. If a node fails, a replica exists on another node and Longhorn can attach it there. While a node is missing, Longhorn shows affected volumes as degraded.

On a single node the volumes are not redundant. The Manager shows the state of all volumes.

What happens when a node fails

Kubernetes and Longhorn recover the services without manual intervention. This takes time, and until then the affected services are unavailable.

1. 1

   A node fails

   The services on this node are no longer reachable.

2. 2

   Kubernetes detects the failure

   The node is marked as unavailable. This typically takes a few minutes.

3. 3

   Restart on another node

   Kubernetes starts the affected components on a remaining node.

4. 4

   Volume is attached

   Longhorn attaches the volume replica there.

5. 5

   Service available again

   The components are running again. If users, CUCM or the NAT for Microsoft point to the failed node's address, switch them over, see Planning addresses.

Every service as a single instance

Every service runs as a single instance today. This applies to the EasyUC components as well as to MariaDB, Elasticsearch, RabbitMQ, the Manager, acd-media and teams-sbc. If a node fails, Kubernetes restarts the affected services on another node. This takes a few minutes; active calls on that node are ended.

The following does not exist today:

• database replication, for example with Galera
• multiple instances per service
• automatic failover without interruption
• rolling updates. During an update all services restart together, see Updates.
• alerting by e-mail, SNMP or Prometheus

Planning addresses: no shared virtual IP

EasyUC does not provide a shared virtual IP address. Users, CUCM and the NAT for Microsoft point to the address of one node. So that you can switch over after a failure, plan the following:

• DNS names instead of fixed IP addresses for users, agents and CUCM.
• For Teams, a 1:1 NAT that you can move to the node where teams-sbc runs after a failure.
• A web certificate that contains every name under which EasyUC should be reachable, see Certificates.
• Towards CUCM, jtapi and acd-media connect to the CUCM host configured in the AXL settings.

Single-node installations

A single node has no redundancy within the cluster. Protection comes from virtualisation: hypervisor high availability, VM snapshots and backups. What the database backup and the full backup cover is described under Backup and restore.

Monitoring

The following information is available:

• Node status and usage in the Manager
• State of the Longhorn volumes in the Manager
• Component status, live in the Manager
• Backup history in the Manager
• Teams SBC gateway status, reported to the admin interface every 20 seconds