High availability

Building redundancy: source failover, edge HA and load balancing

Sources drop, servers fail and data centres lose connectivity. A resilient live service is designed so that none of these events reaches the viewer.

Key takeaways

  • Redundancy has layers: source, process, server, site and operations. Each needs its own answer.
  • Source failover must be driven by real quality checks, not only by "is the connection open".
  • Edge nodes with a local database replica keep serving viewers when the main site is unavailable.
  • Monitoring with deduplicated alerts and controlled, rolling updates are part of availability too.

Map your single points of failure

Start by tracing a channel from its source to the viewer and asking, at every step, "what happens if this disappears?" Typical single points of failure in live delivery include:

  • a single contribution feed or satellite receiver per channel;
  • one encoder or streaming process handling a channel;
  • one streaming server carrying all viewers of a region;
  • a single management server and database that every playback request depends on;
  • one data centre, one upstream provider or one DNS name.

Not every point needs full redundancy — but every point needs a conscious decision and a documented recovery path.

Multi-source failover

The most frequent failures happen at the source: a feed freezes, loses audio or disappears. The defence is to configure several sources per channel — for example a primary multicast feed and one or more backups from a different provider or path.

When the primary fails, the platform switches to the next healthy source automatically. Two details make the difference in practice:

  • Auto-return. Once the primary is healthy again, the channel should move back to it, so backups are not left carrying traffic indefinitely.
  • Seamlessness. The output should remain one continuous stream for players, so viewers see a brief glitch at worst rather than a disconnect.

A further step is "take the best source": scoring all sources continuously and keeping the best one on air, which helps when a primary is technically alive but degraded.

What a good watchdog measures

A source that delivers bytes is not necessarily a working source. Effective health checks look at the content of the stream:

CheckWhat it catches
Video present and advancingFrozen pictures, missing video track
Audio presentSilent channels, lost audio track
Frame rateStuttering or dropped frames at the source
Processing speedSources or processes falling behind real time
Bitrate and continuityStarved or interrupted feeds

Thresholds need hysteresis: a check should confirm a problem over a short window before acting, and confirm recovery before switching back, to avoid flapping between sources.

Edge nodes and site-level HA

Source redundancy does not help if the management server itself becomes unreachable. In many architectures every playback request is authorised against a central database; if that site goes down, all viewers are affected even though streaming servers elsewhere are healthy.

Edge nodes solve this by keeping a live replica of the database locally, synchronised over an encrypted tunnel. If the main site fails, edges continue to authenticate viewers and serve channels from their own data. When the main site returns, replication resumes and the edge catches up. Important properties of a good design:

  • replication is monitored, with alerts when an edge falls behind or loses sync;
  • an edge that cannot reach the main site still serves, but does not accept changes that would conflict;
  • resynchronisation after an outage is automatic and visible to operators.

Load balancing and origin shielding

Spreading viewers across several streaming nodes removes the single-server bottleneck and limits the impact of a node failure. Balancing can be based on bandwidth, user count or overall load; sticky sessions keep a viewer on the same node, and only healthy nodes should receive new viewers.

An origin shield — a reverse-proxy layer in front of the core — hides origin addresses from the public, absorbs abusive traffic and adds another layer that can be scaled independently of the core.

Updates, monitoring and alerts

A large share of real-world outages are caused by changes rather than hardware. Updates should roll out one node at a time, with the primary last, and versions should be pinned and approved before they reach production — with a clear way to roll back.

Monitoring closes the loop: server CPU, memory, disk and network; per-channel health and outage history; and alerts that are confirmed and deduplicated, so the NOC receives one meaningful message instead of an alert storm.

Redundancy checklist

  • At least two independent sources for every important channel, with auto-return.
  • Content-aware health checks: video, audio, frame rate, speed.
  • Automatic restart of failed channel processes.
  • More than one streaming node per region, with load balancing.
  • Edge nodes with a replicated database for site-level failures.
  • Origin addresses protected behind a shield layer.
  • Rolling, approved updates with rollback.
  • Daily database backups, including an off-site copy.
  • Alerts routed to the people on duty, with confirmation and deduplication.

How SaurusRex handles it: channels support multiple sources with seamless failover, auto-return and a "take best stream" mode; health checks cover audio, video, frame rate and speed with automatic restarts. Rescue me edge nodes keep a live database replica over an encrypted tunnel with sync monitoring, while load balancing, an origin shield, rolling updates with per-license release channels, daily backups and Telegram alerts complete the picture. See the platform page.

Let's talk

See SaurusRex running on your own network.

Tell us about your channels, sites and subscribers. We'll walk you through a live demo and propose an architecture that fits.