Resilient comp authorization: two sites, one writer, automatic failover

A casino floor does not stop for a network outage. If comp authorization stops with it, hosting halts or the rules get bypassed — and the second outcome is the one that costs you.

Comp authorization is not a back-office report you can rerun in the morning. It sits in the moment a host offers value to a patron, on a floor that runs around the clock. That places a hard requirement on the software: it has to stay available, and it has to stay useful even when the things around it fail. This guide sets out how AEGIS, the comp authority engine, is built to stay up, and what it does and does not promise when a site is lost.

The stakes: a floor that never stops

An integrated resort operates continuously. Patrons play at 3am; hosts make offers on a graveyard shift with no head-office desk to call. An authorization system that goes down leaves the floor with two bad options. It can halt hosting — no comp moves until the system returns, which frustrates high-value patrons and pushes staff to improvise. Or staff bypass the rules and issue value outside any governed process, which is the worse outcome: value leaves the building with no authority check and no record. The whole point of authorization is to remove that choice. So the system cannot simply be highly available on paper; it has to keep the floor operating without ever abandoning governance.

Two sites, one writer

AEGIS runs as containerised services at two sites in different countries. One site is the primary and takes every write. The other follows it continuously by streaming replication and is a warm standby: its web tier is already running, and it serves reads and what-if simulations locally, forwarding anything that changes data to the primary. Only one site writes at a time because the audit record is a hash chain, and two writers would fork it. That is a deliberate trade: a single, unbroken chain is worth more to an auditor than a second site that can also write.

Automatic failover

A controller at the standby site watches the primary three ways: its database over a private link, its API over the same link, and its public address. If the primary fails continuously for a set window, 20 seconds by default, the controller powers the old primary off through the hosting provider so it cannot return as a second writer, promotes the standby, and repoints the public address. Operators are alerted on their phones, and an independent dead-man's switch raises the alarm if the controller itself goes silent.

What we have measured, and what we have not

We have switched the writer between the two sites on our production system twice, as planned drills, with no data lost: the audit chain had the same head hash on both sides and verified at both ends. Public disruption in those drills ranged from about 37 seconds to a few minutes while the edge network settled on the new origin. We have not yet measured a real, unplanned failover. Replication is asynchronous, so an unplanned failover can lose the last few in-flight writes, typically the last few tens of milliseconds; the replication lag is shown in the console.

If a phone loses its connection

A device that cannot reach the service cannot ask the decision engine, and AEGIS does not put a copy of that engine on the device to guess with. What it offers instead is a deliberately small exception, off until the operator switches it on: an offline allowance.

• Issued beforehand. While online, a host's phone is prepared and given an allowance for that person on that phone. Its categories and amounts are what the host's own authority covers, worked out by running the live matrix for them, and never more than the operator's offline policy allows. It expires on its own, after a number of hours the operator chooses (never more than 72).
• Limited four ways. By amount per comp, in total, per guest, and by number of grants. Gambling inducements and other high-risk categories can never be granted offline, because the checks they depend on, such as self-exclusion and responsible-gaming limits, need live data.
• Signed on the phone, linked in a chain. Each grant is signed with a key created on the phone that cannot be exported, and linked to the grant before it, so a grant cannot be removed, reordered or altered on its way in without being detected.
• Handed in, then reviewed by a person. The next time the app is open with a connection, the grants are handed in and checked in order. Each is entered in the audit record as a comp recorded after the fact and goes to a reviewer. Anything beyond the allowance, outside its dates, after it was withdrawn, for a guest on an exclusion list, or that the matrix would not now approve, is flagged for that review. It is recorded, not hidden.

The limits, plainly. The phone enforces its allowance and the server checks it again, but a phone whose holder is set on cheating can grant beyond its allowance: the server flags it afterwards and cannot prevent it, so the exposure is what one allowance covers, for the hours it lasts. The phone's clock is not trusted. Exclusion checks run when the grants are handed in, not at the moment of each grant. A lost phone's allowance stays spendable, up to its limits, until an administrator withdraws it. With no allowance, a comp simply waits for a governed decision, and approvers can decide from a phone on any network.

Real-time, even under load

Staying up is not enough if the system is slow when it matters. AEGIS keeps the fast path fast by splitting the work into two deliberately separate engines. A decision engine answers authority checks in about ten milliseconds at the median (about 35 at the 95th percentile, measured on our production host) so a host never waits on a comp. A workflow engine runs the slower, longer-lived work: escalations up the reporting line, reminders, holds. Because the two are separate, an escalation that takes minutes to reach the right manager never blocks a decision that should take milliseconds. Every decision is published the moment it is recorded to a Kafka-compatible event stream, and sealed to a tamper-evident audit trail — hash-chained and anchored into immudb, a second ledger kept apart from the operational database. Speed and proof are not a trade-off here; they run on different engines.

Put together, that is what always-on has to mean for comp authorization. Not a promise on a status page, but an architecture that expects failure — a service lost, a site lost — keeps the audit record whole through it, and states its limits plainly.

Frequently asked questions

What happens to comp authorization if the network drops?

If a terminal cannot reach the service, the decision engine cannot be asked, and AEGIS does not run a copy of it on the device. Where the operator has switched it on, a host's phone can still grant a comp from a small, expiring allowance it was given beforehand: limited to what that host's own authority covers, signed on the phone and linked in a chain, handed in when the app next has a connection, and reviewed by a person afterwards, with anything outside the allowance flagged. Without an allowance, the comp waits for a governed decision, and approvers can decide from a phone on any network.

What happens if one of the two sites fails?

A controller at the standby site detects the failure, powers off the failed primary so it cannot write, promotes the standby, and repoints traffic, typically within a few minutes. Replication is asynchronous, so an unplanned failover can lose the last few in-flight writes; planned switchovers lose none.

How does AEGIS avoid a single point of failure?

AEGIS runs at two sites in different countries with continuous replication and automatic failover. Only one site writes at a time, to protect the audit chain, so a site failure causes a short interruption while the standby takes over, not zero downtime.

Does resilience slow the real-time decision down?

No. AEGIS splits the work into two deliberately separate engines: a decision engine that answers authority checks in about ten milliseconds, and a workflow engine that runs escalations, reminders, and holds. Because they are separate, a long-running escalation never blocks the fast path.