Kōami
Back to Resources
Operations6 min read

The Day the Fibre Line Was Cut

K

Kōami

Editorial team

Share this article

A JCB widening the road outside the gate goes through the fibre at ten past eleven on a Monday. Nobody inside the building knows that yet. What they know is that the registration screen has stopped responding, the OPD display is frozen on token eighty-four, and the pharmacy counter shows a spinner that will never resolve. Four minutes later there are twenty-odd people at the front desk and one clerk telling all of them the same thing: system down hai.

Fibre cuts, dead switches, failed transformers, upgrades that go sideways. Ask any hospital IT manager in a tier-2 city for their last three outages and they will give you dates. What varies is not whether this happens. It is whether anybody rehearsed for it.

The protocol exists. Nobody has read it.

Most NABH-accredited hospitals have a downtime policy. It is a document, written for the assessment, produced when asked for, initialled and filed. The distance between owning that document and owning a reflex is the whole story of a bad Monday.

A protocol that works fits on a laminated card at every counter and answers four questions inside two minutes:

  • Who declares downtime? Not IT. The duty manager, because IT is busy diagnosing and someone must call it while they do.
  • How is it announced? Phone tree, WhatsApp group, a runner with legs. Keep all three, because the one you rely on may be down.
  • Which forms come out, and from where? A sealed downtime box at registration, casualty, every ward, pharmacy and lab.
  • How long is it expected to last, and who updates that estimate every thirty minutes? Staff can cope with two hours of paper. They cannot cope with not knowing.

Registration and billing have to keep moving

The instinct is to stop registering patients. It is the wrong instinct. The queue does not stop, it moves outside into the sun and doubles.

Paper registration needs three things prepared long in advance. Pre-printed, pre-numbered downtime slips, because the pre-numbering is what makes reconciliation possible later. A rule for the MRN: existing patients keep the number on their old card, new patients take a temporary identifier from a reserved block, never a guessed number and never a series invented at the counter. And a printed tariff sheet for the sixty commonest items, because otherwise the counter starts estimating prices, and estimates become disputes at discharge.

Billing moves to a pre-numbered receipt book, tallied against cash each shift. The TPA does not care that your fibre is cut, so keep a paper pre-authorisation pack ready with the insurer helplines, and note the time of every call and the name at the other end. When the query lands three weeks later, that time-stamped scribble is the entire defence.

The orders that go quiet

Lab and imaging are where downtime does real clinical damage, because the failure is silent. A consultant writes an order on paper and assumes it has travelled. Nothing travels. The sample is never collected. Nobody notices for six hours.

During downtime, an order becomes a physical object a human being carries. Duplicate requisition slips, one to the lab with the sample and one retained in the file. A logbook at lab reception recording time in, time out, and whoever carried the report back to the ward. Radiology keeps working, since a CT does not need the HIS to scan, but images sit on the modality and the report is handwritten. Critical values go to the ward sister by phone, read back, recorded at both ends.

During downtime, your audit trail is a person with a pen. Treat it as seriously as the database you have lost.

Back-capture is where the errors are born

The link comes back at half past four and everyone relaxes. That is precisely the wrong moment. The outage cost you five hours. A careless back-capture will cost you a month of quiet damage, and it fails in familiar ways. Two clerks enter the same slip and the patient is billed twice. A slip disappears between counter and data-entry desk, and a lakh of legitimate billing is never captured. Temporary identifiers get merged into the wrong permanent record, the worst outcome here, because a bad merge puts one patient's allergy into another patient's chart. Timestamps get keyed as the time of typing rather than of the event, so a discharge summary claims a patient was seen at 5pm when they were seen at noon.

Back-capture has to be run, not merely done:

  • One named owner per department on the day itself, not a vague instruction to catch up.
  • Reconcile against the pre-numbered slips. If the pad ran 1041 to 1096 and only fifty-two records exist, four patients are unaccounted for and somebody goes looking.
  • Enter the event time, not the entry time, and flag every record from that window as retrospective, so an audit finds the gap explained rather than discovers it.
  • Merge temporary identifiers deliberately, by one trained person, with a second pair of eyes on anything ambiguous.

Kōami can flag retrospective entries and hold a temporary-to-permanent merge as a reviewable step, but reconciliation stays a human discipline. No software knows about a slip that never reached the desk.

Redundancy is a set of choices, not a purchase

Nobody buys five-nines uptime on a hospital budget. The question is which failure you are protecting against, and at what price.

  • Two ISPs on genuinely different physical paths. Two providers sharing one conduit under one road are a single connection with two invoices, which is exactly what the JCB found.
  • A 4G or 5G failover router that switches without a human. Test the SIM quarterly. It is usually the SIM.
  • Local caching or an on-premise node, so a WAN outage still leaves the ward holding today's admitted list, active orders and drug charts. Kōami can be deployed either way, but that is an architecture decision taken long before you need it, never on the afternoon of the outage.
  • UPS and generator cover on the network switches, not only the servers. A running server behind a dead switch is a dead system.

Rehearse it, or you do not have it

A written evacuation plan has never evacuated anybody, which is why fire drills exist. Downtime is identical, and almost nobody drills it.

Take a low-volume window, a Wednesday afternoon rather than a Monday morning, and go offline for ninety minutes with department heads informed and counters not. Then watch. The ward's downtime box will be locked and the key will be with someone on leave. The forms will carry a field for a charge you dropped last year, and clerks hired since the last outage will never have seen a receipt book. Two of the three escalation numbers will belong to people who resigned. Then run the drill's back-capture properly, because that is the half everyone skips and the half that costs money.

A hospital that has drilled downtime does not have a better day when the fibre is cut. It has an ordinary one. Busier, slower, more paper, but ordinary. Patients notice the longer queue. What they do not notice is a hospital losing control, because it has not.

Found this useful? Pass it on to someone on your team.

Share this article