What happens when the database goes away

The database goes unreachable for two hours. Someone opens a trend, sees a gap, and concludes the data is gone. Usually it is not — but the reasons behind that “usually” are worth knowing, because each one has a setting attached to it.

Which data actually goes through store and forward

Tag history bound for a SQL historian, and transaction group data, do not write straight to the database. They hand data to the store and forward system, which exists to deliver reliably rather than immediately. Data lands first in a memory buffer, moves to a disk cache if it cannot be forwarded quickly enough, and forwards when the database will take it.

There is an important exception on current gateways. Ignition 8.3’s Core Historian writes directly to disk using a write-ahead log and does not use store and forward at all — Inductive Automation states plainly that it bypasses the buffering and retry mechanisms other historian types rely on. If you are on 8.3 with the Core Historian, most of what follows describes a system you are not using, and that is worth establishing before you go looking for a queue that does not exist.

One nuance, because Inductive Automation words this two ways. The provider documentation is flat, as quoted above. The 8.1-to-8.3 upgrade guide is narrower: the Core Historian uses store and forward only if there are pending writes to the database, and skips it entirely otherwise. The practical reading either way is that store and forward is not the mechanism protecting Core Historian data — we went through the rest of what that provider changes in what the Core Historian changes.

How much can the buffer actually hold?

The cache protects against network failures and database failures. What it does not do is absorb an unlimited outage.

On 8.1 the disk cache has a maximum record count, defaulting to 25,000, and Inductive Automation describes it as short-term storage with a recommendation not to push it past 50,000. When the cache is full data backs up into the memory buffer, and when that fills too, records are dropped. The gateway counts them for you under a Dropped metric, which exists precisely because this happens.

On 8.3 those limits moved to Primary and Secondary Store Maintenance settings, and they default to 0, meaning no limit is enforced. That is not the same as safe. In practice the bound becomes disk space rather than a record count, and the maintenance action is itself configurable — it defaults to preventing new data, and can instead evict the oldest. That is the opposite failure mode from 8.1, which drops incoming records once the buffers fill. Worth checking which of the two arrangements your gateway is actually running, because they fail in different directions.

On 8.1 the disk cache is also a setting rather than a certainty. It can be turned off, and the memory buffer cannot quarantine data — so on a gateway with the disk cache disabled, errors during an outage mean data is lost rather than held. That is the configuration in which a database outage stops being an inconvenience and becomes a data loss event.

So the honest version of the reassuring story is conditional: on a gateway with the disk cache enabled and sized for the outage you actually have, a database going away is usually an inconvenience rather than a loss. Both halves of that sentence are load-bearing.

Cached is not the same as available

Here is the part that catches people out, in Inductive Automation’s own words: history bound for the database “… will be cached until the database is available. However, while the data is cached, it will be unavailable to view or analyze on the clients.”

So the trend really does show a gap during the outage. Nothing is lost, but nothing is visible either, because a client asking for history asks the database and the database does not have those rows yet. When the connection recovers and the queue drains, the gap fills in behind you.

This matters operationally more than it sounds. An operator looking at a screen during an outage cannot distinguish “this data is safe in a cache” from “this data never existed”. Both look like an empty chart. Troubleshoot the instrument and you can lose an afternoon to what was a database connection all along; assume it will always fill in and you will eventually be wrong about that too, for the reasons above.

The page that separates the two cases is the gateway’s Store and Forward page — on 8.1 under Status, Connections; on 8.3 under Platform, System. It reports pending, quarantined and dropped records, labelled slightly differently on 8.1. Checking it is usually quicker than working backwards from the instrument.

Quarantine is not inert

Not everything in the queue lands. Data that errors repeatedly on the way to the database, or that cannot be stored because of a configuration problem, is moved into quarantine and removed from the forward queue so other data can pass, which in practice keeps one bad batch from stalling everything behind it.

It is tempting to treat quarantine as a siding where things sit harmlessly. It is not. Inductive Automation’s guidance is explicit that a large volume of quarantined records is worth exporting and deleting so the store and forward engine does not fill up and start dropping records. Quarantine consumes the same finite resource as everything else, so a backlog nobody looks at eventually turns into live data loss somewhere unrelated.

Ignition will not decide for you. The options are to retry the data, delete it, or export it for later, and quarantined data waits until somebody picks one.

Alarm on it rather than remembering to look

The good news is that none of this requires a calendar reminder. Store and forward exposes system tags per database connection, including quarantined, dropped and pending record counts. They are ordinary tags, which means they can carry an alarm like anything else.

A non-zero quarantine count is worth an alarm. So is any movement on the dropped counter, which is the tag that tells you the buffer ran out rather than that something was rejected by the database. Recommending a periodic manual inspection when the platform ships an alarmable tag is doing it the hard way.

The wider point survives all of the detail: store and forward is a buffer, not a backup. It protects data in transit to the database and does that well, within limits it will tell you about if you ask. It is not protecting the data already in the database, and the reliability of the first job has a habit of making people feel covered on the second.

Common questions

If the database goes down, does Ignition lose tag history?

Usually not, provided the buffer is sized for the outage — on 8.1 that also means the disk cache is enabled. Tag history bound for a SQL historian, and transaction groups, go through the store and forward system, which caches data locally and forwards it when the database recovers. Ignition 8.3 Core Historian is an exception: it writes directly to disk using a write-ahead log and does not use store and forward.

Why is there a gap in my trend if the data was cached?

Because cached data has not reached the database yet, and clients read history from the database. Inductive Automation states that while data is cached it will be unavailable to view or analyze on the clients. The gap fills in once the queue drains.

Can the store and forward cache fill up?

Yes. On 8.1 the disk cache has a maximum record count that defaults to 25,000, and Inductive Automation describes it as short term storage. When the cache and the memory buffer are both full, records are dropped, and the gateway reports a dropped count. On 8.3 the equivalent Store Maintenance settings default to 0, meaning no record limit is enforced and the practical bound becomes disk space instead.

What is quarantined data in Ignition?

Data that has errored multiple times while being forwarded, or that could not be stored because of a configuration problem. It is removed from the forward queue so other data keeps flowing, and then waits until someone retries, deletes or exports it. Inductive Automation advises clearing a large quarantine backlog so the engine does not fill up and start dropping records.

Can I alarm on quarantined or dropped records?

Yes. Store and forward exposes system tags per database connection for pending, quarantined and dropped record counts. They are ordinary tags, so an alarm can be placed on a non-zero quarantine count or on movement in the dropped counter.

Does store and forward replace database backups?

No. Store and forward protects data on its way to the database. It does not protect data already in the database, which still needs its own backup.