OpScale Tech Solutions

How Do You Know Your Automation Stopped Working? Three Levels of Alerts

An automation can stop without a single error. A real case from a client, why error alerts miss it, and the three levels of alerts that catch it.

A client I worked with had an automation that sent emails to their users. When a record was updated, the email went out by itself.

One day they asked me: why did it stop working?

It had not been working for days. Nobody knew, because nothing looked wrong.

That is how automations often stop: quietly. Catching it takes a few levels of alerts, each one catching what the level before it misses, and a person who owns them.

What actually went wrong?

I opened the logs of the email service and of the automation itself. There was no error anywhere.

Two required fields were empty. Without them, no email could be sent.

The automation was built to check every record before sending. If a required detail was missing, it held the record back. Holding it back was right: no half-finished email went out. What came next was wrong. The automation counted the skip as a success, not as an error.

But a skip leaves no trace. When an email went out, the automation added a history entry and changed the record’s status. When it skipped, it did neither. From the outside, a skipped record looked exactly like a record that was still waiting.

So every run finished without a problem, and sent nothing. It is like a web shop that shows “thank you for your order” when the payment was refused. A record that fails a check should come back as an error that says what is missing.

Another time, the same setup stopped because of a payment problem on the email service account. Different cause, same result: emails stopped, and nobody was told.

Why don’t error alerts catch this?

Because, as far as the automation knew, nothing failed.

Most automation tools can tell you when a step throws an error. That is worth having, but it covers only one way of stopping. In this case no error was ever raised. The automation ran, checked the records, held them back, and reported success.

These are the three ways I see automations stop:

How it stops What you see Does an error alert catch it?
A step fails: wrong login, service down an error in the logs yes, if someone reads it
A check fails but counts as success: missing data, held back quietly nothing no, until the skip is made an error
It never runs: no new input, a paused schedule, an unpaid account nothing often no

Two of the three look like nothing. And “nothing” is also what a quiet day looks like.

What alerts should every automation have?

Three levels. Each one catches something the one before it misses.

Level 1: errors come to you. When a step fails, the automation tells you. You should not have to open a dashboard to find out what ran and what didn’t. You will stop opening it. Everyone does. Send the alert to where you already look: your email, or one Slack or Teams channel if your team works there.

Level 2: a failed check is an error. When a record fails a check, the automation marks it as failed and sends it to the same place, with the reason: “Record 14 failed: a required field is empty.” Then it is something someone can fix the same day, not something found a week later. The other records can still go out.

Level 3: an alert on the result. Ask: what should this automation produce, and how often? An email sent, a record updated, an invoice created. Then set an alert for when that has not happened in the time it normally would. For that client, that would have been: “no email sent in 24 hours on a working day”. It would have fired on the first day, whatever the cause: the empty fields, the payment problem, or something nobody thought of.

Levels 1 and 2 tell you when something went wrong. Level 3 tells you when nothing happened. In the case above, only level 3 would have caught both problems.

Who owns the alerts?

One named person, plus a backup for when they are away. Pick the person who owns the work the automation does.

The owner acknowledges every alert: a reply or a reaction that says “seen, on it”. That way everyone can see that the alert reached someone, and an alert nobody has acknowledged stands out.

Without an owner, the alert inbox or channel becomes a graveyard. Everyone assumes someone else is looking. Nobody is.

What happens when alerts keep firing?

People stop reading them.

If the same alert arrives every day, or every hour, and nobody fixes the cause, everyone learns to ignore it. Then a real one lands, and nobody notices it either. That is worse than having no alert, because everyone believes they are covered.

So every alert must need an action. When one fires, the owner either fixes the cause or changes the alert so it stops firing for something that does not matter. An alert that keeps coming back is a task to fix, not something to read.

How to set it up in one afternoon

  1. Send errors to where you already look: your email, or one Slack or Teams channel.
  2. Name the owner and a backup. They acknowledge every alert.
  3. Turn failed checks into errors: every record that fails a check is marked as failed and sent with the reason.
  4. Write down, in one sentence, what the automation should produce and how often. That is the window for the result alert.
  5. Test it. Empty a required field on purpose. Check that the alert arrives and that the owner acknowledges it.
  6. Once a month, check that the alerts still arrive, and fix or remove any alert that fires without needing action.

Questions owners ask

How do I know if my automation is working?
Look at what it produced, not at whether it ran. If it should send emails, count the emails. An alert that fires when the count stays at zero does this for you.

Is an error alert enough?
No. It catches steps that fail. It misses automations that count a failed check as a success, or never start, and those are the ones that run quietly for days.

Should an automation skip bad data, or report an error?
Report an error. Holding back the bad record is right, but it should count as a failure, not a success, so it shows up and someone fixes it.

Should alerts go to email or a chat channel?
Wherever the owner already looks every day. One place, not both.

Why do people ignore alerts?
Because they arrive too often and nothing happens when they do. Fix the cause or change the alert, so that every alert that arrives means something.

How long can an automation be broken before anyone notices?
As long as nobody looks. In the case above, it was days.

If you have an automation running and you are not sure you would know when it stops, bring it to a free 30-minute discovery call. We’ll look at what it should produce and how you would find out when it doesn’t.

Ask an AI about this post

Does this apply to me?

The AI reads this post, asks you a few questions about your business, then tells you whether the idea fits.

ChatGPTClaudePerplexityGrok

What should I check this week?

A short list of things to look at in your own business, based on this post.

ChatGPTClaudePerplexityGrok

Push back on it

The AI looks for where this post would not hold for a company your size, and what to verify before you act on it.

ChatGPTClaudePerplexityGrok

Do it myself, use a tool, or hire?

Tell the AI about your team, and it weighs up three routes: doing it yourself, buying a tool, or bringing someone in.

ChatGPTClaudePerplexityGrok

The chatbot opens in a new tab, in your own account, with the question filled in. Use one you already have an account with, so it can draw on what it knows about you and your business. What you type there stays between you and the chatbot; I only see that a button was clicked.

All posts