Threshold re-alerts not working when Acknowledgment Options set to Suspendible Notification

[tddcacti]

Describe the bug

When using a hi/lo threshold which has Acknowledgement Options set to Suspendible Notification, and which has a Re-Alert Cycle defined, no re-alert emails are sent.

It is possible this issue also applies to baseline thresholds and time-based thresholds, and that it might also affect thresholds with Acknowledgment Options set to Persistent Acknowledgment.

To Reproduce

Define a High / Low threshold, with Acknowledgement Options set to Suspendible Notification. For testing purposes, set the re-alert cycle to 5 minutes, the minimum trigger duration to 5 minutes, and set a high and/or low threshold which you can easily cause to breach.

Cause a breach of the threshold.

Expected behavior

An initial alert for the breach should be received (and it is). Every five minutes thereafter, a re-alert ("is still above Threshold") alert should be received. These are not being sent/received.

Screenshots

View of the example Threshold template:

Plugin

• Version: 1.8.2

Root Cause Analysis and Possible Solution

In the database table, thold_data, the field "acknowledgment" can be set to either "on" or no value ("").

When a threshold is in normal state, the "acknowledgment" field remains set to no value.

When a threshold is breached, and if the threshold Acknowledgment Options is set to Suspendible Notification, then the "acknowledgment" field gets set to "on".

If an administrator then acknowledges a breached threshold, the "acknowledgment" gets set back to no value.

If the above interpretation is correct, then re-alert notifications should be sent whenever "acknowledgment" is set to "on", and they should not be sent if "acknowledgment" is set to no value.

See the below screenshot of code (from the current develop branch of thold):

The red box, line 2318, shows that before we send an alert, we are checking whether the alert email address list is not empty, and whether "acknowledgment" is set to no value. If we are checking the "acknowledgment" field, we should be checking to confirm it is set to "on" per the logic above.

However I believe we should not be checking the "acknowledgment" field here at all. We are already checking the "acknowledgment" field in the code inside the green box. If it is set to no value, we are suspending re-alert notifications (suspend_notify = true), and the if statement in line 2308 will evaluate to false, and we then skip the rest of that section--no email can be sent. If "acknowledgment" is set to on, then suspend_notify will be false and we will not skip the rest of the section...in other words, we've already checked "acknowledgment" in the green box, no need to check it again in the red box.

I've tested this by modifying line 2318 from this:

if (trim($alert_emails) != '' && $thold_data['acknowledgment'] == '') {

to this:

if (trim($alert_emails) != '') {

And this does appear to fix the issue.

Note that simply changing the line to this:

if (trim($alert_emails) != '' && $thold_data['acknowledgment'] == 'on') {

Would fix the re-alert issue but would cause the initial alert not to be sent--since $thold_data['acknowledgment'] starts out with no value. We set the "acknowledge" field in the database to "on" in line 2292, but that doesn't feed back into this function until the next cycle.

I believe we may need to fix this throughout thold_functions.php, wherever we have this line:

if (trim($alert_emails) != '' && $thold_data['acknowledgment'] == '') {

And wherever we have this line:

if (trim($warning_emails) != '' && $thold_data['acknowledgment'] == '') {