Support regex in include-exclude labels

[martialblog]

The PR adds regex support for the include-exclude labels.

This PR also moves the include-exclude evaluation so that it checks each alert, not just the alert rule.
With this we can check the labels that are only available for the individual alert.

Fixes #127

[martialblog]

@gianmarco-mameli Ciao, I think this should do it. Feel free to test it and give me feedback.

[gianmarco-mameli]

Hi, let me try it thanks

[gianmarco-mameli]

@martialblog Hi, I think I'm doing something wrong with the regex, do you have a working example? none of my tests seems to work
Thanks

[martialblog]

Hi, in the unittests I'm using these team=data.+ severity=crit.*. Can you give me examples of something that isn't working?

[gianmarco-mameli]

I made 2 tests but I see strange behavior, even without regex

1st test with a ContainerKilled alert (2 containers killed)

With no filters

[CRITICAL] - 2 Alerts: 2 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [ContainerKilled] - Job: [cadvisor] on Instance: [testnode:8088]
 is firing - value: 60.64 - {"alertname":"ContainerKilled","id":"/system.slice
/docker-c9f11108c4d687eeb644db098dac08b1b25592e2dc93b2d54f7bf4e6605a688b.scope
","image":"nodered/node-red:4.1.10","instance":"testnode:8088","job":"cadvisor
","name":"nodered","severity":"warning"}
\_ [CRITICAL] [ContainerKilled] - Job: [cadvisor] on Instance: [testnode:8088]
 is firing - value: 70.64 - {"alertname":"ContainerKilled","id":"/system.slice
/docker-f000eecbd1f920a64433d12f24f733a007a476be7becd6d060442e77c0b1fd44.scope
","image":"eclipse-mosquitto:2.1.2-alpine","instance":"testnode:8088","job":"c
advisor","name":"mosquitto","severity":"warning"}
|total=2 firing=2 pending=0 inactive=0

Working

'/usr/lib/nagios/plugins/check_prometheus' 'alert' '--hostname' 'prometheus.test.domain.it' '--insecure' '--name' 'ContainerKilled' '--port' '443' '--problems' 'false' '--secure' '--timeout' '30' '--include-label' 'instance=testnode:\d+'

[CRITICAL] - 2 Alerts: 2 Firing - 0 Pending - 0 Inactive\_ [CRITICAL] [ContainerKilled] - Job: [cadvisor] on Instance: [testnode:8088] is firing - value: 60.64 - {"alertname":"ContainerKilled","id":"/system.slice/docker-c9f11108c4d687eeb644db098dac08b1b25592e2dc93b2d54f7bf4e6605a688b.scope","image":"nodered/node-red:4.1.10","instance":"testnode:8088","job":"cadvisor","name":"nodered","severity":"warning"}
\_ [CRITICAL] [ContainerKilled] - Job: [cadvisor] on Instance: [testnode:8088] is firing - value: 70.64 - {"alertname":"ContainerKilled","id":"/system.slice/docker-f000eecbd1f920a64433d12f24f733a007a476be7becd6d060442e77c0b1fd44.scope","image":"eclipse-mosquitto:2.1.2-alpine","instance":"testnode:8088","job":"cadvisor","name":"mosquitto","severity":"warning"}
|total=2 firing=2 pending=0 inactive=0

Not working

'/usr/lib/nagios/plugins/check_prometheus' 'alert' '--hostname' 'prometheus.test.domain.it' '--insecure' '--name' 'ContainerKilled' '--port' '443' '--problems' 'false' '--secure' '--timeout' '30' '--include-label' 'name=nodered'

[OK] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive
\_ [OK] No alerts retrieved
|total=0 firing=0 pending=0 inactive=0

2nd test with a TargetDown alert (2 targets down)

No filters ouput

[CRITICAL] - 2 Alerts: 2 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [TargetDown] - Job: [dococd] is firing - value: 100.00 - {"alert
name":"TargetDown","job":"dococd","severity":"warning"}
\_ [CRITICAL] [TargetDown] - Job: [nut-upstower] is firing - value: 100.00 - {
"alertname":"TargetDown","job":"nut-upstower","severity":"warning"}
|total=2 firing=2 pending=0 inactive=0

Not working

'/usr/lib/nagios/plugins/check_prometheus' 'alert' '--hostname' 'prometheus.test.domain.it' '--insecure' '--name' 'TargetDown' '--port' '443' '--problems' 'false' '--secure' '--timeout' '30' '--include-label' 'job=dococd'

[OK] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive
\_ [OK] No alerts retrieved
|total=0 firing=0 pending=0 inactive=0

Not working

'/usr/lib/nagios/plugins/check_prometheus' 'alert' '--hostname' 'prometheus.test.domain.it' '--insecure' '--name' 'TargetDown' '--port' '443' '--problems' 'false' '--secure' '--timeout' '30' '--include-label' 'job=nut.+'

[CRITICAL] - 2 Alerts: 2 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [TargetDown] - Job: [dococd] is firing - value: 100.00 - {"alertname":"TargetDown","job":"dococd","severity":"warning"}
\_ [CRITICAL] [TargetDown] - Job: [nut-upstower] is firing - value: 100.00 - {"alertname":"TargetDown","job":"nut-upstower","severity":"warning"}
|total=2 firing=2 pending=0 inactive=0

[martialblog]

Hi, currently we only use the labels from the alert rule not the actual alert itself, so yes that might be confusing.
I'm looking into it and if we can change that.

[martialblog]

@gianmarco-mameli I moved the include-exclude evaluation so that it checks each alert, not just the alert rule. This should make things a bit clearer and solve the issues you described. Can you confirm that?

Thanks

[gianmarco-mameli]

Ok thanks, i think i can test it tonight and i get back to you with a feedback