Alerting rules

rate() over the tick counter is TPS. Two-tier: 27 is "visibly behind", 20 is "players are
feeling this". Both use the same alert name so Alertmanager groups them; a very sick
server fires both at once, which is expected, not a bug.

rate() over the tick counter is TPS. Two-tier: 27 is "visibly behind", 20 is "players are
feeling this". Both use the same alert name so Alertmanager groups them; a very sick
server fires both at once, which is expected, not a bug.

Busy time over budget is the leading indicator, before rate() over the tick counter even
moves: a server can hold 30 TPS while spending most of its budget getting there. Only
meaningful when the engine-probe families are being served (the guarded cast in
Pulse/EngineProbe.cs succeeded) -- if it failed, pulse_server_tick_busy_seconds is simply
absent from /metrics and this rule has no data to evaluate, silently not firing either
way. Absence of this alert is not proof of headroom; check /metrics for the family before
trusting silence here.

Two ways to phrase "ticks are overrunning": count buckets past a fixed 50ms line, or take
a quantile against the live budget gauge. Went with the quantile: it tracks
pulse_server_tick_budget_seconds, so it stays correct if an operator retunes the tick rate
with /serverconfig, where a hardcoded 50ms cutoff would quietly go stale. It also doesn't
depend on 0.05 staying a bucket boundary in Pulse/PulseModSystem.cs's TickBuckets array.
The tradeoff is bucket-interpolation error from only ten finite buckets, which is fine
at warning granularity -- this is "is the tail bad", not a latency SLO.

increase() > 0 over a 10m window already is the debounce; a single stray warning ten
minutes ago still counts, which is the point. Same alert name for both severities, kind
excluded from the warning leg so memory doesn't double-fire under its own critical rule
below.

DieAboveMemoryUsageMb: the "memory" kind is the engine crossing 90% of its ceiling, and
the engine kills itself outright at 100%. That gap is small enough to treat as critical
from the first warning rather than waiting for a repeat.

A single stray error is normal noise (a malformed packet, a one-off mod hiccup); a burst
is one thing failing repeatedly. 5 in 10 minutes is a judgement call, not a measured
value like the tick numbers above -- tune it if your server's baseline error rate differs.

Fatal is the engine's own top log severity, so even one is worth a page, not a threshold
like the burst rule above. It is not the only level that can shut the server down though:
VintagestoryLib.dll's ServerSystemMonitor.OnEntryAdded counts error and fatal entries the
same way toward the engine's own DieAboveErrorCount threshold, so a sustained stream of
errors is also a path to an unplanned restart, covered above by the burst rule.

`up` is Prometheus's own synthetic series, one per scrape target; job name here matches
contrib/grafana/prometheus.yml's `job_name: vintagestory`. Change the job label if yours
scrapes Pulse under a different name.

A queue that fills during a player exploration burst and drains afterwards is the queue
doing its job. 15m of staying above 500 is long enough that draining stopped, not that it
is merely busy.

Needs attribution switched on (README.md's Attribution section: the config block or
`/pulse attribution on`). pulse_mod_tick_share is an observable gauge that disappears from
/metrics the moment attribution stops, whether that is switching it off, a reload with
Enabled false, or the duty cycle giving up on its own, so its absence alone already means
nothing is being measured. The increase(pulse_attribution_ticks_total) term below stays as
a second, independent guard: it only moves when a burst actually completes, which is a
more direct proof of live measurement than the share's mere presence, and it is what would
still catch a future change that let the share report again with nothing behind it.

Two conditions on top of that, both required: pulse_mod_tick_share sums to 1 across every
modid including engine and unattributed, so one mod clearing 0.5 already outweighs
everything else on the server combined, and the load gate reuses PulseTickSaturationHigh's
own 80% busy-over-budget threshold, on the same reasoning, a mod owning half the tick is not
worth paging on while the server is coasting at 20% of budget. 10m is several burst
refreshes at the default duty cycle (10 BurstTicks / 10s IntervalSeconds samples roughly
once every 10 seconds), long enough that this is a sustained hog and not one unlucky sample
landing mid-spike, which the README's "What it cannot see" warns can happen on a spiky
server. The excluded modid is not always a third-party mod either: the base game itself
ships as several mods (survival, essentials, game among them), so on an entity-heavy
vanilla server with nothing else installed this can still name one of those.

"and" defaults to matching on the full label set, and the left side carries modid while
neither the load gate nor the ticks guard does, so without "ignoring(modid)" on both no
side ever matches anything and this alert can never fire. Ignoring modid rather than naming
"on(instance, job)" keeps whatever else a target carries part of the match too, for example
a label a scrape job's relabel_configs adds, instead of silently discarding it the way
naming just instance and job would.