OTLP export

OTLP is the push counterpart to the scrape endpoint: instead of waiting for Prometheus to come and read /metrics, the server sends its metrics to a collector on a timer, in the wire format every major observability backend accepts. Grafana Cloud, Honeycomb, Datadog, New Relic and an otel-collector you run yourself all take the same payload.

It ships as a second mod, pulseotlp_x.x.x.zip, and both zips go in Mods/. The base mod stays a single dll with no dependencies; the OTLP one carries the OpenTelemetry SDK and its Microsoft.Extensions.* fan-out, eighteen dlls in all. That split is not tidiness. The game's mod loader puts every root-level dll of every mod into one shared assembly context with no version arbitration, so a bundled dependency is a collision risk against every other mod on the server, and a server that does not push metrics should not be paying it.

The two mods share no code. Pulse.Otlp.dll has no reference to Pulse.dll; it subscribes to the meter named Pulse.Server, which is all System.Diagnostics.Metrics needs, and modinfo.json declares the dependency so the loader guarantees the base mod is there first.

On first boot it writes ModConfig/pulse-otlp.json:

{
  "Enabled": true,
  "Endpoint": "http://localhost:4318",
  "Protocol": "http/protobuf",
  "Headers": {},
  "IntervalSeconds": 60,
  "IncludeRuntimeMetrics": true,
  "ServiceName": "vintagestory"
}

Those defaults suit a collector running on the same host. Endpoint is the base address, without a signal path: Pulse appends /v1/metrics for http/protobuf and leaves it alone for grpc, where the exporter appends its own service path. Protocol takes the two names the OTLP specification defines, http/protobuf and grpc; anything else logs a warning and falls back to http/protobuf rather than leaving you with no export at all. IncludeRuntimeMetrics adds the System.Runtime meter to what gets pushed, and it is separate from the base mod's RuntimeMetrics flag, so you can serve the dotnet_* families locally and not ship them, or the other way round.

ServiceName sets the service.name resource attribute, which is how a backend receiving metrics from more than one server tells them apart: grouping, filtering and dashboard variables are usually keyed off it. The OTEL_SERVICE_NAME environment variable, the ecosystem's standard override, takes precedence over this key when it is set. For a stable instance label, pair OTEL_SERVICE_NAME (not this key) with OTEL_RESOURCE_ATTRIBUTES=service.instance.id=<id>: without OTEL_SERVICE_NAME, a freshly generated id silently overrides that variable on every restart.

IntervalSeconds is floored at 5 and capped at 86400 (24 hours), the cap there so a config typo several digits too long cannot overflow the millisecond count it is converted to. Sixty is the OTLP default and the right answer for almost everyone: the interval also decides how often every observable gauge is polled, and the loaded-chunk read behind one of them is not free.

For a local collector, the whole config is the endpoint:

{
  "Endpoint": "http://localhost:4318",
  "Protocol": "http/protobuf"
}

A hosted backend wants an auth header. Grafana Cloud's OTLP endpoint takes HTTP basic auth, with the instance ID as the user and an access policy token as the password:

{
  "Endpoint": "https://otlp-gateway-prod-eu-west-2.grafana.net/otlp",
  "Protocol": "http/protobuf",
  "Headers": {
    "Authorization": "Basic MTIzNDU2OmdsY19leGFtcGxldG9rZW4="
  },
  "IntervalSeconds": 60
}

Headers goes out with every export, so anything a backend accepts works the same way: x-honeycomb-team for Honeycomb, api-key for New Relic, x-scope-orgid for a multi-tenant Mimir. Values are percent-encoded on the way into the exporter, which the OTLP header format expects, so a base64 token with +, / and = in it needs no special handling. A literal comma in a header value does not survive the trip: the exporter unescapes the whole header string before splitting it on commas, so encoding it going in does not stop it from being read as a separator coming out. Pulse checks every header for this, and for two names that collide once leading and trailing whitespace is trimmed off, before ever handing them to the exporter, and refuses to start exporting rather than let either reach it: the server log names the offending header, never its value.

A collector that is down, refusing, or answering 401 still costs you nothing on the game side: the OpenTelemetry SDK exports from its own background thread and the tick loop never sees the failure. It no longer stays invisible, though. Pulse OTLP listens to the SDK's own diagnostic event source and turns the first failure of each kind into one line in the server log, repeated at most every ten minutes and logged at Warning rather than Error so a struggling backend can never count toward DieAboveErrorCount:

Pulse OTLP export to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/metrics failed: Response status code does not indicate success: 401 (Unauthorized). The backend answered: {"status":"error","error":"authentication error: invalid token"} Metrics are not reaching the backend; check Endpoint and Headers in pulse-otlp.json. This is logged again at most every 10 minutes.

A matching line reports the first successful export after a failure, so recovery shows up too, and a healthy server that has never failed still logs exactly one such line, at Notification rather than Warning, right after its first delivery.

Neither line is meant to carry a header value, or the query string or userinfo half of Endpoint either, for a backend that authenticates a signed URL that way instead of through a header. Before a line is queued, each of those values, at least 6 characters long (shorter than that reads as an ordinary id, not a credential), the credential half of it when the value has a "scheme credential" shape (a Bearer token echoed without its "Bearer ", say), and the JSON-escaped form of both, are matched case-insensitively and redacted out of the backend's answer and out of a gRPC failure's status detail, longest value first so a short one can never land inside a longer one's own match. Anything else shaped like a bearer or basic credential of at least 8 characters is redacted too, whether or not it matches a configured value. This is not exhaustive: a backend that transforms a secret some other way, hashing it or splitting it across two fields, could still get it into the log, so treat the log itself as sensitive before sharing it regardless. The backend's answer, once redacted, is clipped to 200 characters.

A malformed Endpoint, and a Headers entry with a comma in its value or a name that collides with another once trimmed, are cases Pulse checks itself before the exporter is ever built: it logs one error, naming the problem and never the value, and registers nothing. An IntervalSeconds so large it would once have overflowed the millisecond conversion is not one of those cases any more: it is silently clamped to 86,400 seconds (24 hours) and exported at that rate instead, with no error at all. A Headers shape neither check above names, such as a comma inside a header name rather than its value, or a header called User-Agent (which collides with one the exporter sets on its own), still reaches the OpenTelemetry SDK's own option validation, which throws; Pulse catches that too, so nothing crashes and nothing is exported, but the log line only names the exception type, not the header. For anything these lines do not explain, the SDK's own, far more verbose self-diagnostics turn on by dropping an OTEL_DIAGNOSTICS.json file next to the server.

Source: README.md