docs / guide · updated 2026-08-01
Monitor a BlobArchive node
BlobArchive exposes Prometheus-compatible metrics on each process’s private metrics listener. Scrape every process separately. Keep the listener on loopback or a private operations network; it is not a public API.
The tables below are a starting alert policy, not a universal ruleset. Adjust
time windows for your polling and publication cadence, but preserve the
distinctions between process health, readiness, freshness, and data
correctness. up is supplied by the scraper rather than BlobArchive. Standard
Go, process, and host metrics should cover CPU, memory, file descriptors,
restarts, filesystem capacity, and I/O pressure.
The private metrics listener also serves /healthz and /readyz. Health means
the process is running. Readiness means selected state is safe to serve. A
ready node may deliberately serve a complete last-good generation during a
writer outage, so readiness does not prove freshness.
For counters, alert on a sustained increase() rather than their absolute
value. For timestamp gauges, calculate time() - metric only when the metric
is greater than zero. Alert explicitly when a continuously exported required
series is absent. Some event-labelled counters do not materialize until their
first event, so confirm their registration from /metrics metadata or a
controlled test instead of interpreting absence as zero. The labels you add
in your scraper configuration are deployment-specific and are not shown here.
Embedded follower
| Signal | What healthy looks like | Starting alert condition |
|---|---|---|
up, /healthz, /readyz | The target is scraped, the process is healthy, and readiness becomes green after the selected heads are serviceable. | Page when up or health fails. Alert when readiness remains unavailable after startup or catch-up. |
bloar_follow_head_ready{head} and bloar_head_quarantined{head} | Every selected head is ready and none is quarantined. | Page on a ready-to-not-ready transition or any quarantine. Preserve the refusal evidence before recovery. |
bloar_head_synced_to{head}, bloar_head_covered{head}, and bloar_head_adoptions_total{head} | Coverage and the accepted frontier advance when the selected writer publishes. | Alert when the frontier stops advancing beyond the writer’s expected cadence. Compare with source publication before blaming the follower. |
bloar_follow_polls_total{channel,outcome} and bloar_follow_admission_last_success_timestamp_seconds | Polls continue and authenticated generations are admitted periodically. | Alert on sustained poll failures or an accepted-generation age beyond your freshness objective. Do not use successful HTTP responses as a freshness proxy. |
bloar_follow_sync_duration_seconds{head,outcome}, bloar_follow_sync_last_success_timestamp_seconds{head}, and bloar_follow_sync_active | A newly admitted generation finishes retained-closure sync, and active work eventually returns to zero. | Alert when sync remains active without progress or repeatedly fails. A quiet success timestamp is normal when there is no new generation to adopt. |
bloar_follow_refusals_total{reason} and bloar_follow_synced_to_floor_lag{head} | Refusals do not rise in steady state and the accepted frontier does not fall behind its durable floor. | Page on signature, replay, manifest, root, conflict, or quarantine refusals. Investigate persistent floor lag. |
bloar_follow_source_available{source}, bloar_follow_source_selected{head,source}, bloar_follow_conflict_active{head}, and bloar_follow_incomparable_active{head} | For a multi-source follower, authorized sources are available, selection is one-hot, and no conflict or incomparable claim is active. | Alert on loss of every eligible source or any conflict. These series are optional and may be absent in a single-authority configuration. |
bloar_beacon_reads_total{head,status}, bloar_beacon_read_duration_seconds, and bloar_store_corrupt_reads_total{head} | Known blobs return successfully with stable latency and corrupt reads remain zero. | Alert on sustained 5xx/503, latency exhaustion, or any corrupt read. Test one known non-empty blob byte-for-byte. |
bloar_bitswap_fetches_total{outcome}, bloar_bitswap_fetched_bytes_total, bloar_p2p_live_peers{direction,transport}, and bloar_p2p_reachability{state} | Fetches succeed, peers remain available, and a publicly reachable follower reports public reachability. | Investigate repeated fetch failure, loss of all useful peers, or an unexpected reachability downgrade. |
bloar_pin_reconcile_errors_total{head}, bloar_gc_runs_total{outcome}, bloar_gc_last_success_timestamp_seconds, and bloar_scrub_runs_total{outcome} | Pin reconciliation is quiet; scheduled GC and integrity scrubs complete. | Page on reconcile errors or corruption; alert when scheduled maintenance misses its expected window. |
Kubo replica
Scrape both bloar-kubo-replica and Kubo. BlobArchive controls authenticated
generation transitions, while Kubo owns the repository, swarm, pins, and
garbage collection.
| Signal | What healthy looks like | Starting alert condition |
|---|---|---|
up, /healthz, /readyz, and bloar_replica_state_readable | The controller and Kubo are reachable, durable controller state is readable, and the selected generation is ready. | Page on either target being down, unreadable state, or readiness loss. |
bloar_replica_generation_current, bloar_replica_generation_head_present{state,head}, and bloar_replica_generation_synced_to{state,head} | Exactly one current generation is committed and all selected heads are present at the expected frontier. | Page if the current generation disappears or a required head is missing. Alert when the frontier remains stale. |
bloar_replica_generation_pending, bloar_replica_transition_in_progress, bloar_replica_transition_age_seconds, bloar_replica_pin_progress_blocks, and bloar_replica_pin_progress_bytes | A pending generation makes visible pin progress and is eventually committed. | Alert when transition age grows without block or byte progress. Do not delete the previous generation to force completion. |
bloar_replica_last_commit_timestamp_seconds and bloar_replica_transitions_total{operation,outcome} | Commits follow accepted publications; prepare, protect, commit, cleanup, audit, and runtime audit operations succeed. | Alert on a commit age beyond the selected writer’s publication objective or sustained transition failures. |
bloar_replica_last_transition_failure_timestamp_seconds{operation,class} | No new failure timestamp in steady state. | Page on state, pin-safety, or audit failures; investigate transient upstream failures when sustained. |
bloar_replica_cleanup_anchors and bloar_replica_cleanup_oldest_retained_age_seconds | Old controller-owned anchors are retired after a safe transition and cleanup debt remains bounded. | Alert when anchor count or retained age grows across several successful commits. |
bloar_replica_announcements_total{outcome} and bloar_replica_last_announcement_timestamp_seconds | Successful provider announcements follow committed generations. | Alert on sustained announcement failure or a stale announcement timestamp while commits advance. |
bloar_replica_gateway_enabled and bloar_replica_gateway_serving | The configured read-only gateway is serving when enabled. | Page when an enabled gateway stops serving. Confirm one known non-empty blob byte-for-byte. |
| Follower refusal, source, and readiness metrics from the table above | Source authentication and admission remain healthy before Kubo mutation begins. | Treat trust refusals as correctness incidents, not Kubo performance issues. |
| Kubo process/repository and host filesystem metrics | Kubo remains healthy with enough disk, file descriptors, memory, and GC headroom for the selected archive. | Alert well before repository storage is exhausted. Exact Kubo metric names depend on the Kubo release and exporter configuration. |
Writer deployment
A public writer deployment is several failure domains: the private writer, finalized and unfinalized indexers, and the read-only public edge. Scrape each as a distinct target. One green process does not prove the publication path is healthy.
| Component and signal | What healthy looks like | Starting alert condition |
|---|---|---|
All processes: up, /healthz, /readyz | The writer, every configured indexer, and the public edge are scraped and healthy; readiness matches each process’s role. | Page on any required target loss. Keep role-specific alerts active so a writer outage is not mistaken for an indexer failure. |
Writer heads: bloar_head_synced_to{head}, bloar_head_covered{head}, and bloar_head_quarantined{head} | Finalized coverage advances monotonically and no head is quarantined. | Page on quarantine or rollback; alert when expected coverage stops advancing. |
Ingest: bloar_ingest_blobs_total, bloar_ingest_bytes_total, bloar_ingest_rejects_total{reason}, and bloar_ingest_kzg_verify_duration_seconds | Expected blobs are admitted, rejects remain explainable, and verification latency stays within the ingest budget. | Alert on sustained rejects, missing expected blobs, or latency approaching request deadlines. |
Finalized indexers: bloar_index_last_progress_timestamp_seconds{head}, bloar_index_outcomes_total{head,outcome}, bloar_index_retries_total{head,reason}, and bloar_index_archive_available{head} | Each configured head progresses and its archive remains available. | Page when archive availability is zero; alert when progress is stale or retry outcomes rise without success. |
Unfinalized tracker: bloar_unfinalized_last_success_timestamp_seconds{head}, bloar_unfinalized_source_head_slot{head}, bloar_unfinalized_generation{head}, and bloar_unfinalized_retries_total{head,reason} | The bounded optimistic generation is refreshed near the source head; retry rates remain low. | Alert when the last success exceeds the configured live objective or retries rise without a new generation. |
Writer publication: bloar_ipns_publication_last_success_timestamp_seconds and bloar_ipns_publication_stage_total{stage,outcome} | The timestamp advances and every publication stage records successes. | For v0.1.0, warn when publication age exceeds 10 minutes. Page on a sustained stage-specific error-only streak. A stale IPNS pointer keeps serving valid old state, so request success is not freshness evidence. |
Public edge transaction: bloar_edge_publication_transactions_total{operation,outcome,stage}, bloar_edge_publication_stage_duration_seconds, and bloar_edge_publication_wait_duration_seconds | Apply and restore transactions finish within their budgets without queue starvation. | Alert on sustained timeouts, failures concentrated in one stage, or admission waits consuming the work budget. |
Public edge DHT: bloar_edge_dht_routing_table_peers and bloar_edge_dht_routing_table_sample_timestamp_seconds | The routing table is populated and samples remain current. | Alert when the table empties after bootstrap or its sample timestamp stops advancing. Use the detailed DHT query and lookup metrics for diagnosis, not as paging signals by default. |
Public swarm: bloar_p2p_reachability{state}, bloar_p2p_live_peers{direction,transport}, bloar_bitswap_scheduled_bytes_total{peer_class}, and libp2p_rcmgr_* | The edge is publicly reachable, useful peers remain connected, and resource-manager usage stays below its limits. | Page on reachability loss; alert near 90% of a resource limit or on sustained resource-manager blocks. |
Store safety: bloar_store_corrupt_reads_total{head}, pin reconciliation, GC, scrub, and host filesystem metrics | Corrupt reads remain zero and maintenance completes before capacity or integrity risk accumulates. | Page on any corrupt read; alert on missed maintenance or low disk headroom. |
| External canary | An unrelated node can resolve the writer’s DNSLink/IPNS authority, discover the provider, fetch one known non-empty blob, and reproduce its expected digest. | Page when the end-to-end canary fails after bounded retries. Internal static peers do not test public discovery. |
What the metrics cannot prove
- An HTTP
200does not prove freshness; a frozen authority can serve a valid older generation. - Advancing coverage does not prove a filtered writer included every expected blob. Monitor the L2 posting schedule independently and retain byte-exact regression anchors.
- A statically peered local follower proves serving, not public provider discovery. Use an unrelated-node canary for the public path.
- A rule file on disk is not an active alert. Verify loaded rules, successful reloads, and delivery through the real notification path.
For recovery principles and layer-by-layer diagnosis, continue with operate and troubleshoot a node.