docs / guide · updated 2026-08-01

Operate and troubleshoot a node

This guide is for the BlobArchive follower/replica operator, commonly the same person or organization as the L2 node operator. It covers the local archive and read path. The selected writer operator separately owns source-schedule completeness and posting-mechanism monitoring.

In a public writer deployment, diagnose the two processes separately. The private writer owns ingest, archive mutation, signing, and creation of the publication bundle. The read-only public edge owns the Internet-facing Bitswap listener and the verified provider-before-IPNS transaction. Check the writer’s publication-stage counters and the edge’s readiness, scrape, and public-libp2p metrics; one process being green does not prove the other path.

Start from the invariant that a follower or replica has two independent jobs:

1. retain a complete authenticated local generation; and
2. make that generation reachable to the intended client or swarm peer.

A green process check does not prove either job by itself.

Wire a local consumer

The follower/replica operator runs the follower or Kubo read gateway on loopback or a private network. The L2 node operator points Nitro’s blob-client beacon URL at that local view:

--parent-chain.blob-client.beacon-url=http://127.0.0.1:8550/live

Use the physical finalized head instead of live when the selected profile is finalized-only. Keep the URL local; the initial BlobArchive service does not publish a hosted public beacon API.

Before changing a production client, test one known non-empty historical slot through the exact endpoint and retain its byte digest as a regression anchor.

Monitor service and progress separately

Use the role-based monitoring tables for exact metric names and starting alert conditions. The principles below explain how to interpret them during an incident.

The follower/replica operator scrapes the private metrics listener and alerts on at least:

• process and metrics target availability;
• /readyz;
• source availability and last-good service;
• selected-head synced_to and publication lag;
• signature, replay, manifest, root, conflict, or quarantine refusals;
• archive-availability gauges for indexers and replicas;
• pin transition and audit state;
• GC and integrity-scrub success timestamps;
• process restarts, memory, file descriptors, connections, and streams; and
• disk-pool allocation and host I/O pressure.

Verify alerts from the live rule API, not only from files on disk. A correctly written rule which is not loaded is not monitoring.

Readiness and freshness are intentionally different. A node can remain ready while serving a complete last-good generation during a writer outage. That is continuity, not proof that publication is advancing.

Interpret HTTP outcomes

For the beacon blob route:

• 200 means the selected local generation supplied the response;
• 404 means authenticated coverage includes the slot but the selected head has no matching blob;
• 400 on a filtered provisional request usually means required versioned_hashes were omitted or malformed;
• 429 is local admission control and is safe to retry after the indicated delay; and
• 503 means the node cannot currently prove or materialize a coherent answer.

Never rewrite a 503 into a 404. That converts “unknown or temporarily unavailable” into a false permanent absence.

Diagnose discovery before NAT

Use how BlobArchive uses IPFS as the map for the DNSLink, IPNS, DHT, libp2p, relay, and Bitswap layers below.

Check the layers in order:

1. DNSLink returns exactly one /ipns/<name> value.
2. IPNS resolves to an exact publication-document CID.
3. The document passes signer, revision, replay, manifest, and head checks.
4. The node discovers the expected peer and admitted addresses.
5. A direct connection exists or can be upgraded.
6. Exact bytes transfer from the expected peer.

Stopping at step 2 or 4 is a discovery success, not a data-path success.

NAT and relay

Relay reservations are an introduction and DCUtR control plane. A raw blob already consumes the production relay circuit’s 128 KiB data allowance before protocol framing, so relay-only Bitswap is not the fallback.

If a private node discovers a provider but cannot fetch:

• verify whether a connection is direct or relay-limited;
• inspect the peer’s advertised addresses and the local resource-manager blocks;
• confirm the connection-manager high watermark is below—not above—the hard resource limit;
• try a static relay reviewed by the follower/replica operator when AutoRelay discovery is unsuitable;
• preserve both TCP and QUIC evidence rather than assuming one transport wins; and
• when possible, let the private node initiate a direct outbound connection to the public peer.

The field tests found networks where signed discovery succeeded 5/5 while direct fetch succeeded 0/5. Treat that as a normal diagnosable boundary, not as proof the DHT failed.

Storage and maintenance

Back up the whole embedded store atomically. For a Kubo replica, back up Kubo under its own policy and preserve the controller state separately; that state contains replay floors and pin ownership.

Do not:

• delete follower state to clear an outage;
• lower replay or profile-digest floors;
• run Kubo GC as part of a controller rollback;
• bulk-remove pins the controller does not own; or
• treat a successful GC as a full byte-integrity scan.

Monitor GC and scrub independently. A slow pin or source outage should leave the previous committed generation serviceable.

Recovery order

When a source or process returns:

1. restore the authoritative dependency first;
2. wait for its readiness;
3. allow dependants to retry in process;
4. verify stable PID/restart count where continuity is under test;
5. verify coverage resumes without lowering floors; and
6. verify induced alerts both fire and clear.

For manifest changes, missed posting sources, local corruption, backup, disaster recovery, public-service limits, and rollback conditions, use the exact operations runbooks.