Home
Sign in

When something on the network does not work, ORC8R tells you what went wrong in words. A failure shows up in one of three places: when you submit a request, on the pool, or on the endpoint. Each of the three prints its own reason. This page is organised by what you saw, and it quotes the messages exactly, so you can search it for the text on your screen.

Where failures show up

When you submit a request. Anything ORC8R can decide from the request itself is refused the moment you submit it: an endpoint the app does not declare, a setting an endpoint does not have, two apps asking for the same port, a service name someone else has already claimed, no eligible address plan. The message names the exact field that is wrong, as a dotted path:

apps.postgres.endpoints.pg.expose: no public address plan is eligible for this pool:
  plan "edge-v6" is a v6 plan; plan "dc-v4" is routed to no host of this pool
apps.postgres.endpoints.pg.name: service name "maindb" is already claimed by pool "db" in project "shop"
apps.postgres.endpoints.pg.weight: unknown endpoint decoration field; accepted: expose, name, port
apps.web.endpoints.api: app "web" does not declare endpoint "api"; declared: http, metrics
tcp port 8080 is claimed by both "web" endpoint "http" and "proxy" endpoint "front"; override one with an endpoint port

None of this reaches a node, and nothing is dropped quietly. An exposure is either accepted or refused; it is never accepted and then forgotten. Fix the field the message names and submit again.

On the pool. A pool that is blocked or rejected carries a state_change_reason — one line saying why. A configuration failure that would fail the same way every time blocks the pool immediately, instead of retrying into a wall:

configuration failure: <what the node reported>
3 consecutive node failures: <reason>

Exposure, address and certificate problems never block a pool. They were already refused when you submitted.

On the endpoint. Once your app is deployed, each combination of node, app and endpoint carries a status: ready, unready, or no report at all, which reads as unknown. A node counts as endpoint-healthy only when the node is online and the endpoint reports ready. An unready status carries the probe's own words.

An endpoint never turns ready

The status carries the probe's own words. These are the ones you will see, and what each one means:

connect to port 5432 failed: Connection refused     the app is not listening yet, or crashed
connect to port 5432 timed out                      a firewall on the node, or the app is wedged
http probe /healthz returned 503                    the app is up and says it is not ready
http probe /healthz failed: <transport error>       nothing is speaking HTTP on that port
probe command exited with exit status: 1            your probe command's own verdict
probe command timed out                             raise `timeout`, or make the check cheaper

A Connection refused that never clears is almost always the same bug: the app bound loopback. Probes dial the node's routable address, so an app on 127.0.0.1 reports unready forever while looking perfectly healthy over SSH. Bind 0.0.0.0. See Authoring apps.

The status does not flip on the first blip, so give it time before you read it: three consecutive failures to go unready, one success to come back, on a 10-second interval by default.

A name does not answer, or your app cannot find another service

Internal names are answered by the resolver inside the agent on the node itself. When a name gives you nothing, work down this list:

  1. Is the node online and endpoint-healthy? Address answers follow node liveness; SRV answers follow endpoint health. The plain name — db.shop.internal, the one that means the pool — answers empty when no node it may offer is online.
  2. Is it a slot name? db-1.shop.internal answers nothing while slot 1 is unhealthy, by design — a slot name is fenced to its own node. It never substitutes another node. If your client is happy with any node, ask for the plain name instead — but note that the plain name is fenced to slot 1 too whenever any endpoint of the app serves primary, which is the default (Authoring apps).
  3. Are you calling from another project? Then write the name out in full. A node adds its own project's suffix to short names, but that shortcut is not extended across projects, and the endpoint must also be exposed at org scope. From outside its own project, an unexposed sibling's name answers NOT_FOUND. Expose the endpoint at org scope, and use the full name. See Project and organization exposure.
  4. Is the resolver installed on this node? When it is, the agent logs internal DNS resolver listening and internal DNS zone routed to the agent resolver, naming the mechanism it used. If instead you see resolv.conf cannot address a resolver off port 53; internal names will not resolve, the node's operating system cannot route the internal suffix to the resolver's address, and no internal name will ever resolve on that node.

A name inside the internal suffix that names nothing answers NXDOMAIN, so a resolver walking its search list moves on rather than stopping there.

Negative answers, slot answers and SRV answers carry a 1-second TTL; spread answers 5 seconds. If a stale answer persists much longer than that, the cache holding it is not the platform's: application-level resolvers that cache forever are the usual culprit — the JVM's default is the classic — and networkaddress.cache.ttl is the usual fix.

A public name has no address

A public name answers with the pool address, or it answers with no address at all. If the AAAA/A records — the ones that carry an address — are missing while SRV records exist, the pool has drawn no address. Read the rejection you got at submission, and check that the address plan you expect actually covers a host the pool can be placed on.

If no public record exists at all, check the sink first. The sink is whatever holds your public records for you. Only provider-api publishes today; every other sink works the records out, logs them, and serves them to nobody. The server says so, once per sink kind:

public DNS records are derived but not served: no record sink implementation yet   sink=builtin

When records do converge, the sink says that too, with counts, so you can see whether a change actually landed:

converged the org zone's records at Cloudflare   zone=acme.orc8r.com created=2 updated=0 deleted=1
public record convergence failed; retrying       error=cloudflare update failed (403): …

An unchanged zone produces no line at all. Convergence remembers what it last published, so silence means "there was nothing to do", not "nothing happened". A change of leader clears that memory and forces a republish.

Records carry a 30-second TTL and are fed by health, so a withdrawal takes effect in about half a minute — unless your DNS provider refused the sub-minute TTL and clamped it to its own minimum, in which case failover through that provider is as slow as its floor.

A certificate never arrives

The organization wildcard is ordered over ACME. Several things have to be true first, in a strict order, and each step logs:

no record sink is publishing; holding off on public certificates   orgs=1
issued the org wildcard certificate                                zone=acme.orc8r.com renews_in_secs=5184000
public certificate in place                                        zone=acme.orc8r.com serial=…
public certificate issuance failed                                 failures=3 rate_limited=false retry_in_secs=900 reason=…
  • Issuance never starts against a sink that publishes nothing. So a certificate that never appears usually means the DNS tab is not finished, not that ACME is broken. Finish the DNS tab first.
  • Validation is requested only after the challenge record is confirmed published. the record sink did not publish the challenge for {zone} within Ns means the sink accepted your write and the record never appeared. Credentials scoped to the wrong zone are the common cause.
  • Failures are retried after a wait, and a refusal for exceeding a rate limit waits the full time the authority asks for. This is why you prove out a new deployment against the ACME staging directory: a wrong delegation there costs nothing.
  • Test connection on the DNS tab checks public naming, the provider zone, and the ACME directory independently. Run it before you blame issuance.

A public address that does not answer

The name resolves, the certificate is in place, and traffic still does not arrive. The host logs are what to read:

holding pool address                     pool=… address=2001:db8:1::7      (node side, expected on every node)
pool address released                    pool=… address=…
host acknowledged claim snapshot         host=… revision=… addresses=1
claim_update ack timed out; will retry   host=…
pool address claims not settled; retrying

Every node of the pool should log holding pool address, not only the healthy ones. That is deliberate, so that a node that is coming back does not drop the first connections forwarded to it. One host should be acknowledging claim snapshots. If claims are not settling, the address has no way in, no matter how healthy the nodes are.

One case is worth calling out on its own, because it is a gap rather than a fault:

claim: 2001:db8:1::7 (pool …, plan edge-v6) has dsr off and is not realized:
       the NAT funnel that serves such plans is not implemented on this host

dsr is the plan's field name for direct server return: replies leave straight from the node that served the request, bypassing the host that received it. A plan with dsr off is not served on Linux hosts today. If you turned the flag off because of reverse-path filtering — the check that drops a packet whose source address does not match the path it arrived on — the exposure will not work until the filtering is addressed and the flag is turned back on.

Beyond that, the usual suspects sit outside the platform: the block is not actually routed to the hosts you named, something else on the segment answers for the address, or a router upstream filters the ports. The platform believes your plan; it cannot check it.

Common traps

An identical reconfiguration is a no-op. There is no "touch" that re-runs anything. Redeclaring an address plan with the same block, hosts, scope and DSR flag returns without emitting an event. Re-submitting a byte-identical request changes nothing. An unchanged zone is not republished. An address set that is already in place costs the agent no commands. The composer even leaves controls at their defaults out of the document it submits, precisely so that forking a pool and changing nothing stays a genuine no-op. If you want something re-done, change it, or restart the component whose state you suspect.

A pool keeps an address it already holds. An address is drawn once per family. Retiring a plan, or changing its scope, does not move a live service off it — retiring is refused while the plan still backs an address, and the message says how many. A reconfiguration only draws the family a pool is missing.

A v6-only service is unreachable from v4-only clients. The name resolves, the service is correctly configured, and a large part of the public internet still cannot reach it. If the service is for the public, declare both families.

An endpoint setting the platform does not have is rejected, not ignored. An endpoint takes expose, name and port and nothing else; anything else is refused at submission, naming the dotted path of the key. A silently dropped setting is a misconfiguration you would find out about much later.

An unresolvable app cannot be exposed publicly. An app that does not resolve in the project contributes no endpoints to the pool surface, so its public exposure would vanish along with it. Submission refuses instead. Publish the app, or pin a version that exists.

Service names are quarantined after release. Another project cannot take a name that was recently released. The project that released it can re-claim it immediately — that asymmetry is what lets a service migrate between pools without a gap.