Running the partner provider - Documentation

The partner provider is one provider that every project in your zone can deploy from. For each node it picks the partner cloud that sells the requested shape for the least money in the requested region, right now. Vultr and Hetzner are supported today. This page is for the zone operator: how to deploy it, what each setting does, how to change it safely, and what to check when something looks wrong. Tenants need none of it. To them the partner is one more provider in the request wizard.

What it does

Cheapest cloud per node. Every quote goes to each cloud the partner holds a key for. Each cloud answers with its cheapest plan that fits the shape exactly, and the cheapest answer wins. The partner does not prefer a site that is already open, and it never crosses region classes. Two nodes of one pool can land on two different clouds.
Region classes. A tenant picks a coarse region such as eu-central, never a cloud location. Each class is an ordered list of sites. A site is one cloud location under a name you choose, and the list order breaks a price tie.
Neutral site names. A site name such as eu-central-a is the only location the platform records for a partner node. It appears in the site id, the gateway pool's name and the pool address, so no cloud or city name does.
Cross-cloud mesh. Every node joins its project's WireGuard mesh, whichever cloud it runs on. Each site has its own site gateway in that site's cloud. The site's nodes hold no public IPv4 address, and they reach other sites and the internet through that gateway. A site opens when its first node is placed there and closes when its last node ends. You never manage a gateway by hand.
White label. Price lines, invoices, park messages, deploy progress and errors use the partner's own words, such as "2 vCPU · 4 GiB · eu-central" or "no capacity in Europe Central right now". No cloud plan, city or cloud region appears in them. The clouds are still your subprocessors, and a tenant with root can tell them apart from the network or the virtual hardware. The aim is not to name them without need. It does not keep them secret. Sys members, the partner pool's logs and its admin page keep full detail.

Deploying it

You need a zone operator account with the right to manage zone providers, an account on at least one partner cloud, and a checkout of this repository.

1. Push the app

ORC_REGISTRY_TOKEN=<token> scripts/package-partner-provider.sh --push https://<your server>/sys

This builds orc-prov-partner for linux amd64 and arm64, packages it, and publishes it as sys/orc-prov-partner:default, so the system org can deploy it. The script needs two tools, and finds them this way:

The orc CLI. It uses ORC_BIN when that is set, then orc on your PATH, then a copy kept in ~/.cache/orc-cli (set ORC_CLI_CACHE to move it). If none of those exist, it downloads the latest published CLI, checks it against the release's SHA256SUMS, and keeps it in that cache for next time.
zig. cargo-zigbuild uses it to cross-compile. The script uses zig on your PATH, else ZIG_DIR, else ~/.local/zig-tools/zig-0.14.1. It stops and says so when it finds none. pip install ziglang is one way to get it.

ZIGBUILD_GLIBC sets the glibc floor (default 2.28), and STAGE_DIR and CARGO_TARGET_DIR move the build output.

2. Create the partner pool

In project sys, request a pool named partner on provider core, running the app orc-prov-partner, with a size of 2 or more. Keep the name partner. Park messages name the provider, so a pool named after a cloud would name that cloud to every tenant. The admin page also accepts only requests made for sys/partner.

Fill in the params (see Params). The least that works is partner_id, one cloud's key and that cloud's owner tag:

Param	Example
`partner_id`	`zone-1`
`hetzner_api_token`	your Hetzner Cloud token, Read & Write
`hetzner_owner_tag`	`orc-partner-hetzner`
`vultr_api_key`	your Vultr API key
`vultr_owner_tag`	`orc-partner-vultr`
`ptr_domain`	your zone's domain, such as `cloud.example.com`

The partner needs no App data. It keeps nothing on disk, so a new replica rebuilds everything from its params and the clouds.

All replicas offer one shared resource, urn:orc:partner:<partner_id>. One replica is serving: it gets every quote, deploy and destroy. The others are standing by. When the serving replica goes, a standby takes over. Each replica's log says which it is.

3. Widen it to zone scope

Once the partner has made its first offer, the provider appears on the Resource Providers page of project sys. Change its visibility to Global. In the API, that is POST /api/orgs/sys/projects/sys/providers/<provider>/change-scope with {"scope": "zone"}. Every project of every org can then deploy from it. Its site gateways stay in the system org.

4. Set its pricing strategy

The partner quotes its own prices, so it takes a markup strategy. Set one on the provider's card, even a markup of 0. With no strategy, every partner node is unpriced: it runs, but carries no price at all, which is not a price of zero. The markup a node is admitted with stays with it for its life. A node that was admitted with no strategy stays unpriced until it is replaced. The Pricing operations page, on a deployment with billing, covers strategies in full.

Two costs are always yours and never a tenant's:

Site gateways. They are never billed to a tenant, and the partner never refuses one on price.
Pool addresses. A public pool's IPv4 address is quoted to the tenant as one line at rate 0. The admin page's Pool addresses view shows what the clouds charge you for them.

Params

Every setting is a param of the partner pool. Each cloud has its own vultr_ or hetzner_ prefix, so a partner never reads a standalone Vultr or Hetzner provider's settings, even on the same host. An empty param takes its default. A param the partner cannot read stops it from starting, and its log names the param.

The partner

Param	Default	What it does
`partner_id`	required	Names the one resource every tenant pool deploys from: `urn:orc:partner:<partner_id>`. Printable ASCII, no colon, no space. Never change it. Keys, clouds and maps can all change underneath it without touching the resource.
`regions`	the built-in map	The region map, as JSON. See Region map.
`os_map`	the built-in map	The OS map, as JSON. See OS map.
`gateway`	`{"cpu": 1, "mem": 2}`	The smallest site gateway, as JSON. See Gateway shape.
`ptr_domain`	none	The zone's domain for reverse DNS. See Reverse DNS. Empty sets no reverse DNS.
`access_keys_url`	none	Where the zone publishes the keys its access edge signs with: `https://<zone web url>/.well-known/orc-access/jwks.json`. It must be an https URL. The admin page checks every request against these keys. Empty keeps the admin page closed.
`ssh_keys`	none	Break-glass access. OpenSSH public keys, one per line, installed for root on every Linux instance the partner creates from then on. A line that is not a public key stops the partner from starting.

Each cloud

Replace <cloud> with vultr or hetzner. The key param is vultr_api_key for Vultr and hetzner_api_token for Hetzner.

Param	Default	What it does
`vultr_api_key`, `hetzner_api_token`	none	The account the partner places nodes in. A sensitive param. Leave it empty to leave that cloud out. At least one cloud needs a key. Rotating it is a reconfigure. Removing it while the cloud still holds nodes is not supported, so retire the cloud first (see Retiring a cloud).
`<cloud>_enabled`	`true`	Off retires the cloud. No new node is placed there, and the nodes it holds are destroyed through it as they end. Its key stays.
`<cloud>_owner_tag`	required with a key	Stamped on every instance the partner creates in that account, and the only way the partner finds them again. See Owner tags. On Hetzner it is at most 63 letters, digits, `-`, `_` and `.`, and it begins and ends with a letter or digit.
`<cloud>_also_owned_tags`	none	Comma-separated owner tags from an earlier configuration whose instances still belong to the partner. Their instances are adopted and destroyed like the current tag's.
`<cloud>_regions`	every location the region map names	Comma-separated cloud locations the partner may use there.
`<cloud>_plans`	every plan	Comma-separated plan ids the partner may create.
`<cloud>_images`	every validated family	Comma-separated OS families or exact image ids the partner may boot.
`<cloud>_max_nodes`	no cap	The most tenant nodes the cloud may hold. Site gateways are not counted. At the cap the cloud takes no new node.
`<cloud>_max_monthly_spend`	no cap	The most the cloud's nodes may cost a month, in US dollars, gateways included. A node that would take it over is placed elsewhere. Gateways count toward the cap but are never refused by it.
`<cloud>_exchange_rate`	`1`	US dollars per one unit of the currency the account bills in. See Exchange rates.
`<cloud>_gateway_plan`	none	One plan that every gateway in this cloud runs on. It overrides the gateway shape. It applies to gateways created after the change.
`<cloud>_pool_ipv4_limit`	`10`	How many public IPv4 pool addresses the account may hold. The partner offers the sum over its enabled clouds.

The regions, os_map and gateway params are JSON text. The pool form takes them as text, and the app receives each one as a file. They are not structured params (see Authoring apps).

Region map

regions is a JSON object keyed by region class id. Each class has a title, which tenants see, and a sites list. Each site maps one site name to vultr:<location> or hetzner:<location>, using the cloud's own location id. List order breaks a price tie. This is the built-in map, plus a class named eu-west:

{
  "us-east": {
    "title": "US East",
    "sites": [{ "us-east-a": "hetzner:ash" }, { "us-east-b": "vultr:ewr" }]
  },
  "us-west": {
    "title": "US West",
    "sites": [{ "us-west-a": "hetzner:hil" }, { "us-west-b": "vultr:sea" }]
  },
  "eu-central": {
    "title": "Europe Central",
    "sites": [
      { "eu-central-a": "hetzner:fsn1" },
      { "eu-central-b": "hetzner:nbg1" },
      { "eu-central-c": "vultr:fra" }
    ]
  },
  "eu-west": {
    "title": "Europe West",
    "sites": [{ "eu-west-a": "vultr:ams" }]
  },
  "eu-nordic": {
    "title": "Europe Nordic",
    "sites": [{ "eu-nordic-a": "hetzner:hel1" }, { "eu-nordic-b": "vultr:sto" }]
  },
  "ap-southeast": {
    "title": "Asia Southeast",
    "sites": [{ "ap-southeast-a": "hetzner:sin" }, { "ap-southeast-b": "vultr:sgp" }]
  }
}

The rules:

Class ids and site names use lowercase letters, digits and dashes, start and end with a letter or digit, and are at most 40 characters. A site name cannot also be a class id, cannot be used twice, and two sites cannot map to the same cloud location. The partner refuses to start on a map that breaks these rules.
Name sites neutrally. A site name ends up in places tenants can read, so do not put a cloud or a city in it.
Never remap or remove a site name while it is in use. A site is in use while it has nodes, and for up to 7 days after its last node ends while the platform keeps its stable address for a returning site. Remapping eu-central-a to another location would move a live site's identity to a place where none of its gateway, network or addresses exist. The partner does not check a change when you make it, but it does catch a site whose name was remapped while live. It knows where a name is live from the site tag on each node it created (this version on) and, with ptr_domain set, from each stable address's reverse DNS. Such a site is left out of quotes so the next site in its class wins. A deploy into it fails with a generic message, the partner pool's log names the site, where it is live and where the map now puts it (once, until it clears), and the admin page's Children view flags it. A live site with neither record (nodes created by an earlier version and no ptr_domain) is only flagged on the admin page, as a location no site name maps to. Mapping a different name to a live site's old location is harmless: it is a separate site. Nodes in a location whose site name was removed are still kept, inventoried and destroyed. The Overview shows "site guard incomplete" while a cloud's addresses could not be listed. To recover, put the old mapping back and reconfigure. Check before you edit: the admin page's draft validator warns when a draft removes or remaps a live site. Adding a site or a class is always safe: the new site opens by itself when the first node is placed there.
An entry naming a location its cloud does not sell right now is left out of quotes, and one INFO line in the partner's log names it.
Placement is exact-fit. A class whose cheaper cloud has no plan of exactly the shape asked for places every node of that shape on the other cloud. In the built-in map, for example, 2 vCPU / 4 GiB in us-east and us-west always lands on Vultr.

OS map

os_map is a JSON object keyed by router OS id, the id tenants pick. Each OS has a title and, per cloud, a selector: the cloud's own family, plus exactly one of id (an exact image id) or version (a whole word that must appear in the image's title; the plainest match wins). An OS with no selector for a cloud never routes there. Image ids are matched against each cloud's live catalog. Pin one with id only when a version match would be ambiguous. This is an excerpt of the built-in map:

{
  "ubuntu-24.04": {
    "title": "Ubuntu 24.04 LTS",
    "vultr": { "family": "ubuntu", "version": "24.04" },
    "hetzner": { "family": "ubuntu", "version": "24.04" }
  },
  "debian-12": {
    "title": "Debian 12",
    "hetzner": { "family": "debian", "version": "12" }
  },
  "alma-9": {
    "title": "AlmaLinux 9",
    "vultr": { "family": "almalinux", "version": "9" },
    "hetzner": { "family": "alma", "version": "9" }
  },
  "windows-2022": {
    "title": "Windows Server 2022",
    "vultr": { "family": "windows", "id": "windows-2022-standard-x64" }
  }
}

The full built-in map also has Ubuntu 22.04 and 26.04, Debian 13, AlmaLinux 10, Rocky Linux 9 and 10, and Windows Server 2025. Windows routes only to Vultr. A map you set replaces the built-in one, so copy every OS you want to keep. Changing an OS id's selector does not touch running nodes. Removing an id takes it off the form, and pools that use it can no longer place new nodes.

Gateway shape

gateway is a minimum: {"cpu": <vCPUs>, "mem": <GiB>}. In each location, the partner takes the cheapest x86 plan with at least that much CPU and memory. The default, {"cpu": 1, "mem": 2}, is the smallest shape both clouds sell in every built-in location. A <cloud>_gateway_plan overrides the shape with one named plan for that cloud. A gateway always boots the gateway image, whatever OS the node that opened its site asked for.

{ "cpu": 2, "mem": 4 }

A change applies to gateways created after it. Running gateways keep their shape, and the admin page lists the ones on an old shape. Replace one when it suits you by terminating its node. Its replacement keeps the site's stable IPv4 address and the pool addresses, but cross-site traffic and egress for that site stop for the few minutes it takes.

Owner tags

An owner tag is how the partner tells its own instances apart from everything else in the account. The partner only adopts an instance that carries its own child tag plus its owner tag or one of its also_owned_tags.

Give each partner installation its own tag in each account, and make it different from any standalone Vultr or Hetzner provider's tag in the same account. A shared tag would let one installation claim another's machines, and the platform destroys a machine reported by a provider that does not hold it.
Never just change a tag while the cloud holds nodes. To move to a new tag, set the new one in <cloud>_owner_tag and put the old one in <cloud>_also_owned_tags. New instances get the new tag. Remove the old tag once no instance carries it.
Never put another installation's tag in also_owned_tags.

Exchange rates

The platform prices in US dollars. A cloud account that bills in another currency needs <cloud>_exchange_rate: US dollars per one unit of the account's currency, such as 1.08 for a euro account. The default is 1, which is right for an account that bills in US dollars. A Hetzner project billed in euros with no rate set stays unpriced, rather than being sold at one euro to the dollar.

The rate is recorded on each instance when it is created, and that recorded rate is used for the instance's whole life. Changing the rate therefore prices only nodes created after the change. Running nodes are never repriced.

Caps

<cloud>_max_nodes and <cloud>_max_monthly_spend hold a cloud back. A cloud at a cap is unavailable for quotes and in the request form's availability, so the next cheapest site wins. A public pool pinned to a site whose cloud is at its cap parks instead (see Public pools). Raising the cap lets it place again.

After a roll or a failover, a newly serving replica has to count each capped cloud's nodes before it can use that cloud. While a cheaper capped cloud is still being counted, quotes answer "counting capacity, try again shortly" instead of placing the node on a dearer site, and pools ask again 5 seconds later. This wait is bounded. After 30 seconds, or as soon as a count fails, the cloud sits quotes out while its count is retried, and the next site wins.

Reverse DNS

Set ptr_domain to your zone's domain, such as cloud.example.com, and the partner names the reverse DNS of every address it holds under it:

each site's stable gateway address as gw-<site name>.<ptr_domain>;
each pool address as pool-<pool id>-<site name>.<ptr_domain>, with the pool id written as a hostname label (p-d000000000002).

A reverse lookup then names your zone and not the cloud. On every inventory pass, the serving replica compares each address's record with what the cloud reports and sets only the ones that differ, so a restart, a failover or a replaced gateway is fixed on the next pass. A failure is logged and retried. The partner sets reverse records only. It does not create forward records for these names.

Public pools

A pool that exposes an endpoint publicly keeps all its nodes in one site: the site that holds its pool address, or, before it has one, the site of its oldest live member. This gives the pool one public IPv4 address and one name, served by every node, as on a single-cloud provider. The first node is placed by price as usual.

When the pinned site cannot take another node, because its cloud is out of stock, at its cap or retired, the pool parks instead of leaving the site, with a generic reason such as "no capacity in Europe Central for this shape right now". It places again once the site can serve. When the pool stops exposing anything publicly, the pin is lifted and its next node goes to the cheapest site. Pools without public endpoints are never pinned.

Reconfiguring

Every change is made by reconfiguring the partner pool's params, including key rotation, a new cloud, a map edit or a cap change. The partner reads its params once at start, so a reconfigure is a rolling update: replicas are replaced one at a time.

With 2 or more replicas there is no downtime. A live replica always holds the resource, tenant nodes keep running, and no deploy is lost. If no replica is live, tenant nodes still keep running, and only deploys and destroys wait until one is back.
No node is repriced by a reconfigure. Prices come from what was recorded on each instance.
Right after a roll, expect up to 30 seconds of "counting capacity" answers for a capped cloud (see Caps).
Check a draft before you apply it. A replica given a param it cannot read does not start. Paste regions, os_map, gateway and ptr_domain drafts into the admin page's validator first, and copy its output back into the pool form.

Retiring a cloud

Set <cloud>_enabled to false and reconfigure. No new node is placed in that cloud, and its existing nodes keep running and are destroyed through it as they end. Leave its key in place.
Wait until it holds zero nodes. The admin page's Children view shows the count. Replace or move the remaining tenant nodes if you do not want to wait. Each site's gateway scales to zero when the site's last node ends.
Delete what it left behind, by hand, in the cloud's console. The partner does not delete these: the sites' stable IPv4 addresses, the site networks, and the sealed SSH key it created. Each carries the cloud's owner tag. For a retired cloud with no nodes, the Children view lists the stable addresses still held and names the other objects.
Then remove the cloud's key and owner tag from the params, if you want to.

Do not remove the key of a cloud that still holds nodes. The partner would no longer see those instances, and they would keep running, and billing, in the cloud account.

The admin page

The partner serves a read-only operator page on its admin endpoint, reached through the access edge (see Identity at the access edge).

Opening it. Its access name follows the usual pattern: sys--partner.<org zone>, where the org zone is the system org's zone. The pool has one http endpoint, so the name carries no endpoint part. sys--partner-<slot>.<org zone> opens one particular replica.
Who can open it. Holders of access_service on project sys: its project owners and members, and system org owners and members. Project viewers cannot. The page also checks the edge's signed access assertion on every request, and refuses anything not made for sys/partner/admin, even from inside the sys network.
Turning it on. The page checks assertions against the keys at the access_keys_url param, so set it to https://<zone web url>/.well-known/orc-access/jwks.json, using your zone's web URL. Until it is set the page is closed: every request is refused with a message naming the param, and each replica's log says so once at start.
It writes nothing. Params are changed only through the pool form.

It has six views:

Overview. Which replica you are on (its host's name and process id), whether it is serving or standing by, the version, the resource URI, when each cloud's price list was last read, and each cloud's last answer.
Config. The effective params: the partner id, ptr_domain and access_keys_url; per cloud, whether a key is set (never any of its characters), enabled, owner tags, caps, exchange rate and gateway plan; every class and site and whether its location is sold on the live catalog; every OS id with the image it resolves to on each cloud, flagging ids that resolve nowhere; and the gateway minimum with the plan and monthly price it gives per site.
Children. At the top, any site out of step with the region map: a site live at one place but mapped to another, or a location holding nodes that no site name maps to, with the fix. Then per cloud: key working or failing, enabled or retired, nodes held, caps used, sites cooling down after a refused deploy, open sites and their gateways, gateways on an older shape, and, for a retired cloud with no nodes, what it left behind.
Quote preview. Pick a class, CPU, memory, OS, optional pinned site and disks. The page shows every candidate site with its cloud, plan, image and price, and the winner, or why each site was excluded (config, cap, not sold, cooling down, retired, counting). It runs the real quote's code.
Pool addresses. Every pool address held, per site, with what the cloud charges for it.
Validate. Paste drafts of regions, os_map, gateway and ptr_domain. Each is checked with the same parsers the partner uses at start, resolved against the live catalogs, and written back as normalised params ready to paste into the pool form. It warns when a draft removes or remaps a live site.

A replica that is standing by shows a banner. It still shows the config, catalogs and quote preview, but anything that needs the cloud accounts is shown only by the serving replica. The endpoint is served from the pool's first slot, which may be the standby. If you see the banner, open the other slot by its slot name.

Troubleshooting

Standing by or serving. Each replica logs one of these lines:

"Standing by: this replica reads no cloud account until it is handed work". A standby lists no cloud account and sends no inventory. That is normal.
"This replica is serving the partner resource now…". From this point it keeps inventory, prices and reverse DNS current.

A replica becomes serving when it gets its first work, a quote included. The admin page's overview shows the same.

"counting capacity, try again shortly". A newly serving replica is counting a capped cloud's nodes before it places anything there. It clears within 30 seconds (see Caps). If it keeps coming back, check that cloud's key on the Children view.

Tenant messages are generic. The partner's logs have the detail. A tenant sees messages like these:

Tenant sees	Usually means
no capacity in class right now	Every site in the class is out of stock, at a cap, cooling down after a refused deploy, or retired. For a public pool, its pinned site is.
could not finish in class right now; it will be retried	A transient failure, or a cloud still being counted.
node … cannot be created at site … until the provider's operator changes its configuration	Something only you can fix: the site's cloud has no key in the params, or the OS has no image on that cloud.
node … could not be created at site … this time	Any other create failure. The node takes its normal backoff.

The partner pool's own logs carry the node and the site at INFO, such as "The cloud has no room for this node here right now; the site sits out quotes for ten minutes". The cloud's exact words are logged at DEBUG. Read the partner pool's logs at debug level to see them, and use the quote preview to see why a site was skipped. A site whose cloud refused a create for lack of room sits out quotes for 10 minutes, and the node is quoted again at another site.

A cloud's inventory cannot be read. When any cloud cannot be listed, the partner sends no inventory that pass, so nothing is declared lost. Lost-node detection pauses for every cloud until it can be read again. The log says "One cloud's inventory could not be read…".

Reverse DNS. Look an address up from anywhere:

$ dig -x 203.0.113.10 +short
gw-eu-central-a.cloud.example.com.

A gateway address should answer gw-<site>.<ptr_domain>, and a pool address pool-<pool id>-<site>.<ptr_domain>. If one does not, check that ptr_domain is set, wait for the next inventory pass on the serving replica, and look for "Some addresses' reverse DNS could not be set yet" in its log.

Nodes land on the dearer cloud. Run a quote preview for the shape. The usual causes are a cap on the cheaper cloud, a site cooling down, or no exact-fit plan on the cheaper cloud.

Providers — site gateways, pool addresses and provider visibility.
Authoring providers — the protocol behind quotes, site placement and shared resources.
Identity at the access edge — how the admin page is reached and who is let in.
Pools — requesting and reconfiguring the partner pool.