Scaling out

Scaling out

A single Weavy node is enough for many installations, but for higher availability or more capacity you can run two or more nodes behind a load balancer. All nodes run the same Weavy code, share the same database and use a backplane to deliver realtime events to clients connected to other nodes.

Requirements

Every node must be configured with the same:

Weavy:ApplicationUrl (URL of the load balancer, not of the individual node).
Weavy:License (the license key for Weavy).
ConnectionStrings:Weavy (all nodes must use the same shared database).
ConnectionStrings:SignalR or ConnectionStrings:Redis (must use the same realtime backplane).

If you have configured a blob provider other than the default database provider, all nodes must use the same storage, e.g. Azure Blob Storage container.

Load balancer

Weavy requires HTTPS. If the load balancer terminates TLS, it must send the X-Forwarded-Proto (and X-Forwarded-For/X-Forwarded-Host) headers.
Should forward all paths to the nodes, not only /api.
Enable WebSockets.
Do not expose the nodes directly. Weavy trusts the forwarded headers from any proxy, so the load balancer should be the only way to reach the nodes.

Health probes

Each node exposes two anonymous endpoints that can be used by load balancers and container orchestrators. Both return the node id and the Weavy version:

Endpoint	Description
`/live`	Always returns `200 OK` while the process is running. Use it as a liveness probe.
`/ready`	Returns `200 OK` when the node can receive traffic and `503 Service Unavailable` otherwise. Use it as a readiness probe.

Example: Response from /ready:

{ 
  "node_id": "weavy-02", 
  "version": "32.2.0", 
  "status": "healthy" 
}

The status is one of:

Status	`/ready`	Description
`healthy`	`200 OK`	Everything works.
`degraded`	`200 OK`	The node can receive traffic, but something needs attention. Either a dependency only works in part (e.g. blobs can be read but not uploaded), or an administrator must act: no valid license is installed or the database must be upgraded.
`unhealthy`	`503 Service Unavailable`	The node can't serve requests, e.g. because the database, Redis or Azure SignalR Service can't be reached, or the database schema is from a newer version of Weavy.
`unknown`	`503 Service Unavailable`	A dependency has not been checked yet.

A missing license or an outdated database schema affects every node, since they share the database. That's why those nodes are reported as degraded and stay in rotation, so that you can reach the admin area to install the license or upgrade the database. Until then, the node answers all other requests with 503 Service Unavailable. An unhealthy node also keeps serving the admin area as long as the database can be reached, so that you can read the logs.

Each node is identified by Weavy:NodeId, which defaults to the machine name (the container id in Docker). You can set it to something more descriptive. The node id is included in the log entries, so you can tell the nodes apart in Admin > Logs.

Database upgrades

When a new version of Weavy requires a newer database schema, its nodes are reported as degraded and answer 503 Service Unavailable to everything except /admin/database until the database has been upgraded. Either set Weavy:AutomaticMigrations to true to let the nodes upgrade the database on startup, or upgrade it from the admin area at /admin/database. Nodes recover automatically once the database has been upgraded.

Nodes still running the previous version report unhealthy once the database has been upgraded, so a readiness probe takes them out of rotation during a rolling upgrade. All nodes should run the same version of Weavy.

Realtime backplane

Weavy uses ASP.NET Core SignalR to push realtime events, such as new messages and notifications, to connected clients. A client is connected to a single node, so when the event happens on another node it has to be forwarded through a backplane. Without a backplane, clients only receive events that happen on the node they are connected to.

The backplane is selected by connection string:

Connection string	Backplane	Load balancer requirements
`ConnectionStrings:SignalR`	Azure SignalR Service	None. Clients are redirected to the Azure SignalR Service, so no sticky sessions are needed.
`ConnectionStrings:Redis`	Redis backplane (Redis or Valkey)	Sticky sessions (session affinity) are required.

If both are configured, Azure SignalR Service is used for realtime events. The Redis connection is still monitored by the health checks.

Sticky sessions are required with Redis because a SignalR client first sends a negotiate request and then opens the WebSocket connection with a second request, which must reach the same node. See Sticky sessions for details. Most load balancers support cookie based affinity, e.g. ARR affinity on Azure App Service and Application Request Routing on IIS.

As environment variables, e.g. in a Docker compose file, the settings look like this:

environment:
  - CONNECTIONSTRINGS__WEAVY=host=db;database=weavy;username=weavy;******;
  - CONNECTIONSTRINGS__REDIS=redis:6379
  - WEAVY__APPLICATIONURL=https://weavy.example.com

Split deployments

By default every node serves both the Web API (/api) and the admin area (/admin). With the Weavy__Modules setting you can create split deployments with for instance, two or more API nodes behind a public load balancer and a single admin node that is only reachable from your internal network:

  # public, behind the load balancer
  weavy-api:
    environment:
      - WEAVY__MODULES=Weavy.Api
  # internal, behind a firewall
  weavy-admin:
    environment:
      - WEAVY__MODULES=Weavy.Admin

Things to keep in mind:

Database upgrades (/admin/database) and license updates (/admin/license) are performed on the admin node. If WEAVY__AUTOMATICMIGRATIONS is not enabled, API nodes answer 503 Service Unavailable until the database has been upgraded from the admin node, after which they recover automatically.
Background jobs run on every node, including the admin node, so it needs the same outbound network access as the API nodes (e.g. for webhooks and AI providers).
API nodes only accept bearer tokens (API keys and access tokens), since the cookie authentication used by the admin area is not registered.
The load balancer should forward all paths to the API nodes, not only /api, since endpoints such as /hubs, /live, /ready, blobs and OAuth callbacks are served by every node.