Outpost Performance Tuning

Outpost Performance Tuning

The Arbitex Hybrid Outpost is designed for low-latency, high-throughput policy enforcement. This guide covers the four primary levers for tuning outpost performance: connection pool sizing, response caching, graceful drain configuration, and load testing methodology for establishing production performance baselines.

---

Hardware requirements

The Outpost now delivers the same advanced PII detection as Arbitex Cloud — the Presidio + GLiNER NER backend with structured validators (passport, SSN, IBAN, Luhn) and zero-shot entity expansion — on-premises and air-gap-ready, on a GPU appliance you control.

GPU

A GPU is required for production and air-gap Outpost deployments. The NER backend (Presidio + GLiNER) and the DeBERTa Tier 3 contextual validator both run GPU inference at the throughput Outpost deployments target.

Tier	GPU	Notes
Production (recommended)	NVIDIA L4 or A10 — or equivalent with ≥ 24 GB VRAM	Recommended for sustained throughput
Production minimum	NVIDIA T4 (16 GB VRAM)	Acceptable for low-throughput sites
Below T4	Not supported	Provision at least a T4-class GPU

A CPU-only profile remains available for development and evaluation only — it is not a supported production or air-gap target.

CPU and memory

CPU and RAM requirements are unchanged from previous releases. Size them per the sizing guidance for your expected request rate — the GPU handles model inference while CPU and memory continue to serve connection pooling, caching, audit buffering, and policy evaluation.

---

Connection Pool Configuration

The outpost maintains a persistent HTTP connection pool to the Arbitex platform for policy lookups and telemetry upload. Proper pool sizing prevents connection exhaustion under peak load while avoiding unnecessary resource consumption during quiet periods.

How the pool works

Each time the outpost evaluates a policy or uploads audit telemetry, it draws a connection from the pool. If an idle connection is available it is reused immediately (a keepalive reuse). If no idle connection is available and the pool has not reached PLATFORM_MAX_CONNECTIONS, a new connection is created. If the pool is at capacity, the request waits in queue until a connection is released.

On startup the outpost pre-establishes PLATFORM_MAX_KEEPALIVE / 2 connections to the platform. This warmup avoids cold-start latency on the first burst of incoming traffic after a deployment.

Environment variables

Variable	Default	Range	Description
PLATFORM_MAX_CONNECTIONS	50	10–500	Maximum number of concurrent connections to the platform. Requests exceeding this limit wait in queue.
PLATFORM_MAX_KEEPALIVE	20	5–200	Maximum number of idle keepalive connections held open between requests. Must be ≤ PLATFORM_MAX_CONNECTIONS.
PLATFORM_KEEPALIVE_EXPIRY	60	10–300	Idle keepalive connection timeout in seconds. Connections idle longer than this value are closed and removed from the pool.

Tip

Size PLATFORM_MAX_CONNECTIONS to approximately 2× your expected peak concurrent requests. Policy evaluation is fast (sub-millisecond for cache hits, tens of milliseconds for platform round-trips), so connections are held for short durations. A 2× headroom ensures the pool does not become the bottleneck even during traffic spikes.

Pool metrics endpoint

Retrieve live connection pool statistics from the outpost admin API:

GET /admin/api/metrics/connections
Authorization: X-API-Key <admin-key>

Response — ConnectionMetrics:

Field	Type	Description
active_connections	int	Connections currently in use by an in-flight request.
idle_connections	int	Connections held open and available for immediate reuse.
max_connections	int	The effective value of PLATFORM_MAX_CONNECTIONS.
total_created	int	Total connections created since process start (cumulative).
total_reused	int	Total keepalive connection reuses since process start (cumulative).
pool_wait_count	int	Number of requests that had to wait for a free connection since process start.
pool_wait_duration_ms	float	Average wait time in milliseconds for requests that queued for a connection.

Example response:

{
  "active_connections": 12,
  "idle_connections": 8,
  "max_connections": 50,
  "total_created": 1847,
  "total_reused": 94302,
  "pool_wait_count": 3,
  "pool_wait_duration_ms": 2.4
}

A non-zero pool_wait_count that grows over time indicates that PLATFORM_MAX_CONNECTIONS is too low for the current traffic load. Increase the value and monitor until pool_wait_count stops accumulating. A high total_reused relative to total_created confirms keepalive connections are working effectively.

---

Response Caching

The outpost can cache policy evaluation responses in memory to reduce platform round-trips for repeated identical requests. Caching is most effective for workloads where many requests share the same authentication context, path, and request shape — for example, API gateway traffic where a small set of routes is called repeatedly by many clients.

How the cache works

Before forwarding a policy evaluation request to the platform, the outpost computes a cache key from the normalized request:

1. HTTP method (uppercased)
2. Request path (normalized)
3. Headers included in policy evaluation (as configured in policy rules)
4. SHA-256 hash of the request body (if present)
5. Query parameters sorted alphabetically before hashing

The full set of inputs is hashed with SHA-256 to produce a fixed-length cache key. If a valid (non-expired) cache entry exists for that key, the cached policy response is returned immediately without contacting the platform.

Environment variables

Variable	Default	Range	Description
POLICY_CACHE_ENABLED	true	—	Enable or disable the in-memory policy response cache. Set to false to disable caching entirely.
POLICY_CACHE_TTL_SECONDS	300	30–3600	Time-to-live for cached policy responses in seconds. After this period, the cached entry is treated as expired and the next request for that key will contact the platform.

Caution

After updating a policy in the Cloud Portal, the outpost may continue serving the previous policy response for up to POLICY_CACHE_TTL_SECONDS. In most environments the default 5-minute TTL is acceptable. For sensitive policy changes that require immediate enforcement — such as revoking access or tightening DLP rules — either call the cache clear endpoint immediately after the update or lower POLICY_CACHE_TTL_SECONDS to a shorter interval for that deployment.

Cache statistics endpoint

GET /admin/api/cache/stats
Authorization: X-API-Key <admin-key>

Response — CacheStats:

Field	Type	Description
enabled	bool	Whether the policy cache is currently enabled.
entry_count	int	Number of entries currently held in the cache.
hit_count	int	Total cache hits since process start (cumulative).
miss_count	int	Total cache misses since process start (cumulative).
hit_rate_percent	float	Rolling hit rate: hit_count / (hit_count + miss_count) × 100.
eviction_count	int	Total entries evicted due to TTL expiry since process start.
memory_bytes	int	Approximate memory used by the cache, in bytes.
ttl_seconds	int	The effective value of POLICY_CACHE_TTL_SECONDS.

Example response:

{
  "enabled": true,
  "entry_count": 412,
  "hit_count": 187430,
  "miss_count": 21804,
  "hit_rate_percent": 89.57,
  "eviction_count": 19312,
  "memory_bytes": 2097152,
  "ttl_seconds": 300
}

Tip

Monitor hit_rate_percent as part of regular outpost observability. A healthy cache should maintain a hit rate above 80% under normal traffic patterns. A hit rate below 50% suggests one of three conditions: request patterns are highly unique (many different paths or headers), POLICY_CACHE_TTL_SECONDS is too short relative to the request repeat interval, or the cache has been cleared recently. Investigate before increasing POLICY_CACHE_TTL_SECONDS, as a longer TTL widens the window between policy updates and enforcement.

Cache clear endpoint

Use the cache clear endpoint to force immediate re-evaluation of all cached policy decisions — for example, immediately after a policy update:

POST /admin/api/cache/clear
Authorization: X-API-Key <admin-key>

Response:

{
  "cleared": true,
  "entries_removed": 412
}

entries_removed reflects the number of entries that were in the cache at the time of the clear. After clearing, the next request for any previously cached key will contact the platform, which causes a brief spike in platform round-trips. In high-traffic environments, consider clearing during a low-traffic window or in conjunction with a rolling restart.

---

Graceful Drain

When the outpost process receives SIGTERM — from a Kubernetes pod termination, systemctl stop, or a manual signal — it enters drain mode rather than terminating immediately. Drain mode ensures in-flight requests are completed before the process exits, preventing request failures during planned deployments and restarts.

Drain sequence

1. Stop accepting new connections. The outpost immediately closes its listen socket. New connection attempts are refused. Upstream load balancers should detect this via health check failure (GET /health returns non-200 or times out) and stop routing new traffic to the draining instance.
2. Finish in-flight requests. All requests currently being processed continue to completion.
3. Wait for in-flight requests to drain. The outpost waits up to DRAIN_TIMEOUT_SECONDS for all in-flight requests to finish.
4. Forceful exit. If in-flight requests have not all completed by the timeout, the outpost forcefully closes remaining connections and exits with code 0. Any requests that did not complete are abandoned at the client level.

Environment variables

Variable	Default	Range	Description
DRAIN_TIMEOUT_SECONDS	30	5–120	Maximum time in seconds to wait for in-flight requests to complete after receiving SIGTERM before forcefully terminating.

Tip

Set DRAIN_TIMEOUT_SECONDS to at least your p99 request processing latency plus a reasonable safety margin (e.g., p99 + 10 seconds). For most outpost deployments, where uncached requests complete in under 50ms, the default of 30 seconds provides ample headroom. If your traffic includes long-lived streaming requests or unusually high platform latency, increase the value accordingly.

Drain status endpoint

Poll this endpoint to monitor drain progress during a rolling restart:

GET /admin/api/drain/status
Authorization: X-API-Key <admin-key>

Response — DrainStatus:

Field	Type	Description
draining	bool	Whether the outpost is currently in drain mode. false under normal operation.
started_at	ISO-8601 string | null	Timestamp when drain mode was entered, or null if not draining.
in_flight_requests	int	Number of requests currently being processed. Decreases toward 0 as drain completes.
drain_timeout_seconds	int	The effective value of DRAIN_TIMEOUT_SECONDS.

Example — active drain:

{
  "draining": true,
  "started_at": "2026-03-15T14:22:08Z",
  "in_flight_requests": 3,
  "drain_timeout_seconds": 30
}

Example — normal operation:

{
  "draining": false,
  "started_at": null,
  "in_flight_requests": 0,
  "drain_timeout_seconds": 30
}

Zero-downtime rolling restart

Use the following procedure to restart or update outpost replicas without dropping requests. This requires at least two outpost replicas behind a load balancer.

1. Send SIGTERM to one replica. The replica enters drain mode and stops accepting new connections. The load balancer routes all new traffic to the remaining healthy replicas.
2. Wait for drain to complete. Poll GET /admin/api/drain/status on the draining replica and wait for in_flight_requests to reach 0. Alternatively, wait DRAIN_TIMEOUT_SECONDS to guarantee the process has exited.
3. Start the replacement replica. Deploy the new replica and wait for its health check to return HTTP 200 on GET /health. Do not proceed until the health check passes — the new replica is not yet in the load balancer rotation.
4. Confirm the new replica is healthy. Once GET /health returns 200, the load balancer will begin routing traffic to it. Verify connections are being served by checking GET /admin/api/metrics/connections on the new replica.
5. Repeat for the next replica. Proceed to the next instance and repeat from step 1.

This procedure maintains service continuity throughout the update. The minimum number of healthy replicas available at any point is N − 1, where N is the total replica count. For environments that require stricter availability guarantees, complete the drain and replacement of one replica before starting the next.

---

Performance Benchmarking

Establishing performance baselines before production deployment allows you to detect regressions early, size infrastructure correctly, and set realistic SLA expectations.

Baseline test approach

Run load tests against a staging outpost configured to mirror production: same PLATFORM_MAX_CONNECTIONS, POLICY_CACHE_TTL_SECONDS, and network topology relative to the Arbitex platform region.

Recommended tooling: k6, wrk, or vegeta. All three can generate sustained HTTP load and report latency percentiles.

Test procedure:

1. Start with a low baseline rate (10 rps) and confirm error rate is 0%.
2. Ramp in steps (10 → 50 → 100 → 250 → 500 rps, or to your expected peak).
3. Hold each step for at least 60 seconds to allow the cache and connection pool to reach steady state.
4. Record p50, p95, and p99 latency plus error rate at each step.
5. Note the load level at which any of the performance targets listed below are first exceeded — that is your current capacity ceiling.

Example k6 ramp script:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  stages: [
    { duration: '60s', target: 10  },
    { duration: '60s', target: 50  },
    { duration: '60s', target: 100 },
    { duration: '60s', target: 250 },
    { duration: '60s', target: 500 },
    { duration: '30s', target: 0   },
  ],
};

export default function () {
  const res = http.post(
    'https://outpost.example.internal/v1/proxy/completions',
    JSON.stringify({ model: 'gpt-4o', messages: [{ role: 'user', content: 'Hello' }] }),
    { headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer <token>' } }
  );
  check(res, { 'status 200': (r) => r.status === 200 });
}

Performance targets

Metric	Target	Notes
p99 latency (cache hit)	< 2 ms	Policy response served from in-memory cache; no platform round-trip.
p99 latency (cache miss)	< 50 ms	Policy response requires platform round-trip. Varies by region and platform load.
Error rate (normal load)	< 0.01%	Errors during normal load indicate connection pool exhaustion or platform availability issues.
Connection pool utilization	< 80% of PLATFORM_MAX_CONNECTIONS	Sustained utilization above 80% risks pool exhaustion under traffic spikes.
Cache hit rate	> 80%	Below 50% indicates unusual request diversity or TTL misconfiguration; see Response Caching.
Drain completion time	< DRAIN_TIMEOUT_SECONDS	In-flight requests should complete well within the configured drain window under normal load.

Caution

Always run benchmarks in a staging environment that mirrors production network topology as closely as possible. The p99 latency for uncached policy responses is dominated by the round-trip time between the outpost and the Arbitex platform, which varies by geographic region. A benchmark run in the same cloud region as the platform will show materially lower uncached latency than a benchmark run cross-region or on-premises. Size your capacity and set your SLAs based on the topology that matches your production deployment.

Interpreting results

Pool wait count increasing: pool_wait_count in GET /admin/api/metrics/connections increasing monotonically under load means PLATFORM_MAX_CONNECTIONS is undersized. Increase it by 25–50% and retest.

Cache hit rate below target: If hit_rate_percent from GET /admin/api/cache/stats is below 80%, review whether request headers or body content vary significantly across requests. Requests with high cardinality body content (e.g., unique per-user prompts) will not benefit from caching at the policy evaluation layer.

Error rate spike at high load: A sudden error rate increase at a specific load level (rather than a gradual rise) usually indicates a hard resource limit: connection pool exhaustion, platform rate limiting, or file descriptor limits on the outpost host. Check active_connections against max_connections in the pool metrics and review outpost container resource limits.

p99 latency exceeds cache miss target: If uncached latency consistently exceeds 50 ms, investigate network path latency between the outpost and the platform using standard tools (traceroute, mtr). Consider deploying outpost replicas in the same cloud region as the platform or enabling a platform edge endpoint closer to the outpost deployment site.

---

See Also

• Outpost Administration — budget enforcement, CredInt, health monitoring, JWT validation, PVC recovery, security hardening, SIEM direct integration