Staff Software Engineer, AI Inference Gateway (Remote, US)

We are looking for a Staff Software Engineer with 7+ years of experience to architect and build a new AI inference gateway product from the ground up at Compute Exchange. You'll be the technical lead for this greenfield effort, designing how inference requests are routed across a network of GPU providers, optimizing for cost, latency, and reliability using proprietary performance data. This is a high-ownership role ideal for a former founder or entrepreneurial engineer who thrives in small, fast-moving teams. You'll work alongside two experienced engineers on a lean, AI-native development team where architects drive fleets of coding agents and ship production code in weeks, not quarters.

What will you be doing?

• Architect the inference gateway end-to-end, request routing, load balancing, failover, and graceful degradation across heterogeneous GPU backends from ~100 providers

• Build the routing logic that uses proprietary cost and performance data to intelligently direct traffic based on customer-configurable parameters (speed vs. Cost)

• Own the metering, usage accounting, and enterprise billing layer, tracking tokens and compute time accurately enough to bill on, with all the B2B commercial infrastructure around it

• Establish observability, reliability, and multi-tenancy foundations including auth, API key management, rate limiting, and per-tenant cost attribution

• Drive the product from proof-of-concept to market, wearing multiple hats in a startup environment and helping shape the product direction with minimal guidance