Rapid-MLX

Rapid-MLX is an open-source inference server that runs large language models locally on Apple Silicon Macs. It provides OpenAI and Anthropic-compatible APIs, allowing multiple applications like Cursor, Claude Code, and custom agents to query models concurrently on a single machine. The project emphasizes performance and reliability for tool calling in coding workflows.

Key features include continuous batching for concurrent requests, prompt caching with KV trimming for sub-100ms cached responses, and support for 231+ models across 15 families. Built on MLX, it achieves 261 tokens/second aggregate throughput across 4 concurrent streams on M3 Ultra hardware, with a desktop app and command-line interface available.