oMLX

What it is

oMLX runs large language models locally on Apple Silicon and manages the whole thing from your Mac's menu bar. It serves an OpenAI-compatible endpoint at http://localhost:8000/v1, so any client that speaks that API — coding agents included — can point at your own machine instead of a hosted provider.

The caching is the point

Plenty of tools will run a model locally. The problem with using them for actual work is that every turn re-processes the prompt, and long agent conversations become unusably slow. oMLX persists its KV cache across two tiers, hot in memory and cold on SSD, and keeps past context reusable across requests even when the context changes mid-conversation. Combined with continuous batching, that is what moves local inference from a demo to something you can leave a coding agent pointed at.

The project's stated motivation is exactly this gap: existing local servers made you choose between convenience and control, with no way to pin everyday models in memory, auto-swap heavier ones on demand, and cap context — all from a menu bar.

What it serves

Text LLMs, vision-language models, OCR models, embeddings and rerankers, discovered automatically from subdirectories of your model folder. An admin dashboard at /admin handles real-time monitoring, model management, chat, benchmarking and per-model settings, localised into eight languages, with CDN dependencies vendored so it works fully offline. There is a built-in chat UI too, and documented integrations for OpenClaw, OpenCode, Codex, Hermes Agent and Copilot. Source builds add an experimental multi-Mac mode that splits one model across machines with unequal memory over Thunderbolt or Ring RDMA.

Requirements and the one real gotcha

macOS 15.0 (Sequoia) or newer, Apple Silicon (M1 through M5), and Python 3.11-3.13. Install by dragging the signed .dmg from Releases — which also drops a small CLI shim so terminal commands and Apple Shortcuts can drive the app — or via Homebrew (brew tap jundot/omlx), or from source.

The gotcha: a plain pip install -e . does not build the native Metal custom kernels, and affected model families then fall back silently to much slower generic paths. The project measures GLM-5.2 fused DSA prefill at roughly 845 tok/s with the kernels versus about 29 tok/s without on an M3 Ultra, with higher memory use in the fallback. Building them needs the full Xcode Metal toolchain; the official DMG ships them precompiled, which is the easiest route if you serve GLM, MiniMax or Qwen models. Verify with python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_status())".

Free and open source under Apache 2.0. Latest release 0.6.4 (August 2026), with in-app auto-update in the macOS build.