If you build with AI, then you know cloud inferno drains your money. It limits what you can test, and it forces your data through someone else’s servers. There’s this tool called Exo, and Exo changes all that. Instead of paying forever, you use the hardware you already own—and instead of trusting one provider, everything stays local. This is about control, cost, and being able to experiment on your own. This is where things start to get interesting.
We have videos coming out all the time. Be sure to subscribe. So really, Exo is an open-source tool that lets you build a peer-to-peer AI cluster on everyday devices. I want you to think about MacBooks, Raspberry Pis, all that fun stuff. Now, you might assume that means manual networking and a lot of setup. That’s not actually what happens. Exo auto-discovers devices already on your local network automatically. The surprise is not that it actually works, but rather how little effort it takes to set up.
In this post, we’ll dive into how Exo makes it easy to build an AI cluster, pooling resources from your laptops, desktops, and even phones into a powerful distributed setup. We’ll cover its key features, setup, real-world benchmarks, and why it’s a game-changer for AI builders tired of skyrocketing cloud bills. Whether you’re a hobbyist tinkering with massive models or a developer seeking data privacy, Exo puts AI inference back in your hands—literally.
What is Exo? Your Home-Grown AI Supercomputer
Exo, developed by Exo Labs and available on GitHub, transforms a ragtag collection of consumer devices into a unified AI cluster. It’s licensed under Apache-2.0, completely open-source, and already boasts over 39,000 stars and 2,600 forks on GitHub. The core idea? Automatic device discovery, smart model sharding, and lightning-fast communication via RDMA over Thunderbolt—turning your setup into something that rivals cloud GPUs without the subscription fees.
Exo looks at your network and measures bandwidth, latency, and available memory. Then it decides how to split up the model. So instead of just forcing one machine to load everything, Exo shards the model across multiple devices using tensor and pipeline parallelism. This just means faster inference in shared memory. This is also basically how models like DeepSeek V3 become possible locally.
And this is where expectations break again. Most people just assume mixed hardware breaks performance. Okay, but for this, that’s not really the case. A MacBook Pro with an M4 Pro chip could potentially work alongside a Raspberry Pi—that would be on the CPU part. macOS GPUs run through Apple’s MLX framework, and Linux systems fall back to CPUs. It’s not about being perfect by any means. No, it’s not going to be. It’s more about pooling from what you already have. And the flexibility is insane. Plus, the fact that it’s blown up on GitHub and it seems people are really putting this to use.
Key Features of Exo for Seamless Local AI Inference
Exo’s magic lies in its hands-off approach to distributed computing. Here’s what makes it stand out when you build an AI cluster:
- Automatic Device Discovery: Nodes broadcast their presence on your local network—no manual IP configs or config files required.
- Topology-Aware Model Sharding: Exo scans resources in real-time (CPU/GPU memory, bandwidth, latency) and auto-selects the best parallelism strategy—tensor for speedups or pipeline for memory efficiency.
- RDMA over Thunderbolt 5: On supported Apple hardware, this enables direct GPU-to-GPU memory transfers, slashing latency by up to 99% and making devices act like a single system.
- Tensor Parallelism: Scale up to 1.8x faster on two devices or 3.2x on four, all while handling massive models that exceed single-device VRAM.
- MLX Backend Integration: Leverages Apple’s MLX for efficient macOS inference, with Linux CPU fallback (GPU support incoming).
- OpenAI-Compatible API: Query your AI cluster via a simple endpoint at
http://localhost:52415/v1/chat/completions—plug it into any app that supports OpenAI.
The setup sounds complex talking about it, but it was actually a lot easier than I thought. Once you see it happen, everything else makes sense.
How to Set Up Exo: From Zero to AI Cluster in Minutes
Getting started with Exo to build an AI cluster is surprisingly straightforward. My machine here isn’t connected to anything special—it’s a Mac M4 Pro. No config files, no IP addresses. I launch Exo, and that’s it. The first time you run it takes some time to set up, but then it’s good to go. I’m only on the M4 Pro here doing this, but if you have other machines, you’re going to see them start to pop up automatically. So, it’s all that real-time discovery happening in action.
Quick Start for macOS (Recommended for GPU Power)
- Download the macOS app (requires macOS Tahoe 15.2+). It runs in the background and auto-enables RDMA via system settings.
- Or build from source:
brew install uv macmon node
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup toolchain install nightly
git clone https://github.com/exo-explore/exo
cd exo/dashboard && npm install && npm run build && cd ..
uv run exo- Enable RDMA: Reboot into Recovery Mode and run
rdma_ctl enable.
Linux Setup (CPU-Focused)
sudo apt update && sudo apt install -y nodejs npm
curl -LsSf https://astral.sh/uv/install.sh | sh
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup toolchain install nightly
git clone https://github.com/exo-explore/exo
cd exo/dashboard && npm install && npm run build && cd ..
uv run exoNow, I’ll send a prompt using the OpenAI-style endpoint. Nothing crazy. Let’s just do the simple prompt: “Explain quantum computing in simple terms.” The point here isn’t speed. The point is that it already works. At this moment, the model is split across machines without me touching anything. Well, in my case, it’s one machine. If there’s more, it splits things up without touching things.
Access the dashboard at http://localhost:52415 to monitor your AI cluster and issue requests. Supported hardware includes Apple Silicon Macs (M-series) for full GPU glory, with Linux on x86/ARM CPUs. Windows? It’s on the roadmap.
Real-World Benchmarks: Proof That Local AI Scales
This is the part people don’t actually think about. On supported Apple hardware, Exo enables day-zero RDMA over Thunderbolt 5. All this means is GPU-to-GPU memory transfers without bouncing through the CPU. So, latency drops a lot, and the machines start behaving like a single system. This isn’t just theoretical.
Community benchmarks, like those from Jeff Geerling’s blog, show four M3 Ultra Mac Studios running Qwen 3 235B at around 32 tokens per second—a 3.2x speedup over a single device. Exo Labs themselves ran DeepSeek V3 671B on 8 M4 Mac Minis with 512 GB pooled memory.
| Model | Quantization | Devices | Speedup vs. Single Device | Tokens/Second |
|---|---|---|---|---|
| Qwen 3 235B | 8-bit | 4x M3 Ultra Mac Studio | 3.2x | ~32 |
| DeepSeek V3 671B | 8-bit | 8x M4 Mac Mini | 3.2x | N/A (Memory-Pooled) |
| Kimi K2 Thinking | 4-bit | 4x M3 Ultra | 3.2x | ~28 |
Mixed hardware works. Wired setups scale better than Wi-Fi. Power usage goes up, but it’s still small compared to cloud GPUs. Now, Exo proves something important here: Running serious AI locally is no longer unrealistic. It’s practical.
Why Ditch the Cloud? Control, Cost, and Privacy with Exo
Cloud services are convenient, but they come at a premium—especially when you’re iterating on AI models. Exo flips the script by letting you build an AI cluster, eliminating recurring bills and keeping sensitive data off third-party servers. No more vendor lock-in; just pure, scalable compute from your own gear.
That said, if you need burstable power beyond your local setup or a dedicated remote machine, a VPS (Virtual Private Server) could bridge the gap. It’s like a mini-cloud you control, but Exo keeps things even simpler and cheaper for everyday experimentation.
Well, that is if you just happen to have a few MacBooks laying around, which I get it—most people don’t. I don’t either. But you don’t need a cloud bill, and you don’t need to hand over your data. You just need the machines. Remember, it’s not perfect by any means, but for doing a fully local setup like this, it is pretty cool to see what we can do from pulling from our own resources.
Join the Exo Community and Start Building
With 65 contributors and growing, Exo’s GitHub repo is buzzing. Check out the CONTRIBUTING.md to add support for new hardware or tweak the dashboard. Feature requests? Thumb-up issues to vote.
Ready to say goodbye to those AI cloud bills? Head to the Exo GitHub and spin up your AI cluster today. We’ll see you guys in another.
