Edge0 has released separate open-source inference engines for iOS, macOS, Android and Windows, extending its on-device large-model work beyond the Mac systems used in its technical report.
The September 30 release is four platform-specific projects, not one shared runtime. Edge0 says a common inference framework is still on its Q4 2026 roadmap.
The paper starts with a 35-billion-parameter mixture-of-experts (MoE) model. It routes each token through only a subset of specialist blocks, leaving about 3 billion parameters active at a time; the full 4-bit checkpoint still occupies 19.5 GB. Edge0 keeps those weights on local storage and loads selected blocks as needed, instead of requiring all of them in working memory.
A small trained component called the prerouter predicts which blocks the next layer will need one token ahead. That lets storage reads start while the current token is being processed. The prediction becomes the model's route, and a separate low-rank recovery adapter is trained to reduce quality loss from 4-bit weights and the changed routing.
On a Mac mini M4 Pro with 24 GB of memory, the report measures 20.4 generated tokens per second with 2.9 GiB of peak active memory. A standard mlx-lm run on the same machine measured 3.9 tokens per second and 18.2 GiB. Across five public benchmark scores on a 0-to-100 scale, Edge0 averaged 3.9 points below the original model in fp16. These are the authors' results, not an independent replication; active memory is not the size of the model files.
The phone tests use different setups. Edge0's iPhone 16 Pro test reports 6.4 tokens per second for the 35B model, a 2.1-second warm wait for the first token, and a built app of about 20 GB. Its Android reference tablet, with a Snapdragon 8 Elite chip and 16 GB of RAM, reports 6–9 tokens per second in the app; the first response on a new topic takes about 10–15 seconds when the device has to fetch cold model data. The model formats and measurements differ, so these figures are not a head-to-head comparison.
The release gives developers four codebases to inspect and test, not ready-to-install apps for ordinary phones. The paper's engine handles one request at a time and remains sensitive to storage reads, caching and CPU-side work. Wider independent device tests and a working shared runtime are still needed to show whether the approach is reliable beyond these development setups.