Stillwater

The description states that Stillwater:

• Turns routed MoE experts into a storage-backed execution backend for Hugging Face Transformers.
• Uses dense indexed banks to store immutable expert weights.
• Keeps shared model scaffolds resident in memory.
• Executes selected experts from storage via memory mapping and SQLite cataloging.
• Supports BF16, FP8, grouped, and packed expert formats.
• Includes an OpenAI-compatible chat-completions server and streaming support for OpenWebUI.

Inference The product is a software runtime that enables inference of large MoE models using local disk storage instead of full RAM residency. It is not a commercial product or service but a technical prototype built by one developer.

The author claims:

• Stillwater addresses the impracticality of building custom silicon for LLM inference.
• The core idea is to use ordinary hardware to simulate part of an accelerator architecture in software.
• It aims to enable low-cost, single-user inference appliances for schools, libraries, and rural environments.

Inference The positioning has evolved from a speculative hardware question (could a hobbyist build an ASIC?) to a practical software solution that makes otherwise impossible model execution feasible on limited hardware. However, the description does not indicate any commercial or productized version of this idea.

The author states:

• The long-term goal is to create low-cost, single-user inference appliances for schools, libraries, and rural environments.
• These users may lack access to cloud inference or large GPU systems.

Inference The target customer segment appears to be individuals or institutions with limited computational resources who want to run large language models locally. However, no evidence of actual customers or user feedback is provided.

The description does not contain any information about:

• Revenue streams
• Pricing models
• Commercial partnerships
• Monetization strategy

Not evidenced

The author reports:

• Dense, append-only expert-bank formats.
• SQLite-based record catalog.
• Memory-mapped expert execution.
• Configurable drop, bounded-residency, and retain policies.
• Support for multiple model families (Qwen3.6-35B-A3B, GLM-5.2, etc.).
• BF16, FP8, grouped, and packed formats.
• Exact checkpoint reconstruction tests.
• OpenAI-compatible API endpoint.
• Live dashboard with telemetry.

Inference The technical implementation is sophisticated, involving memory mapping, SQL-based indexing, and model-specific adapters. It demonstrates a working prototype capable of running large models on constrained hardware.

The description states:

• A proof-of-concept existed before Build Week.
• During Build Week, it was turned into a more general runtime and deployment stack.
• Demonstrated exact semantic parity across several models.
• Successfully executed models up to 1.5 TB using only small fractions of that as resident memory.

Not evidenced No evidence of revenue, customers, or adoption beyond the author’s own development efforts.

The description does not mention:

• Competitors
• Market positioning relative to existing tools
• Prior art or similar projects in the space

Not evidenced

Key risks and red flags based on the self-reported information:

• The project is a single-person effort with no team or external validation.
• No evidence of commercial viability, traction, or customer feedback.
• Performance limitations are acknowledged (e.g., CPU inference is slow).
• The system relies heavily on Linux page-cache behavior and memory mapping — not suitable for production-scale use without significant optimization.
• The author explicitly notes that the project was not designed as a speed benchmark but to show feasibility.

Inference This is a technical prototype, not a commercial product. It lacks scalability, performance optimization, and any indication of market readiness or demand.

1. What are your plans for scaling this beyond a single-person development effort?
2. Have you tested the system with real users or in production-like environments?
3. Are there any known limitations or trade-offs that would prevent adoption at scale?
4. How do you plan to monetize or commercialize this technology if at all?
5. What are the performance implications for different hardware configurations (e.g., SSD vs HDD)?
6. Is there a roadmap for improving speed and efficiency beyond current CPU-only execution?