Why Local Models Are Overrated
Open-weight models are essential for AI ecosystem competition, but the hype around running them locally on consumer hardware is delusional. Models like GLM-4.2 require 400GB+ VRAM, cost $75k+ in hardware, and consume $2k/year in electricity alone. The real value of open-weight models is cloud hosting competition, not local deployment.
The Gap Between Downloadable and Runnable
Open-weight models are not local models
Models like GLM-4.2 are incredible achievements but require 400GB of VRAM in full precision (1.5TB in BF16 format) to run properly. Even quantized versions need 200GB+, making them impossible to run on any consumer hardware. Downloading a model is not the same as running it.
Unified memory is the only viable consumer path
Only three realistic consumer options exist: MacBooks with 128GB unified memory (now $3,000 extra), DGX Spark (impractical), or Strix Halo laptops/Framework desktop. MacBook prices increased 30-50% for higher RAM tiers, making them prohibitively expensive.
GPU VRAM bottleneck kills performance
A model fitting on both a MacBook and RTX 5090 runs 3x faster on the 5090. But with Deep Seek V4 Flash, the 5090 becomes bottlenecked because it lacks sufficient VRAM, forcing the model to run on system RAM instead. This creates an impossible choice: high-compute low-VRAM or low-compute high-VRAM.
Hardware Costs Are Prohibitive
Enterprise GPU pricing is absurd
RTX 5090 costs $4,300 on resale. RTX 6000 Pro with 96GB VRAM costs $13,000 (3x more expensive for identical chip, just more VRAM). A four-GPU box capable of barely running GLM-4.2 costs $75,000. These are better viewed as resale investments than functional purchases.
Electricity costs are substantial
Running an RTX 5090 24/7 in San Francisco costs approximately $5 per day, totaling $2,000 per year in electricity alone. This is just one GPU; larger deployments multiply this cost significantly. Ownership doesn't mean free operation.
Parallelism and Workflow Scaling
Real dev work requires running multiple models simultaneously
Actual workflows fluctuate between 1 and 40 parallel agents. A developer might run two threads in an IDE, spawn sub-agents to explore different code paths, or use multiple models for different tasks (orchestrator + specialized workers). Local hardware cannot scale this way; cloud infrastructure handles it trivially.
Idle hardware is wasted money
If you buy GPUs for five parallel agents but only run three, you're paying for unused capacity. If you need six agents and only have five lanes, you're blocked waiting for a workload to complete. This creates either over-provisioning waste or under-provisioning bottlenecks.
Vision and multimodal capabilities are missing
GLM-4.2, despite being an incredible coding model, has no vision capability. It cannot accept screenshots or see what it created. State-of-the-art open-weight models lack the multimodal features needed for UI work and code review feedback loops.
The Efficiency Problem
Open-weight models burn more tokens than frontier models
GLM-4.2 at $3 per million output tokens seems 10x cheaper than Opus at $25, but Opus is far more efficient. A typical Opus run costs $8 with Deep Seek while GLM-4.2 costs $4, only 2x cheaper. GLM-4.2 generates 3x more tokens to reach the same answer, negating the per-token price advantage.
Speed doesn't matter when token count triples
Even if GLM-4.2 generates output 20% faster, it requires 3x more tokens to reach the same answer. The speed gain is completely negated by the token generation overhead. This is why 3.5 Flash fails despite being fast.
GLM-4.2 is the exception, not the rule
GLM-4.2 Max is slightly cheaper than Gemini 3.5 and slightly smarter, making it a genuinely competitive option. However, GPT-5.5 on medium or low settings is neck-and-neck with GLM-4.2 while being cheaper despite higher per-token costs, due to superior efficiency.
Mobile and Edge Devices Won't Catch Up
Budget phone performance has stalled
High-end Android phones show modest improvements over 3 years, but budget Android phones ($200) have seen zero meaningful performance gains since 2022, with slight regressions. This trend will worsen in 2026 due to price hikes and manufacturing constraints.
Phone inference destroys battery and causes overheating
Running heavy models on phones drains batteries egregiously and causes thermal throttling. While on-device inference for lightweight tasks like message summaries is valuable for privacy, it's impractical for serious workloads.
Secure compute is the real privacy solution
Apple's on-device processing for lightweight tasks and secure compute enclaves offer genuine privacy benefits without requiring local model inference. These approaches are more practical than betting on device performance improvements.
The Real Value of Open-Weight Models
Competition in cloud hosting is the actual benefit
Open-weight models enable providers to compete on price and performance. OpenRouter shows this clearly: GLM-4.2 has multiple hosting options at different price points (Wafer Fast at $1.025/M tokens and 115 TPS, Deep Infra at $0.44/M tokens and 30 TPS). Closed models like Opus have no such competition.
Cloud hosting solves all local model problems
Running open-weight models on cloud providers eliminates hardware costs, electricity costs, parallelism limits, and scaling bottlenecks. The only trade-off is privacy (data sent to servers), which secure compute will eventually mitigate.
Token efficiency matters more than per-token price
When choosing between models, focus on total cost per task, not per-token pricing. A model that costs $0.03/M tokens but uses 3x more tokens is more expensive than one costing $0.25/M tokens but using 1/3 the tokens.
The Nuanced Position
Open-weight models are essential for ecosystem health
Without competitive open-weight developments like Deep Seek R1, the industry has no incentive to improve. Open-weight models are crucial for preventing monopolistic control by Anthropic and OpenAI.
Small models have legitimate use cases
Not every task requires frontier models. Lightweight models like Gemini 3.5 on phones for message summaries or weather summaries are genuinely useful. The problem is not small models; it's pretending frontier models can run locally.
Frontier models keep getting bigger
Models like Fable are estimated at 2-10 trillion parameters, far exceeding GLM-4.2. The trend is toward larger, more capable models that will never run on consumer hardware.
Notable quotes
Just because a model is open weight does not mean you can run it at home. — Theo
Open weight models really shine. They allow for competition in the hosting space. — Theo
I just think it's delusional to pretend anyone's going to run good frontier open weight models on their own hardware. — Theo