Introducing ByteShape’s Latest Device-Optimized Quantizations for Large Language Models — Compatible with All Hardware, from GPUs to Raspberry Pi
Welcome to ByteShape’s Performance-Driven LLM Optimization
At ByteShape, we specialize in creating highly efficient GGUF quantizations tailored for a broad range of hardware. Our approach emphasizes not only producing quantized models but also thoroughly evaluating their performance, enabling users to make informed decisions based on real tradeoffs between throughput and quality.
Revolutionizing Quantization with ShapeLearn
Traditional quantization techniques often rely on fixed formats or heuristics, which can lead to suboptimal performance or significant drops in model accuracy. Our core innovation, ShapeLearn, departs from this paradigm by utilizing a fine-tuning process that learns the most suitable data type for each tensor, optimizing for both speed (TPS) and fidelity. This method systematically avoids the pitfalls of “smaller but slower” formats and helps keep model quality on point without hitting accuracy cliffs.
Our Mission and Expertise
Developed by researchers from the University of Toronto, ByteShape’s goal is straightforward: empower the machine learning community with tools and insights that demystify quantization choices. To this end, we rigorously evaluate models across various hardware configurations—ranging from high-end GPUs to modest devices like the Raspberry Pi—often utilizing research-grade equipment to ensure robust testing.
For those interested in the technical foundation of our approach, we invite you to explore our recent publication at MLSYS: “Schrödinger’s FP”.
Available Models in This Release
- Devstral-Small-2-24B-Instruct-2512: Optimized for GPU enthusiasts with RTX 40/50 series cards.
- Qwen3-Coder-30B: Designed to perform efficiently across various hardware levels, from Raspberry Pi to high-end GPUs like the RTX 5090.
Simplified Download Options for Practical Use
While we offer detailed tradeoff analyses in our blog, here are our recommended default models for straightforward deployment:
For NVIDIA RTX 4080/4090/5090 Users:
- Devstral-Small-2-24B-Instruct-2512-IQ3_S-3.47bpw.gguf
Achieves approximately 98% of the baseline quality with a model size of 10.5 GB, fitting comfortably within 16GB GPU memory with 32K context.
For Qwen3-Coder 30B:
- GPU (16GB): Qwen3-Coder-30B-A3B-Instruct-IQ3_S-3.12bpw.gguf
- CPU (16GB RAM): Qwen3-Coder-30B-A3B-Instruct-Q3_K_M-3.31bpw.gguf
Both models deliver over 96% of the original quality and integrate seamlessly with 32K context windows on standard 16GB memory systems.
How to Access and Use
Our models are hosted on Hugging Face, but due to label-sharing constraints, specific filenames must be referenced directly. Here’s how to load models using Ollama, a popular interface for deploying LLMs:
bash
ollama run hf.co/byteshape/Devstral-Small-2-24B-Instruct-2512-GGUF:Devstral-Small-2-24B-Instruct-2512-IQ3_S-3.47bpw.gguf
Similarly, for Qwen3:
bash
ollama run hf.co/byteshape/Qwen3-Coder-30B-A3B-Instruct-GGUF:Qwen3-Coder-30B-A3B-Instruct-IQ3_S-3.12bpw.gguf
The same process applies when using other platforms such as llama.cpp.
Key Insights and Performance Considerations
- Quantization Cliffs: Our data reveals a “quantization cliff” around 2.30 bits per weight (bpw). Going beyond this threshold leads to exponential quality degradation unless carefully managed with techniques like ShapeLearn.
- Performance Floors: In some cases, increasing model compression (lower bpw) does not translate into higher throughput. Our models use routing strategies to bypass such bottlenecks, maintaining a favorable TPS-quality balance.
Transparency and Reproducibility
- Model benchmarking follows the llama.cpp b7744 implementation.
- All comparisons utilize a consistent template setup, including assessments with Unsloth methodologies.
- Models are evaluated with a minimum context size of 4,096 tokens to ensure compatibility and fair benchmarking.
Get Started Today
Explore the models and methodology further through our official links:
- Devstral GGUFs: Hugging Face
- Qwen3 GGUFs: Hugging Face
- In-depth Blog & Interactive Visualizations: ByteShape Blog
Note: Our GGUF files include a custom prompt template optimized for tool calling and parallel execution, enhancing performance in llama.cpp workflows.
Final Thoughts
ByteShape’s innovative quantization approach demonstrates that it’s possible to achieve high performance and quality across diverse hardware platforms. Whether you’re running large models on high-end GPUs or deploying on resource-constrained devices like the Raspberry Pi, our models and methodologies aim to empower your AI applications with efficiency and precision.
For questions, collaborations, or contributions, feel free to reach out via our website or GitHub. Stay tuned for more updates and experimental results!
