Artificial Intelligence (AI)

Microsoft Debuts 1-Bit Compact LLM that Runs on CPUs

Ken Metral

Ken Metral

— 2 min read
Cosmico - Microsoft Debuts 1-Bit Compact LLM that Runs on CPUs
Credit: Microsoft Corporation

In a groundbreaking leap toward efficiency in AI, Microsoft Research has introduced BitNet b1.58 2B4T, a highly compact large language model (LLM) that delivers full-scale performance with a fraction of the usual computational cost. Packing 2 billion parameters, this model operates using just 1.58 bits per weight, as opposed to the traditional 16 or 32-bit precision typically seen in large models.

Despite its radically compressed size, BitNet b1.58 2B4T delivers performance on par with leading full-precision models, and does so with remarkable efficiency across both GPU and CPU environments. It excels in a wide array of tasks—language understanding, math, coding, and conversational AI—thanks to training on a massive 4 trillion-token dataset.

Built for Speed, Scale, and Simplicity

The secret to BitNet’s efficiency lies in its architecture. Based on the Transformer backbone, the model incorporates major innovations via the BitNet framework. Standard full-precision linear layers are replaced with BitLinear layers, where weights are quantised to just 1.58 bits during the forward pass. These weights take ternary values: {-1, 0, +1}, using a technique called absmean quantisation.

On the activation side, the model uses 8-bit integers quantised per token with an absmax strategy, maximizing information density without compromising accuracy.

Key architectural choices include:

  • Subln normalisation for training stability
  • Squared ReLU (ReLU²) activation in feed-forward layers
  • Rotary Position Embeddings (RoPE) for positional encoding
  • Bias-free linear and normalisation layers, consistent with models like LLaMA
  • LLaMA 3-style tokenizer using byte-level BPE with a 128,256-token vocabulary

Efficient Training Pipeline

BitNet’s training follows a three-phase process:

  1. Pre-training on the extensive dataset
  2. Supervised fine-tuning (SFT) to align with task-specific objectives
  3. Direct Preference Optimisation (DPO) for aligning model outputs with human preferences

This disciplined approach ensures robust generalisation while maintaining high efficiency at inference.

Real-World Implications

According to Microsoft’s technical report, BitNet b1.58 2B4T offers “substantially reduced memory footprint, energy consumption, and decoding latency,” all while matching the capabilities of other state-of-the-art, open-weight LLMs. These traits make it highly practical for deployment, especially in environments with limited compute resources.

The model weights are now publicly available on Hugging Face, alongside open-source code, making it accessible for developers, researchers, and enthusiasts alike.

A Glimpse into the Future of AI

BitNet b1.58 2B4T underscores a powerful insight: Bigger doesn’t always mean better. With clever architectural changes and efficient quantisation strategies, it’s possible to build LLMs that are lean, fast, and remarkably capable.

As AI adoption accelerates, innovations like BitNet pave the way for more sustainable, energy-efficient, and inclusive AI technologies—making cutting-edge capabilities available beyond just big tech labs and into everyday applications.

Read more