[ LOG_DATE: 2025-10-14 ]
#AI#LLM#SLM#Machine Learning#Research

Small Language Models: Concepts & Resources

A curated collection of concepts, papers, tools, and resources for learning about Small Language Models (SLMs) - covering efficiency, architectures, fine-tuning, quantization, and practical implementations.

0 views

Small Language Models — Concepts & Learning Resources

A learning note covering the concepts behind building a small language model that runs on phones and student laptops. Companion note: [[Project ATOM-SLM - Plan]].

How to use this note

Read top-to-bottom for the mental model, then use the Resources section to watch/read. Concepts marked [mobile] are the ones that matter most for running on small devices.

1. The big picture

A language model is just a function that predicts the next token given previous tokens. Everything else (chat, reasoning, math) emerges from that objective plus the data.

  • LLM = large (billions of params, server GPUs).
  • SLM (Small Language Model) = typically < 1B params, often 100M–500M, designed for on-device use.
  • Rule of thumb: capability scales with params × data quality × training tokens, not params alone. The TinyStories result (below) shows a ~10M-param model can write coherent English if the data is simple and it is trained long enough.

Key lesson for us: our first model (atom-slm) was undertrained and had a pathologically large vocabulary — a data + architecture problem, not only a size problem.

2. Tokens & tokenizers

  • Text is split into tokens (sub-word units) by a tokenizer (BPE / WordPiece / Unigram).
  • Vocabulary size = number of distinct tokens.
  • Trade-offs:
VocabSequence lengthEmbedding paramsNotes
32k–50klongersmallSLM sweet spot (SmolLM2 = 49,152)
100k+shorterlargemultilingual / better compression
262kshortesthugeGemma tokenizer — bad for tiny models
  • Embedding weight size = vocab × hidden. At 262,144 × 256 that is ~67M params of atom-slm’s 68.7M — the model was basically an embedding table with a tiny brain.
  • Tied embeddings: reuse the input embedding matrix for the output projection (saves vocab × hidden params). Gemma and most SLMs tie.

3. Embeddings

Each token maps to a vector of size hidden. Position matters, so a positional encoding is added (see RoPE below). Token embeddings are learned; “semantic” structure emerges from training.

4. The Transformer block

Each layer contains two sub-blocks, each wrapped with a residual connection and a normalization:

  1. Attention — mixes information across positions.
  2. Feed-forward network (FFN/MLP) — transforms each position independently.

Standard modern recipe (“pre-norm transformer”):

x = x + Attention(Norm(x))
x = x + FFN(Norm(x))

4.1 Self-attention (the core idea)

Each token produces a Query (Q), Key (K) and Value (V). Attention = weighted average of values, where weights come from softmax(Q·Kᵀ / √d).

  • Causal mask: a token may only attend to itself and earlier tokens (so it can’t see the future).
  • Cost scales O(n²) with sequence length n → long context is expensive.

4.2 Heads, MHA → GQA → MQA [mobile]

  • MHA: many Q heads, equally many K/V heads.
  • MQA: one K/V head (max saving).
  • GQA (Grouped-Query Attention): a few K/V heads shared by groups of Q heads — the modern default (Llama 3, Mistral, Gemma, Qwen).

Why it matters: at generation time the KV cache dominates memory for long context:

KV cache bytes ≈ 2 (K+V) × layers × n_kv_heads × head_dim × context × batch × bytes_per_val

Going from n_kv_heads = 8 to 2 cuts KV memory 4×. This is the single biggest long-context win for mobile.

4.3 RoPE and RoPE scaling [mobile]

Rotary Position Embeddings (RoPE) rotate Q/K by position-dependent angles. The max context is set during training.

To extend context:

MethodHowWhen
Train at target lengthtrain with longer packed sequencesbest for small models
Linear scalingdivide position index by factor squick, some quality loss
NTK-awareincrease RoPE base thetainference-time extension
YaRNNTK + attention temperature fixbest inference-time scaling; supported in llama.cpp

Important: tiny models degrade quickly with pure scaling — prefer continued pretraining at the target length (e.g., 4096), then optionally YaRN for a bit more.

4.4 FFN / MLP [mobile]

Two (or three, with SwiGLU) linear layers with a nonlinearity, expanding to ~2.5–4× hidden then back. Most parameters live here. SwiGLU (gated) is standard and worth ~1–2% quality.

4.5 Normalization & residuals

  • LayerNorm → RMSNorm: cheaper, no mean subtraction; standard in modern LLMs.
  • Pre-norm + residuals make deep stacks trainable.

4.6 Mixture-of-Experts (MoE) — and why NOT for mobile

MoE replaces each FFN with N expert FFNs + a router that sends each token to top-k experts.

  • Total params ↑ (all experts are stored); active params per token ≈ dense (only k compute).
  • Benefit: more capacity at ~same compute (FLOPs) → great for servers (Mixtral, DeepSeek-V3, Gemma-3 26B-A4B, Qwen3-MoE).
  • Memory cost: worse, not better. Every expert must be resident in RAM/VRAM to run (or be offloaded to disk = slow). Routing adds latency and needs load-balancing; tiny batches waste experts.

Verdict: for a phone/student-laptop SLM, use a dense model, not MoE. MoE is about compute efficiency at scale, not memory efficiency.

5. Depth vs Width (should I add more layers?) [mobile]

You have a fixed parameter budget; spend it on depth (more layers) vs width (bigger hidden/FFN).

More layers (depth)More hidden/FFN (width)
Addssequential computeparallel compute
Helps mainlycomposition / reasoningknowledge / memory
Trainingharder (needs pre-norm + residuals)easier
Inference speedslower (sequential), higher latencyattention/FFN cost grows ~O(d²)
Params per unit~linear~quadratic in hidden

Guidance:

  • For reasoning/math, depth helps more than width — so yes, more layers is usually the right call.
  • But don’t go too narrow: keep head_dim ≈ 64–128 (hidden / n_heads) and FFN ≈ 2.5–4× hidden.
  • Many strong small models are deep and moderately narrow (width/depth ratio ~20–35):
ModelLayersHiddenHeads / KVFFNVocabContext
SmolLM2-135M305769 / 3 (GQA)153649,1528192
SmolLM2-360M3296015 / 5256049,1528192
Qwen2.5-0.5B2489614 / 24864151,93632,768
TinyLlama-1.1B22204832 / 4563232,0002048

(Figures are approximate — treat as reference points.)

Balanced recommendation for a math SLM: ~135M params, 16–20 layers, hidden 768, GQA 12/3, FFN ~2048 (SwiGLU), vocab ~49k, train at 4096 context. That is deep-enough and wide-enough without bloat.

6. Training

  • Stages: pretraining (raw text, next-token loss) → SFT (supervised fine-tuning on Q/A or CoT) → optional alignment (DPO/RLHF) → optional distillation from a stronger teacher.
  • Loss: cross-entropy over next token. Perplexity = exp(loss).
  • Optimizer: AdamW; cosine LR schedule with warmup.
  • Precision: bf16 (if supported) or fp16 + GradScaler (T4/P100 only support fp16).
  • Efficiency: FlashAttention-2, gradient checkpointing, sequence packing.
  • PEFT: LoRA (train small adapters) or QLoRA (4-bit base + LoRA) — great for Kaggle.
  • Chinchilla rule: ~20 training tokens per parameter for pretraining (135M → ~2.7B tokens).

7. Quantization & deployment [mobile]

  • GGUF (llama.cpp) is the standard for CPU/mobile. Types: Q8_0, Q6_K, Q5_K_M, Q4_K_M (~4.5 bits/weight, the sweet spot), down to Q2_K.
  • Rough deploy sizes: 135M ≈ ~90–100 MB at Q4_K_M; 360M ≈ ~250 MB.
  • Other runtimes: MLC-LLM / WebLLM (GPU/WASM, browser), ONNX Runtime Mobile, ExecuTorch / MediaPipe LLM Inference.

8. Evaluation

  • Math: GSM8K, MATH. General: MMLU, HellaSwag, ARC. Harness: lm-eval-harness.
  • Track perplexity during training, but validate on task accuracy (e.g., exact-match on GSM8K).
  • Always compare against the base model before your fine-tune to prove the gain.

9. Resources

Foundational — read/watch first

Specific concepts

Courses / long-form

TED talks (context, ethics, the big picture)

Links are provided as search links where the exact URL may change over time; search the title if a link breaks.

10. Cheat-sheet (one line each)

  • Next-token prediction is the whole game.
  • Vocab size is a memory tax: keep it ~50k for tiny models.
  • GQA shrinks the KV cache (mobile long-context win).
  • RoPE scaling extends context, but train at target length for tiny models.
  • SwiGLU + RMSNorm + pre-norm = modern block.
  • Depth helps reasoning, width helps knowledge; keep head_dim 64–128.
  • MoE = compute savings, NOT memory savings → skip for mobile.
  • Q4_K_M GGUF is the mobile deployment target.
  • Data quality and enough tokens beat raw parameter count (TinyStories).
<-- RETURN_TO_ARCHIVE END_OF_TRANSMISSION