Scaling Has More Than One Dial: Jie Tang on Why Parameter Count Isn't Enough

⬅️ Back to Articles

Jie Tang, the Tsinghua professor behind Z.ai’s GLM line, posted “Thoughts About Scaling Law” on X as a course correction. His core claim: parameter count is only meaningful alongside how much data you have, where you spend compute, and who runs the model under what conditions. The post doubles as the release note for GLM-5.3, which kept every dial fixed except post-training.

  1. Parameter count is one number in a four-variable equation. Data volume, compute allocation, and deployment conditions define what those parameters are worth.

  2. The 2.7:1 rule was the field’s most expensive mistake. Kaplan et al. (2020) said grow parameters roughly 2.7x faster than data; GPT-3, Gopher, and MT-NLG complied. Hoffmann et al. (2022) reran it across 400 models and found the compute-optimal split is closer to 20 tokens per parameter, both growing at the same rate. The error compounded with each order of magnitude, so the biggest models of that generation were the most misallocated.

  3. Chinchilla optimized for training, not for life. It assumed a model trained once and evaluated once. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference in the objective and the optimum moves toward smaller models trained far longer: Llama-2-7B at roughly 290 tokens per parameter, Gemma-2-9B at 889.

  4. MoE splits capability into two. Total parameters roughly set how much a model can hold; activated parameters and effective depth set how far it can think. A dense 20:1 ratio does not transfer. At fixed tokens-per-parameter, adding total parameters can degrade reasoning while activating more experts reliably helps.

  5. Advanced skills are not retrieval. Finding a software vulnerability is not about memorizing more CVEs; it is carrying a twenty-step inference chain to the end. That capability does not live in total parameter count.

  6. GLM-5.3 is the cleanest test of this view. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of RL on long-horizon environments, and the gains are not marginal. An ML-infrastructure task drops the model into an engineer’s environment (compute clusters, storage, docs, codebases) and demands a measurable end-to-end speedup.

  7. Synthetic all the way down. Scaling post-training needs many environments that are executable, verifiable, and close to real work. Z.ai synthesizes them end to end: research agents turn real task patterns into long-horizon environments with hidden state, a judge agent verifies each is solvable, and verifiers are built without the reference solution.

  8. There are five knobs, not one: base model size, pretraining data, compute per forward pass, post-training, and MoE sparsity. They don’t have to turn together, and “the one worth turning next is rarely the one that was worth turning last.” Tang bets that leads to an open-weights Fable-class model by year’s end: two 2-3T models (Qwen 3.8 Max, Kimi K3) are out, and Fable is estimated at 3-7T, two points ahead on the AA index.

The takeaway: When someone quotes a parameter count, ask the other three questions: how much data, where the compute goes, and who runs it under what conditions. Then ask which dial got turned. Capability is moving to post-training and environments, not raw size.

Related TMFNK Content

Crepi il lupo! 🐺