Skip to main content

Textbooks Are All You Need — Improving Small Code LLMs with High-Quality Textbook Data

· 7 min read
p4r4d0xb0x
p4r4d0xb0x
Rustacean, AI, OSS Enthusiast

Microsoft Research's Textbooks Are All You Need focuses on phi-1, a small 1.3B-parameter LLM specialized for code, and experimentally shows that curated textbook-quality data and a small synthetic exercise dataset can achieve strong code-generation performance without scaling models and data to enormous sizes. The central reported results, based on the paper's abstract and main-text summary, are that phi-1 achieves 50.6% pass@1 on HumanEval and 55.5% on MBPP while using approximately 6B tokens of filtered web data, fewer than 1B tokens of synthetic textbook data, and approximately 180M tokens for finetuning.

This post summarizes and analyzes the paper based on its abstract and major sections from the selected chunks and quotations. The authors note that the source does not disclose some details of synthetic-data generation, constraining reproducibility. The analysis below is therefore explicitly an interpretation based on the supplied evidence pack and detected structure.

Main Points​

  • Goal: improve code-generation performance through high-quality data selection and a small synthetic exercise dataset without greatly increasing model or compute scale.
  • Data composition: a pipeline consisting of roughly 6B tokens from a filtered code-language corpus, including filtered portions of The Stack and Stack Overflow; fewer than 1B tokens of textbook-style text synthesized with GPT-3.5; and approximately 180M tokens of synthetic CodeExercises.
  • Results: phi-1—1.3B parameters and exposure to approximately 50B tokens—reports 50.6% pass@1 on HumanEval and 55.5% on MBPP. The pretrained-only phi-1-base also achieved 29%, while the 350M-parameter phi-1-small reported approximately 45%, emphasizing the influence of data quality.
  • Methodological characteristics: the model architecture is relatively standard, using a decoder-only Transformer, FlashAttention, and related techniques. The authors attribute the principal performance gains to the combination of data selection and synthetic textbooks and exercises.
Term explainer: HumanEval

Plain definition: HumanEval is a benchmark of Python programming problems based on function descriptions, or docstrings, used to evaluate an LLM's code-generation ability. Everyday example: it resembles a coding test in which you receive a function description and must complete the function.

Term explainer: pass@1

Plain definition: a metric measuring the probability that a model generates a correct, passing answer on its first attempt. Everyday example: it can be viewed as the pass rate for solving a coding test with a single submission.

Term explainer: scaling laws

Plain definition: empirically observed relationships describing how model performance improves as resources such as parameter count, data volume, and computation increase. Everyday example: it resembles the tendency for scores to improve as study time increases, though not at a constant rate.

Term explainer: finetuning

Plain definition: additional training of a pretrained model on a small dataset tailored to a specific purpose, such as a type of problem, to improve performance. Everyday example: it resembles preparing someone with general English skills for an interview through targeted mock interviews.

Term explainer: emergent properties

Plain definition: new abilities or characteristics that suddenly appear after a change in model scale or training procedure but were not visible before. Everyday example: it resembles a friend suddenly seeming to understand a particular field far better than before.

Data Design and Experimental Evidence​

The paper's central claim is that "improving data to textbook quality can produce high performance with much smaller models and fewer tokens." The supporting elements are as follows.

  • Filtering: GPT-4 annotated 100k samples to select samples with high educational value from large code corpora such as The Stack. A random-forest classifier based on code embeddings was then trained to filter the larger sample collection, according to Chunk 3.
  • Synthetic textbooks: GPT-3.5 generated fewer than 1B tokens of Python textbook material with examples and explanations. Topic and audience constraints were used to encourage reasoning and algorithmic thinking, according to Chunk 5.
  • Synthetic exercise data: finetuning on approximately 180M tokens of exercises in a docstring-completion format produced a substantial observed improvement on HumanEval, according to Chunks 2, 5, and 6.

The source presents experimental evidence that filtering itself is necessary—for example, filtered data achieved higher HumanEval performance at the same number of training steps—and reports competitive performance with a small model and fewer tokens, quoting figures in Chunk 4. However, it explicitly states that some details of synthetic-data generation were not disclosed: "we omit some details of the synthetic data generation, for proprietary reasons." This limits reproducibility.

Implementation Details: Architecture and Training​

  • Architecture: a decoder-only Transformer with standard techniques including FlashAttention, parallel MHA and MLP blocks, and rotary position embeddings, according to Chunk 6.
  • Training: sequence length 2048, fp16, AdamW, and linear-warmup-linear-decay. Phi-1-base achieved 29% on HumanEval after approximately 36k steps, corresponding to exposure to around 50B tokens, and subsequent finetuning on CodeExercises raised the final result above 50%, according to Chunk 6.

Limitations and Uncertainty​

  • Sensitive details of synthetic data, including prompts and some filtering criteria, were not disclosed, making complete reproduction difficult. The paper itself acknowledges this.
  • The paper focuses on the narrow task of code generation, primarily short Python functions. Generalization to natural-language understanding or more complex software-engineering tasks requires further validation.
  • A separate section discusses possible contamination of downstream benchmarks such as HumanEval, but the supplied evidence pack contains only some results from that analysis. A complete judgment requires checking the full source, which states, "we study possible contamination ..."

Analysis of the Paper Structure​

  • IMRaD classification: according to the detected headings and supplied chunks, the paper has the main elements of IMRaD but does not completely follow it:

    • Introduction: present in Section 1, presenting the problem and motivation around data quality and scaling laws.
    • Methods or Materials and Methods: present in Section 2, "Training details and the importance of high-quality data," which describes data selection, synthetic-data generation, model architecture, and training settings.
    • Results: present in Section 3 on emergent properties and in Figure 2.1 and performance tables, reporting HumanEval and MBPP performance and the effects of filtering and finetuning.
    • Discussion: no explicit Discussion heading was detected. Some discussion of alternative benchmarks, contamination checks, and related issues is distributed across Sections 4 and 5. The paper therefore partially follows IMRaD, but a complete standalone Discussion block is either absent or mixed with results.

  • Logical flow from big picture to detail: the paper forms a coherent problem-gap-contribution structure:

    1. Big picture in the Introduction: it presents the convention of improving performance through Transformers and scaling laws, then offers data quality as an alternative axis to establish the research motivation, based on the quotation in Chunk 1.
    2. Methods: it diagnoses why existing code datasets are educationally weak, including non-self-contained snippets and boilerplate, then presents the filtering process and the design of synthetic textbooks and exercises, including dataset composition and methods for encouraging diversity, based on Chunks 3–5.
    3. Results: it uses figures and tables to show how curated data and small-scale finetuning substantially improved HumanEval and MBPP performance, based on Chunk 2 and Figure 2.1.
    4. Subsequent discussion and validation in Sections 4–5: it addresses alternative benchmarks, contamination, and the disclosure boundary around synthetic data, presenting limitations and future directions through the cited evidence.

    This structure flows naturally from problem statement, through a data-centric solution and experimental evidence, to limitations and further validation. Each section progressively tests the hypothesis introduced earlier: that data quality changes performance. The detected structure also shows that Discussion is not clearly separated into its own section and is partly mixed with results.

Practical Implications​

  • Prioritize data engineering: when building a code LLM, data-selection and synthesis strategies can substantially reduce model and compute budgets while still producing a large effect.
  • Small models can achieve practical performance: with appropriately designed data and finetuning, a 1.3B-class model can achieve competitive performance on widely used benchmarks.
  • Mind reproducibility: undisclosed elements of synthetic-data generation and possible contamination are risks that must be checked and addressed in practical adoption.

Sources

Sources​

// COMMENTS

Comments