Teaching Llama 2 to answer questions about one university.
Students at Abdul Wali Khan University Mardan ask the same admissions and programme questions every term. I wanted to know how small a dataset could be and still shift a 7B model's behaviour, so I wrote 41 question–answer pairs by hand and fine-tuned on those alone.
It works for questions close to the training set and invents details for anything outside it. 41 pairs buys you tone and format, not knowledge. The honest fix is retrieval over the university's actual documents, which is what I would build if I started again.
- Base model
- Llama-2-7b-chat, loaded in 4-bit NF4
- Method
- QLoRA at rank 64, alpha 16, dropout 0.1
- Training run
- 1 epoch, batch size 4, lr 2e-4, on a single Colab T4
- Evaluation
- Manual inspection only. There was no held-out set, which is the main gap
Training configuration
- Dataset
- 41 hand-written pairs
- Adapter
- LoRA rank 64, alpha 16
- Precision
- 4-bit NF4 · fp16 compute
- Libraries
- transformers · peft · trl
- Hardware
- Single T4, free tier
