پست Build your own decision model

Build your own decision model Nish Tahir shows how to turn a small Qwen model into a fixed-choice decision model by masking outputs to allowed tokens, then evaluates and fine-tunes it on CommonsenseQA. Because the raw token probabilities are overconfident, he applies temperature scaling so the reported confidence better matches actual accuracy. https://nishtahir.com/build-your-own-decision-model/