training

This post discusses one of the essential ingredients for training a model: how to see whether your model is actually improving, in other words, how do you benchmark the result of your training? Please read our prior blog posts (e.g. On incentive (part 1)) to understand the concept of sample Read more…

One of the first objectives for subnet 29 was to see whether the sample packing during training makes a difference for the achievable loss. TL;DR Sample packing makes significant difference. Per-token loss of a trained model is generally higher when training on packed samples, and evaluating on single samples. Or, Read more…