In this post, we revisit model comparison metrics. Should we compare sample-by-sample, or group several samples together? Or does it make more sense to “pack” samples (i.e. join multiple samples with an EOS token in between)? Does length matter? And what’s up with these “pages” in the dataset? TL;DR A Read more…
In the previous post on incentive we described how loss values are used to score models against each other, and how the relative standing between models changes over time due to decaying advantage. From that conceptual idea a few steps are needed to get to a ranking of models and Read more…
Disclaimer: The text below is written for a wide audience. This means that the tech vocabulary is deliberately kept simple and, to some readers, the explanations may appear a bit long-winded. Week zero The biggest changes we implemented in our first week of existence are the modifications to the incentive Read more…