As explained in our previous article, growing a model in most dimensions in quite simple, but increasing the hidden size comes with a few problems. This article dives deep and shows how it can be done. Why, again? We can simply grow a model’s MLP (intermediate size) or number of Read more…
In this post, we revisit model comparison metrics. Should we compare sample-by-sample, or group several samples together? Or does it make more sense to “pack” samples (i.e. join multiple samples with an EOS token in between)? Does length matter? And what’s up with these “pages” in the dataset? TL;DR A Read more…
In this blog, and in our Discord channel, we discuss training in detail. A topic that is often overlooked, is how to grow a model. Especially in incentivized, collaborative and distributed training, this is a key ingredient. This post explores the concept of model growth with concrete Python code examples Read more…
This post discusses one of the essential ingredients for training a model: how to see whether your model is actually improving, in other words, how do you benchmark the result of your training? Please read our prior blog posts (e.g. On incentive (part 1)) to understand the concept of sample Read more…
In the previous post on incentive we described how loss values are used to score models against each other, and how the relative standing between models changes over time due to decaying advantage. From that conceptual idea a few steps are needed to get to a ranking of models and Read more…
One of the first objectives for subnet 29 was to see whether the sample packing during training makes a difference for the achievable loss. TL;DR Sample packing makes significant difference. Per-token loss of a trained model is generally higher when training on packed samples, and evaluating on single samples. Or, Read more…
Disclaimer: The text below is written for a wide audience. This means that the tech vocabulary is deliberately kept simple and, to some readers, the explanations may appear a bit long-winded. Week zero The biggest changes we implemented in our first week of existence are the modifications to the incentive Read more…
So we registered a subnet. Time to turn all our ideas into actual plans, and start building right away. Have a working validator up and running within a week. Build a website, register at GitHub, W&B, HuggingFace, open a Discord channel. With so many ideas and so little time, we Read more…
Saturday morning, 8:12 am. Awake. Kids screaming. Weekend. Coffee. What’s happening in the world… Laptop. Checking some messages. Chain fully online again? nice. Wait, what?! But that means… [digging through subtensor code] https://github.com/opentensor/subtensor/blob/main/pallets/subtensor/src/root.rs#L1229 less than 200 Tao to register the next network! 🤯 This is my chance. All those ideas, Read more…