This is part of our series on a year of Bittensor experience, leading up to our anniversary at the 13th of July. We discuss 9 miner archetypes before digging deeper into incentive landscapes. After discussing the honest, the rogue, the calculative and the blockchain-savvy miner, we’re now considering the overfitting miner. Please let us know what you think in our Discord channel!
Sometimes the optimal solution to win is an extreme version of the optimal solution for the stated goal of the subnet, rendering the subnet completely useless.
In one LLM training subnet, the validator tested submission quality on only a few hundred unique publicly known samples. One miner submitted a model that could exactly reproduce those samples (and nothing else), scoring zero on the loss function, and securing a firm #1 position for days, with a model that had no real-world use at all.
In a different subnet, “good” embedding vectors needed to be generated for a set of texts, supposedly by an embedding model. Their mutual orthogonality was a decisive factor in the scoring metric. Needless to say, submitting orthogonal unit vectors was a winning strategy, despite them being useless for any real-world application.
In general, many subnets have such a small sample space, that miners quickly realize they can simply take the entire sample space and construct optimal submissions without doing any AI work. These subnets will work extremely well on the dataset, but barely do anything useful on real-world data. This can go as far as miners caching the full sample set locally (we may be talking about terabytes!) and indexing it for fast, subsecond lookup. Miners will score off the charts on synthetic queries, while performing moderately at best on organic queries.
A rogue overfitting miner may even decide to influence the sample set directly, for example by injecting their own samples into the sample set, giving them advance knowledge of challenges. Needless to say, this all optimizes reward, but not subnet output.
Tomorrow we will discuss the devops miner!
0 Comments