Q: Is there a public miner available?
Coldint is working hard to make its miner ready for release on GitHub. From that point on, it will be maintained by coldint with the help of the community. Pull requests are welcome and significant code contributions may be eligible for miner rewards – if you have more brains than high-end hardware, you can still earn!
Q: How can I assess whether my model is good enough for publishing?
It can be challenging to assess whether your model will win in the competition. In the end it is up to the validators to compare models against eachother. However, the validators all run the same code, which is available for download on GitHub to anyone – including miners. The validator code reads a file benchmark.json which can be used to inject a model into the evaluation loop. The validator is also quite verbose about how it scores models against each other, so as a miner you can quickly assess the standing of your model before uploading it, just by running the validator code. Note that your validator will not take part in on-chain weight setting, but that does not matter.
Q: What metrics does coldint apply to competition effectiveness? When and how is a competition stopped?
Each competition should see at least one improved model per 4 days. If the pace drops below that, the competition is assumed to have reached an optimal model for the set of boundary conditions, and the competition can be stopped. When a competition is stopped, this is announced in a blog posting detailing statistics and learnings of the competition.
Q: What metagaming tactics are prevented, and how is it done?
Model copying is an effective tactic to increase the “win rate” of a model in various subnets. By uploading an identical model, at a later time, the first uploaded copy will win 100% of its copies. A model winning only 40% of the top model, will win 240% if two copies are uploaded. A singular top model will only win 3 * 60% = 180% in such a scenario, making the top model lose from a model that is objectively worse. This is prevented by awarding one win per sample.
Publishing without publishing is an effective tactic to claim a place in the blockchain without actually making the model public, e.g. by setting the HuggingFace repository to private or not pushing the commit to HuggingFace. The validator code will mark a pushed model as invalid if HuggingFace keeps returning errors on download for two hours after publishing the model on chain. This is persistent state so it will survive restarts of the validator. Only clean installs of validators will now know a model was not initially public, but thanks to Yuma consensus, one validator is not able to move the needle for one model.
Training for validation is perhaps not a metagaming tactic per se, but it comes close. Models are trained to perform a task, e.g. text generation. A good model will show low loss values for samples of a particular dataset. Any deviation in the validator logic from assessing the loss on an actual sample, incentivizes miners to train not for the actual task, but rather to train for validation. For example, if validation is done on 4096 token long “packed samples” separated by EOS tokens, this will make models trained on such packed samples perform better than models that are trained on actual samples. Apart from the misalignment between miner and subnet objective, this also (wrongly) benefits miners that can afford high-end hardware, as training for 4096 token long packed samples requires more resources than training for actual samples that are typically 600 tokens long.
Q: Where and how can I get in touch with the team?
The team closely monitors the Discord channel that is dedicated to SN29. Questions, issues, bug reports and suggestions should be posted there, or by DM to the team.
Q: Can’t I just send an email?
No. Unless we exchange email addresses via Discord first.
Q: I found a bug, can I get a bug bounty? How does it work, what will happen?
Certainly. Any bug report or code contribution may be eligible for a reward in the form of emitted TAO. After coldint decides that your contribution should be rewarded with an amount of TAO, your hotkey is stored in https://coldint.io/hall_of_fame.json (which redirects to a page on GitHub), along with a bug description, an initial weight and a starting block. Validators will retrieve this file and start giving your hotkey weight from the starting block. Every epoch (of 361 blocks) the weight is reduced with 0.5%. The resulting emission eventually adds up to 200 times the initial weight, half of which is emitted in the first week. The amount of TAO this reflects depends on the total subnet emission, which is a factor determined by the weight setting of root validators.
Q: Why are bug bounties not awarded in full, right away?
As bug bounties are a form of miner reward, they are part of miner emissions. The reward is spread out in time so that the regular mining operation is not disturbed too much.
Q: What are some examples of factors that are used to set the initial bug bounty emission?
Completeness of bug report and analysis, inclusion of a fix, severity of the bug, impact on emissions, whether the bug is already being exploited, whether the bug is already being researched and fixed, amount of work to analyse and fix, …
Q: Will coldint participate in mining SN29?
Yes and no. In order to ensure that the canonical miner is up to standards, and produces viable models for each and every newly publicized training objective, coldint will produce and submit models that serve as a starting point for each new competition, along with the miner code that produced it. This way, each competition is kickstarted with a trained model, and miners are supplied with working code, allowing them to participate in mining on a level playing field. As more miners appear in a competition, coldint will silently drop out. Coldint explicitly wants to provide an open environment where new miners can jump in with low end hardware and make a few TAO, without having to first analyze 1000+ lines of code and 100+ Discord posts. Coldint incentivizes itself to incentivize miners to join in and improve.
Q: What are the future plans for coldint, what is the business model?
Please refer to the roadmap and to the blog to see what we are up to.