☰
Frequently Asked Questions
18th July 2024 (v1.7)
The AI-IMO Prize launched on 27 November 2023. The following FAQs have been collated from the initial questions submitted to aimo@xtxmarkets.com by 15 December 2023. Questions have been summarised and duplicates merged. Some we think we can answer now, others will take time. Most answers given here are likely to be provisional and subject to refinement as things develop.
1. Objectives
1.1. What are the long-term goals of the AIMO Prize for AI and mathematical problem-solving?
We believe that the creation of a publicly-shared AI model that is capable of winning an IMO gold medal will be an important step towards making AI useful for mathematics, including new research. The AIMO Prize aims to drive experimentation and innovation, especially among students and researchers who are working outside of AI research labs.
1.2. What does the AIMO Prize hope to achieve through the public sharing of models?
Many new developments in AI are proprietary. We believe that publicly sharing models will foster collaboration in the field, enabling participants to build on each others’ work and supporting our common pursuit of new knowledge. The public sharing protocol will be designed to balance incentives to participate with collaboration and safety.
2. Structure and Timeline
2.1. How are the grand prize and progress prizes structured?
The grand prize of $5mn will be awarded to the first publicly-shared AI model to enter an AIMO approved competition and perform at a standard equivalent to a gold medal in the IMO. The AIMO Advisory Committee will work on the specific formulation of the grand prize, e.g. what is the specification required for the AIMO approved competition, how will we measure that performance is at a standard equivalent to a gold medal in the IMO.
There will also be a series of progress prizes, totalling up to $5mn, for publicly-shared AI models that achieve milestones towards the grand prize. These prizes may take different forms, for example: pre-IMO competitions and Olympiads, foundational steps in mathematical reasoning, and even facilitating tasks, such as creating training data.
2.2. Are prizes time-bound or will they be open-ended, i.e. first participant(s) to achieve criteria?
It is envisaged that the grand prize will be open-ended, i.e. the first participants(s) to achieve the criteria are the winners. However, it is likely that the AIMO approved competitions will be held periodically.
Progress prizes will be set by the AIMO Advisory Committee on an on-going basis. As noted, these may take different forms, including a mix of time-bound and open-ended prizes.
2.3. What are the future plans and wider timeline for the AIMO Prize?
The AIMO Advisory Committee will work on the formulation of the grand prize and the design of the public sharing protocol. It is intended that these will be announced in the first quarter of 2024.
The first progress prizes will open in early 2024. It is intended that these will be contested throughout the first half of the year, culminating in a presentation of progress at the 65th IMO, which will be held in England in July 2024.
3. Entry Criteria for Participants
3.1. What are the eligibility criteria to participate in the AIMO Prize competitions?
The AIMO Prize is open to individuals and teams. The maximum team size is 5. Participants can enter under an organisational affiliation (e.g. university, school, company) or independently. Participation is on a prize-by-prize basis, allowing for teams to evolve over time.
Participants will need to meet the AIMO Prize eligibility criteria, including formally committing to the public-sharing protocol. There will be restrictions on participants (e.g. AML, sanctions) and potentially some restrictions on the AI models too.
Please see the rules on Kaggle for more information about eligibility.
4. Support for Participants
4.1. Will the AIMO Prize be able to help connect and ‘matchmake’ people to create teams?
Yes. We will setup a thread on reddit.com/r/AIMOprize for this purpose.
4.2. Will the AIMO Prize be able to provide access to compute, or cover costs?
No. This is not something that we are currently planning to provide.
4.3. Will the AIMO Prize cover expenses, e.g. for travel to events?
Yes. Where travel expenses are required and pre-approved, the AIMO Prize will cover these.
5. Data Inputs
5.1. What kinds of input or learning data can be used for AIMO Prize competitions?
There is a good amount of publicly available data online. We will aim to collate this as a library of resources in useful formats (e.g. Latex).
6. Problems and Assessment
6.1. How will the problems be formulated?
The grand prize will be informal-to-informal. It will either use actual problems from the IMO or problems that have been formulated in an equivalent way. As such, the AI models must consume problems in the same format as human contestants. The grand prize will be contested under conditions that emulate the IMO as closely as possible.
As noted, the progress prizes may include: pre-IMO competitions and Olympiads, foundational steps in mathematical reasoning, and facilitating tasks. As such, the prizes may be formulated and contested in different ways, including via Kaggle or other platforms.
6.2. How will the problems for AIMO Prize competitions be generated?
Ideally, the grand prize would use actual IMO problems in real-time. If this is not possible, a series of equivalent problems will be generated by experienced Olympiad problem-setters.
For progress prizes, problems may be generated in different ways. It is expected that most progress prizes will be based on unseen problems, which could be human-generated or computer-generated.
6.3. How do solutions have to be presented?
For the grand prize, problems and solutions will be presented in natural language. AI models must consume the problems in the same format as human contestants and must produce human readable solutions that can be graded by an expert panel, using standard Olympiad criteria. AI models will have one attempt to solve the problems, within stipulated constraints (e.g. time limit, compute limit, length of solutions).
For progress prizes, different types of solution may be required, depending on the problem. For example, a prompting competition will have different outputs to a pre-IMO competition.
In all cases, there may be certain parameters placed on solutions. For example, solutions may be capped in length to encourage mathematical reasoning rather than extended computation.
6.4. How will solutions be judged?
For the grand prize, it is intended that solutions will be judged in the same way as at the IMO, by a human panel of experts. Marking will be in the same way too, including partial credit as appropriate. Gold, silver and bronze medals, and honourable mentions, may be awarded.
If the grand prize uses IMO problems in real-time, the same medal thresholds will be used as the human competition. If alternative problems are used, the medal threshold will be based on the historic average.
For progress prizes, different types of solution may be judged in different ways. For example, some competitions may be done via automated marking as appropriate.
6.5. What steps will be taken to deter and detect human cheating?
We are creating a general framework to deter and detect human cheating. We will also add further security steps on a case-by-case basis, depending on the design and of each prize.
7. Model Parameters
7.1. What will be the restrictions on AI models?
There will be no restrictions on AI models during training. There may be some restrictions on AI models during competitions (e.g. time limit, compute limit, length of solutions).
7.2. What tools will be permitted for the AI models to use?
For the first progress prizes, competition entries may only use AI models and tools that are open source and were available before 23 February 2024. For example, programming languages, such as Python and Lean, and LLMs with publicly available weights, such as Llama or Gemma.
This will ensure that the openness requirement (see 8.2) cannot be partially bypassed by training a model on a private dataset and releasing weights only. We reserve the right to extend the set of permitted tools for future prizes, including the grand prize.
7.3. Will human assistance be allowed?
For the grand prize, human assistance will not be allowed. As noted, AI models must consume the problems in the same format as human contestants and must produce human readable solutions. AI models will have one attempt to solve the problems, within stipulated constraints (e.g. time limit, compute limit, length of solutions).
For progress prizes, human assistance may be allowed in some cases. For example, a prompting competition would be based on human input.
8. Prizes and Publication
8.1. How will the prize pool be distributed?
The prize pool is $5mn for the grand prize and a total of up to $5mn for progress prizes. Prizes will be awarded only after verification that the winning solutions have met the criteria. We are reviewing options for distributing prizes, including financial and legal requirements of both the winning participants and the AIMO Prize.
8.2. What is the public sharing protocol?
The AIMO Advisory Committee will work on the design of the public sharing protocol. This will aim to balance incentives to participate with collaboration and safety. We are looking for people with experience in these areas to join the AIMO Advisory Committee.
Our initial view is that, in the interest of advancing scientific knowledge, the AI models should be reproducible by any third party with sufficient resources. In particular, the training data, training script and final model (architecture and corresponding weights) should be made public. If the training data is too large to provide, then alternatively the procedure used to construct the training data should be provided. We will determine the licences under which AI models would need to be shared in order to be eligible for prizes. This may include a requirement to publish open-source for non-commercial uses and on a royalty-basis for commercial uses.
9. Governance and Partnership
9.1. How does the AIMO Prize relate to the International Mathematical Olympiad?
The AIMO Prize runs independently. It is not formally part of the IMO or the IMO Grand Challenge. However, it aims to collaborate with IMO members and a range of Olympiad competitors in the awarding of AIMO prizes.
There will be presentations of progress at the 65th IMO (England, July 2024) and the 66th IMO (Australia, July 2025). It is hoped that this will continue in future years and that, in time, this will develop into a partnership where the grand prize would use actual IMO problems in real-time.
9.2. What is the role of the AIMO Advisory Committee and who are its members?
The AIMO Advisory Committee is a non-fiduciary committee that will advise the AIMO Prize on various matters including competition design, prize awards and the public sharing protocol.
We are currently seeking Advisory Committee members, especially mathematicians, deep learning experts and experienced members of the IMO community. The first members of the Advisory Committee will be announced in early 2024.