The official IB subject guide sets forth a series of guidelines, the most notable of which is the “concise” nature of the mathematical investigation. In other words, a good exploration should aim to cover both the breadth and the depth of the IB curriculum, and be able to reach a certain result/conclusion.
The "breadth" aspect often involves linking several areas of the syllabus together. As a simple example, an investigation dealing with population trends would encapsulate both the “statistics” and the “functions and equations” sections of the syllabus. However, student must take care not to pick a topic that is too broad. The IB Math exploration is limited to only twelve (12) pages!
The "depth" aspect is linked to the technical difficulty of the problem being studied. In fact, the official mark-scheme includes a whole criterion dedicated solely to mathematical rigor of the exploration. Should a student choose a superficial topic, he/she will struggle to pick up marks and needlessly jeopardize his/her overall score. Special care must be taken by HL students as they are held to a higher standard than their SL or studies colleagues.
One of the most overlooked (and useful) tips for picking a good topic is the real world application/relevance of the problem being studied. When it comes to purely mathematical explorations, students often struggle with the “big picture” aspect of the task and miss out on the easy “reflection” and “real world links” marks. Therefore, explorations that focus on modeling a certain phenomenon/problem tend to score well in that regard.
Last but not least, students need to make sure that the topic of choice has a logical result/conclusion that can be achieved within the 12 page limit. One of the easiest ways to lose organization and structure marks is to have an exploration that is either bloated with content, incomplete (lacks reflection, conclusion) or both. The easiest way to prevent such a problem is to narrow down the scope of the exploration. “What is the best model for population growth in country X?” is a much better topic than “Modeling population growth”, as a crude example.
The exploration is an opportunity for a student to use his/her imagination and explore any area of mathematics that may be of interest, although it is recommended not to stray too far into the unknown. Depending on the topic choice, the task will be either filled with fascination or frustration. Thus in order to avoid the latter, follow the above recommendations and get the best grade you can.