QSSIM is a simulation program designed to study the behaviors of several proposed Qualifying System Algorithms (QSA's.) Each QSA is repeatedly applied to a partially randomized 4-year tourney data set. (Follow the links below for a more complete description.) The "partial randomization" is the introduction of 32 robot players into the population of about 3700 real players. These robots play the schedules of 32 real expert players. At the end of each simulation, each QSA determines the ranking of the 32 robots. These rankings are compared to the known robot abilities, allowing an evaluation of the goodness of each QSA.
The first graph below displays the Kendall's tau statistic, which is a measure of the correlation between the QSA-assigned ranks and the known ranks. Error bars are calculated, after n simulates, as average +- 1.65*stdev/sqrt(n), admittedly ignoring the multiple comparisons problem.
The second graph shows the n-out-of-10 statistic. This is the number of the known top 10 players (well, in this case robots) who are assigned to the top 10 by the QSA.
The methods 1-11 are described in QSSIM Round 1 - Details.
After the initial investigation, I revised the simulation to address the question of fairness of the QSA - Does it favor players who dodge low-rated players, or otherwise (by proximity to a population of high-rated players) play stronger schedules?
The SCRABBLE® trademark is owned by J.W. Spear and Sons, PLC, a subsidiary of Mattel, Inc. outside of the United States and Canada.