We developed 4 module recommendation algorithms and tested them on Drupal.org. And we used Google Analytics and tracked the click-through rates. The overall click-through rate was 0.263%, co-occurrences 0.097%, relevance 0.141%, recency 0.114% and uniqueness 0.138%. The relevancy algorithm appeared to have the highest click-through rate, but it was only significantly higher than the co-occurrences algorithm. Despite that, we can still say that the relevancy algorithm is the best alternative among the 4.

Now we have the baseline for future module recommendation algorithms, such as recommendations based on download statistics, or on module usage statistics. The new algorithm has to have a better click-through rate than the relevancy algorithm.

The 0.141% click-through rate, however, does not seem to be a large number. A general rule-of-thumb is that the best online ads would have 1% click-through rate. But it might be calculated using different method in different context. Therefore, I'm thinking to test the click-through rate if we put the "New forum posts" block on the module pages. Hope it can be done soon.