A new paper in Nature Methods has some interesting and world-first comparison of
97 metrics reported in the field of biomedicine alone, each with its own individual strengths, weaknesses and limitations and hence varying degrees of suitability for meaningfully measuring algorithm performance on a given research problem
By forming an international multidisciplinary consortium of 62 experts they performed a multistage Delphi process identifying pitfalls related to the inadequate choice of the problem category (P1), to poor metric selection (P2) and poor metric application (P3. Here is one P1 example of this highly recommended paper.
The pixel metrics are github while the code from the paper is also online. And do not miss the sister publication by Maier-Hein L. et al. “Metrics reloaded: recommendations for image analysis validation” also in Nat. Methods 2014.