Benchmarking Procedures for Continuous Optimization Algorithms

Karol Opara , Jarosław Arabas


Reliable comparison of optimization algorithms requires the use of specialized benchmarking procedures. This paper highlights motivations which influence their structure, discusses evaluation criteria of algorithms, typical ways of presenting and interpreting results as well as related statistical procedures. Discussions are based on examples from CEC and BBOB benchmarks. Moreover, attention is drawn to these features of comparison procedures, which make them susceptible to manipulation. In particular, novel application of the weak axiom of revealed preferences to the field of benchmarking shows why it may be misleading to assess algorithms on basis of their ranks for each of test problems. Additionally, an idea is presented of developing massively parallel implementation of benchmarks. Not only would this provide faster computation but also open the door to improving reliability of benchmarking procedures and promoting research into parallel implementations of optimization algorithms.
Author Karol Opara
Karol Opara,,
, Jarosław Arabas (FEIT / PE)
Jarosław Arabas,,
- The Institute of Electronic Systems
Journal seriesJournal of Telecommunications and Information Technology, (B 7 pkt)
Issue year2011
Keywords in Englishblack-box optimization, comparing optimization algorithms, evaluation criteria, parallel computing
ASJC Classification2208 Electrical and Electronic Engineering; 1705 Computer Networks and Communications
ProjectResearch on measurment, circuit and signal theory and electronic circuits and systems. Project leader: Romaniuk Ryszard, , Phone: +48 22 234 7986, +48 22 234 5360, start date 05-04-2012, planned end date 31-12-2012, end date 30-11-2013, ISE/2012/DS, Completed
WEiTI Działalność statutowa
Languageen angielski
Score (nominal)7
Score sourcejournalList
Publication indicators GS Citations = 10.0; Scopus SNIP (Source Normalised Impact per Paper): 2013 = 0.507
Citation count*10 (2020-01-10)
Share Share

Get link to the record

* presented citation count is obtained through Internet information analysis and it is close to the number calculated by the Publish or Perish system.
Are you sure?