2.4 Other GBML Models

This section covers some models which are orthogonal to those discussed earlier.

Online Evolutionary Computation

In many problems, especially sequential ones, feedback is very noisy and needs averaging. Whiteson and Stone [293] allocated trials to chromosomes in proportion to their fitness with the following procedure. At each new generation evaluate each chromosome once only. Allocate subsequent evaluations using a softmax distribution based on the initial fitnesses and recalculate the average fitness of a chromosome after each evaluation. In non-stationary problems a recency-weighted average of fitness samples is used. They call this approach online Evolutionary Computation. Its advantages are that less time is wasted evaluating weaker chromosomes, and, in cases where mistakes matter, fewer mistakes are made by agents during fitness evaluations. However, the improvement is only on average; worst-case performance is not improved. This is related to other work on optimising noisy fitness functions [262,20], except that they do not reduce online mistakes.

Steady State EAs

Whereas standard generational EAs replace the entire population each generation, steady-state EAs replace a subset (e.g. only two in XCS). This approach is standard in Michigan LCS because they minimise disruption to the population, which is useful for on-line learning. Steady-state EAs introduce selection for deletion as well as reproduction and this is typically biased toward lower fitness chromosomes or to reduce crowding.

Co-evolving Learners and Problems

Another possibility not mentioned in our earlier classifications is to co-evolve both learners and problems. When successful, this allows learners to gradually solve harder problems rather than tackling the most difficult problems from the start. It also allows us to search the space of problems to find those which are harder for a given learner, and to explore the dynamics between learners and problems.