Stop optimisation once fitness converged

[lgravina1997]

Hello. I have a function which returns values between -2 and -14. I do not know to optimal value a priori.

Using bboptimize converges towards what seems to be the correct minimum (say -4) but never stops. Is there a way to put a stopping condition on the fitness convergence?

Say for example if after 20 steps its constant then stop.

[robertfeldt]

There is already such a default value but it is much higher, 1000 consecutive without improvement, I think. Or you can limit the time the optimisation takes. Hope this helps.

[chris-hampel-CA]

I also have the same question. I am finding that I can only specify a FitnessTarget and FitnessTolerance where the solver stops once the func eval is within the range of Fitness target +/- FitnessTolerance. I find this to be problematic if the target is not well known a priori, as mentioned previously. Shouldn't there be a way to specify a tolerance to quit solving if the relative change in the func eval or change in decision variable is below a threshold? This would be especially helpful for cases when the func eval is "expensive". Running 1000 consecutive cases without improvement as the stopping criteria wastes a lot of time for expensive func evals.

[robertfeldt]

Yes, I agree but you can also do this with the existing callback interface. Have you tried this?

For an example see here:

https://github.com/robertfeldt/BlackBoxOptim.jl/blob/master/examples/early_stopping_in_callback.jl

and by setting a higher CallbackInterval you can check this say every second or so in case you have a quick to execute fitness function and don't want to check all the time (for long-running fitness functions it shouldn't really matter since the relative time needed will be vastly different). Hope this helps!

[chris-hampel-CA]

Thanks for response!

However, I might not be fully understanding because the example looks like it accomplishes nearly the same thing that using the TargetFitness and FitnessTolerance inputs accomplish. My understanding is that callbacks still only allow you to check the "current" obj func eval against some predefined value, but my goal is to check the current eval against the previous eval. Is that possible with callbacks?

[robertfeldt]

Yes, you have full control so just save the previous value and compare them. If there is no change for some time, then shutdown. You can implement any stopping criteria in this way.

[robertfeldt]

If you are still confused just ask and I can put together a concrete example. No problem.

[chris-hampel-CA]

I tried something like this that seems to work. My saving method might not be the best but it works.

const rel_tol = 1e-3
global prev_fitness = 1e10 #initialize with dummy value

function set_rel_tol(oc)
    global prev_fitness
    @show prev_fitness

    rel_diff = abs(best_fitness(oc) - prev_fitness)/abs(prev_fitness) #needs abs() in denominator if objfunc is negative (i.e., maximizing)
    bool =  rel_diff < rel_tol
    prev_fitness = best_fitness(oc) #re-assign for next iteration

    @show rel_diff
    @show prev_fitness

    return bool
end

function cbearlystopping(oc)
    if set_rel_tol(oc)
        BlackBoxOptim.shutdown!(oc)
        @info "rel_tol achieved"
    end
end

Then I passed in cbearlystopping into the CallBackFunction kwarg and set CallbackInterval to 20.0. I found that it could be dangerous to set CallbackInterval = 0.0 because if the solver takes sequential steps early on that have identical decision variable values, then the optimization would terminate while it might not yet be close to the optimum.

[robertfeldt]

Yes, this is what I meant. I agree you shouldn't use CallbackInterval = 0.0 since it might not give a robust sampling of actual progress, at the start for sure but possibly also later. Also, with a 10 or 20 seconds interval you don't risk loosing much unless your fitness function is really costly.

[chris-hampel-CA]

Do you think there might be any issues with using the CallbackInterval technique when running on multiple cores using Distributed ?

[robertfeldt]

Hmm, not sure, the part that checks and calls the callback is done in a single / the main thread so depending on how you mean to use it it might work or not. Please report back so we can debug if not. We need to ramp up the support / simplicity of using multithreads anyway.