Schedules

This module holds various schedules for parameters such as the step rate or momentum for gradient descent.

A schedule is implemented as an iterator. This allows it to have iterators of infinite length. It also makes it possible to manipulate scheduls with the itertools python module, e.g. for chaining iterators.

climin.schedule.decaying(start, decay)

Return an iterator of exponentially decaying values.

The first value is start. Every further value is obtained by multiplying the last one by a factor of decay.

Examples

>>> from climin.schedule import decaying
>>> s = decaying(10, .9)
>>> [next(s) for i in range(5)]
[10.0, 9.0, 8.100000000000001, 7.290000000000001, 6.561]

climin.schedule.linear_annealing(start, stop, n_steps)

Return an iterator that anneals linearly to a point linearly.

The first value is start, the last value is stop. The annealing will be linear over n_steps iterations. After that, stop is yielded.

Examples

>>> from climin.schedule import linear_annealing
>>> s = linear_annealing(1, 0, 4)
>>> [next(s) for i in range(10)]
[1.0, 0.75, 0.5, 0.25, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

climin.schedule.repeater(iter, n)

Return an iterator that repeats each element of iter exactly n times before moving on to the next element.

Examples

>>> from climin.schedule import repeater
>>> s = repeater([1, 2, 3], 2)
>>> [next(s) for i in range(6)]
[1, 1, 2, 2, 3, 3]

class climin.schedule.SutskeverBlend(max_momentum, stretch=250)

Class representing a schedule that step-wise increases from zero to a maximum value, as described in [sutskever2013importance].

Examples

>>> from climin.schedule import SutskeverBlend
>>> s = iter(SutskeverBlend(0.9, 2))
>>> [next(s) for i in range(10)]
[0.5, 0.75, 0.75, 0.8333333333333333, 0.8333333333333333, 0.875, 0.875, 0.9, 0.9, 0.9]

[sutskever2013importance]

On the importance of initialization and momentum in deep learning, Sutskever et al (ICML 2013)