In this case, the optimization problem is said to have a low intrinsic dimensionality. It can outperform Grid search, especially when only a small number of hyperparameters affects the final performance of the machine learning algorithm. This can be simply applied to the discrete setting described above, but also generalizes to continuous and mixed spaces. Random Search replaces the exhaustive enumeration of all combinations by selecting them randomly. The green bars show that more individual values for each hyperparameter are considered compared to a grid search. In this example, 100 different random choices are evaluated. Random search across different combinations of values for two hyperparameters. Grid search suffers from the curse of dimensionality, but is often embarrassingly parallel because the hyperparameter settings it evaluates are typically independent of each other. Finally, the grid search algorithm outputs the settings that achieved the highest score in the validation procedure. Grid search then trains an SVM with each pair ( C, γ) in the Cartesian product of these two sets and evaluates their performance on a held-out validation set (or by internal cross-validation on the training set, in which case multiple SVMs are trained per pair). Both parameters are continuous, so to perform grid search, one selects a finite set of "reasonable" values for each, sayĬ ∈ Since the parameter space of a machine learner may include real-valued or unbounded value spaces for certain parameters, manually set bounds and discretization may be necessary before applying grid search.įor example, a typical soft-margin SVM classifier equipped with an RBF kernel has at least two hyperparameters that need to be tuned for good performance on unseen data: a regularization constant C and a kernel hyperparameter γ. Or evaluation on a hold-out validation set. A grid search algorithm must be guided by some performance metric, typically measured by cross-validation on the training set The traditional way of performing hyperparameter optimization has been grid search, or a parameter sweep, which is simply an exhaustive searching through a manually specified subset of the hyperparameter space of a learning algorithm. Blue contours indicate regions with strong results, whereas red ones show regions with poor results. For each hyperparameter, 10 different values are considered, so a total of 100 different combinations are evaluated and compared. Grid search across different values of two hyperparameters.
0 kommentar(er)
