Bayesian Optimization for SVM

This function estimates parameters for SVM(Gaussian Kernel) based on bayesian optimization

svm_opt(train_data, train_label, test_data, test_label,
  gamma_range = c(10^(-3), 10^1), cost_range = c(10^(-2), 10^2),
  svm_kernel = "radial", degree_range = c(3L, 10L),
  coef0_range = c(10^(-1), 10^1), init_points = 4, n_iter = 10,
  acq = "ei", kappa = 2.576, eps = 0, optkernel = list(type =
  "exponential", power = 2))

Arguments

train_data

A data frame for training of SVM
train_label

The column of class to classify in the training data
test_data

A data frame for training of SVM
test_label

The column of class to classify in the test data
gamma_range

The range of gamma. Default is c(10 ^ (-3), 10 ^ 1)
cost_range

The range of C(Cost). Default is c(10 ^ (-2), 10 ^ 2)
svm_kernel

Kernel used in SVM. You might consider changing some of the following parameters, depending on the kernel type.

  • linear: \(u'v\)

  • polynomial: \((\gamma u'v +coef0)^{degree}\)

  • radial basis: \(exp(-\gamma|u-v|^2)\)

  • sigmoid: \(tanh(\gamma u'v + coef0)\)
degree_range

Parameter needed for kernel of type polynomial. Default is c(3L, 10L)
coef0_range

parameter needed for kernels of type polynomial and sigmoid. Default is c(10 ^ (-1), 10 ^ 1)
init_points

Number of randomly chosen points to sample the target function before Bayesian Optimization fitting the Gaussian Process.
n_iter

Total number of times the Bayesian Optimization is to repeated.
acq

Acquisition function type to be used. Can be "ucb", "ei" or "poi".

  • ucb GP Upper Confidence Bound

  • ei Expected Improvement

  • poi Probability of Improvement
kappa

tunable parameter kappa of GP Upper Confidence Bound, to balance exploitation against exploration, increasing kappa will make the optimized hyperparameters pursuing exploration.
eps

tunable parameter epsilon of Expected Improvement and Probability of Improvement, to balance exploitation against exploration, increasing epsilon will make the optimized hyperparameters are more spread out across the whole range.
optkernel

Kernel (aka correlation function) for the underlying Gaussian Process. This parameter should be a list that specifies the type of correlation function along with the smoothness parameter. Popular choices are square exponential (default) or matern 5/2

Value

The test accuracy and a list of Bayesian Optimization result is returned:

  • Best_Par a named vector of the best hyperparameter set found

  • Best_Value the value of metrics achieved by the best hyperparameter set

  • History a data.table of the bayesian optimization history

  • Pred a data.table with validation/cross-validation prediction for each round of bayesian optimization history

Examples

library(MlBayesOpt)

set.seed(71)
res0 <- svm_opt(train_data = iris_train,
                train_label = Species,
                test_data = iris_test,
                test_label = Species,
                svm_kernel = "polynomial",
                init_points = 10,
                n_iter = 1)
#> elapsed = 0.01	Round = 1	degree_opt = 5.0000	coef0_opt = 1.2641	Value = 0.9333 
#> elapsed = 0.01	Round = 2	degree_opt = 7.0000	coef0_opt = 7.6410	Value = 0.9333 
#> elapsed = 0.01	Round = 3	degree_opt = 5.0000	coef0_opt = 1.5038	Value = 0.9600 
#> elapsed = 0.01	Round = 4	degree_opt = 4.0000	coef0_opt = 7.6924	Value = 0.9333 
#> elapsed = 0.01	Round = 5	degree_opt = 5.0000	coef0_opt = 8.4372	Value = 0.9333 
#> elapsed = 0.01	Round = 6	degree_opt = 10.0000	coef0_opt = 7.7898	Value = 0.9333 
#> elapsed = 0.01	Round = 7	degree_opt = 8.0000	coef0_opt = 1.4249	Value = 0.9200 
#> elapsed = 0.01	Round = 8	degree_opt = 9.0000	coef0_opt = 8.0788	Value = 0.9333 
#> elapsed = 0.01	Round = 9	degree_opt = 5.0000	coef0_opt = 8.9781	Value = 0.9333 
#> elapsed = 0.01	Round = 10	degree_opt = 6.0000	coef0_opt = 9.2771	Value = 0.9333 
#> elapsed = 0.01	Round = 11	degree_opt = 10.0000	coef0_opt = 4.4448	Value = 0.9333 
#> 
#>  Best Parameters Found: 
#> Round = 3	degree_opt = 5.0000	coef0_opt = 1.5038	Value = 0.9600