Support Vector Machines (SVMs)¶
Support Vector Machines from [LIBSVM]¶
- class mlpy.LibSvm(svm_type='c_svc', kernel_type='linear', degree=3, gamma=0.001, coef0=0, C=1, nu=0.5, eps=0.001, p=0.1, cache_size=100, shrinking=True, probability=False, weight={})¶
LibSvm.
- Parameters:
- svm_typestring
SVM type, can be one of: ‘c_svc’, ‘nu_svc’, ‘one_class’, ‘epsilon_svr’, ‘nu_svr’
- kernel_typestring
kernel type, can be one of: ‘linear’ (uT*v), ‘poly’ ((gamma*uT*v + coef0)^degree), ‘rbf’ (exp(-gamma*|u-v|^2)), ‘sigmoid’ (tanh(gamma*uT*v + coef0))
- degreeint (for ‘poly’ kernel_type)
degree in kernel
- gammafloat (for ‘poly’, ‘rbf’, ‘sigmoid’ kernel_type)
gamma in kernel (e.g. 1 / number of features)
- coef0float (for ‘poly’, ‘sigmoid’ kernel_type)
coef0 in kernel
- Cfloat (for ‘c_svc’, ‘epsilon_svr’, ‘nu_svr’)
cost of constraints violation
- nufloat (for ‘nu_svc’, ‘one_class’, ‘nu_svr’)
nu parameter
- epsfloat
stopping criterion, usually 0.00001 in nu-SVC, 0.001 in others
- pfloat (for ‘epsilon_svr’)
p is the epsilon in epsilon-insensitive loss function of epsilon-SVM regression
- cache_sizefloat [MB]
size of the kernel cache, specified in megabytes
- shrinkingbool
use the shrinking heuristics
- probabilitybool
predict probability estimates
- weightdict
changes the penalty for some classes (if the weight for a class is not changed, it is set to 1). For example, to change penalty for classes 1 and 2 to 0.5 and 0.8 respectively set weight={1:0.5, 2:0.8}
Example on spiral dataset
:
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import mlpy
>>> f = np.loadtxt("spiral.data")
>>> x, y = f[:, :2], f[:, 2]
>>> svm = mlpy.LibSvm(svm_type='c_svc', kernel_type='rbf', gamma=100)
>>> svm.learn(x, y)
>>> xmin, xmax = x[:,0].min()-0.1, x[:,0].max()+0.1
>>> ymin, ymax = x[:,1].min()-0.1, x[:,1].max()+0.1
>>> xx, yy = np.meshgrid(np.arange(xmin, xmax, 0.01), np.arange(ymin, ymax, 0.01))
>>> xnew = np.c_[xx.ravel(), yy.ravel()]
>>> ynew = svm.pred(xnew).reshape(xx.shape)
>>> fig = plt.figure(1)
>>> plt.set_cmap(plt.cm.Paired)
>>> plt.pcolormesh(xx, yy, ynew)
>>> plt.scatter(x[:,0], x[:,1], c=y)
>>> plt.show()

Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a library for support vector machines. 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm
N Cristianini and J Shawe-Taylor. An introduction to support vector machines. Cambridge University Press.
V Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, 1995.
Kernel Adatron¶
- class mlpy.KernelAdatron(C=1000, maxsteps=1000, eps=0.01)¶
Kernel Adatron algorithm without-bias-term (binary classifier).
The algoritm handles a version of the 1-norm soft margin support vector machine. If C is very high the algoritm handles a version of the hard margin SVM.
Use positive definite kernels (such as Gaussian and Polynomial kernels)
- Parameters:
- Cfloat
upper bound on the value of alpha
- maxstepsinteger (> 0)
maximum number of steps
- epsfloat (>=0)
the algoritm stops when abs(1 - margin) < eps
Example:
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import mlpy
>>> np.random.seed(0)
>>> mean1, cov1, n1 = [1, 4.5], [[1,1],[1,2]], 20 # 20 samples of class 1
>>> x1 = np.random.multivariate_normal(mean1, cov1, n1)
>>> y1 = np.ones(n1, dtype=int)
>>> mean2, cov2, n2 = [2.5, 2.5], [[1,1],[1,2]], 30 # 30 samples of class 2
>>> x2 = np.random.multivariate_normal(mean2, cov2, n2)
>>> y2 = 2 * np.ones(n2, dtype=int)
>>> x = np.concatenate((x1, x2), axis=0) # concatenate the samples
>>> y = np.concatenate((y1, y2))
>>> K = mlpy.kernel_gaussian(x, x, sigma=2) # kernel matrix
>>> xmin, xmax = x[:,0].min()-1, x[:,0].max()+1
>>> ymin, ymax = x[:,1].min()-1, x[:,1].max()+1
>>> xx, yy = np.meshgrid(np.arange(xmin, xmax, 0.02), np.arange(ymin, ymax, 0.02))
>>> xt = np.c_[xx.ravel(), yy.ravel()] # test points
>>> Kt = mlpy.kernel_gaussian(xt, x, sigma=2) # test kernel matrix
>>> fig = plt.figure(1)
>>> cmap = plt.set_cmap(plt.cm.Paired)
>>> for i, c in enumerate([1, 10, 100, 1000]):
... ka = mlpy.KernelAdatron(C=c)
... ax = plt.subplot(2, 2, i+1)
... ka.learn(K, y)
... ytest = ka.pred(Kt).reshape(xx.shape)
... title = ax.set_title('C: %s; margin: %.3f; steps: %s;' % (c, ka.margin(), ka.steps()))
... plot1 = plt.pcolormesh(xx, yy, ytest)
... plot2 = plt.scatter(x[:,0], x[:,1], c=y)
>>> plt.show()

Friess, Cristianini, Campbell. The Kernel-Adatron Algorithm: a Fast and Simple Learning Procedure for Support Vector Machines.
Kecman, Vogt, Huang. On the Equality of Kernel AdaTron and Sequential Minimal Optimization in Classification and Regression Tasks and Alike Algorithms for Kernel Machines. ESANN’2003 proceedings - European Symposium on Artificial Neural Networks, ISBN 2-930307-03-X, pp. 215-222.