Skip to content

A geometrical approach to ML🔗

Support Vector Machines, the bias-variance tradeoff and a bit of kernel theory🔗

This class takes a geometrical approach to Machine Learning through the prism of Support Vector Machines. It covers linear classifiers for data separation first. On the way, it introduces the bias/variance tradeoff. Then it presents a bit of kernel theory and applies it in linear classifiers to reach non-linear SVMs. It then provides perspectives on multi-class classification, support vector regression and density estimation with one-class SVM. Two full practical examples are provided at the end.

Notebook (colab)

Pre-class refresher activities and solution
Summary card
Lecture notes

References

  • On the general theory of SVMs for classification:
    A tutorial on Support Vector Machines for Pattern Recognition.
    C. J. C. Burges, Data Mining and Knowledge Discovery, 2, 131-167, (1998).

  • On support vector regression (and its extension to \(\nu\)-SVR):
    A tutorial on Support Vector Regression.
    A. J. Smola and B. Schölkopf, Journal of Statistics and Computing, 14(3), 199-222, (2004).
    New support vector algorithms. B. Schölkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett. Neural computation, 12(5), 1207-1245, (2000).

  • On One-Class SVMs:
    Support vector method for novelty detection.
    B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, and John C. Platt. Neural Information Processing Systems, 12, 582-588, (1999).

  • On multi-class SVMs:
    On the algorithmic implementation of multiclass kernel-based vector machines.
    K. Crammer and Y. Singer. Journal of machine learning research, 2, 265-292, (2001).

An application of SVMs to multi-label classification🔗

This short application session proposes to use SVMs as a building block for Multi-Label Classification. It serves both as a practice session for the previous class and as an introduction to Multi-Label Classification.

Notebook source
Notebook on Colab

References

  • J. Read, P. Reutemann, B. Pfahringer, and Geoff Holmes. MEKA: A multi-label/multi-target extension to Weka. Journal of Machine Learning Research, 17(21):1–5, 2016.
  • J. Read, B. Pfahringer, G. Holmes, and E. Frank. Classifier chains for multi-label classification. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 254–269, 2009.
  • G. Tsoumakas and I. Katakis. Multi-label classification: An overview. International Journal on Data Warehousing and Mining, 3(3):1–13, 2007.
  • G. Tsoumakas, I. Katakis, and I. Vlahavas. Mining multi-label data. Data mining and knowledge discovery handbook, pages 667–685. Springer, 2010.
  • G. Tsoumakas, I. Katakis, and I. Vlahavas. Random k-labelsets for multi-label classification. IEEE Transactions on Knowledge and Data Engineering, 23(7):1079-1089, 2011.