Название: Machine Learning Evaluation: Towards Reliable and Responsible AI, 2nd Revised Edition
Автор: Nathalie Japkowicz, Zois Boukouvalas
Издательство: Cambridge University Press
Год: 2025
Страниц: 427
Язык: английский
Формат: pdf (true), epub
Размер: 10.1 MB

As Machine Learning applications gain widespread adoption and integration in a variety of applications, including safety and mission-critical systems, the need for robust evaluation methods grows more urgent. This book compiles scattered information on the topic from research papers and blogs to provide a centralized resource that is accessible to students, practitioners, and researchers across the sciences. The book examines meaningful metrics for diverse types of learning paradigms and applications, unbiased estimation methods, rigorous statistical analysis, fair training sets, and meaningful explainability, all of which are essential to building robust and reliable Machine Learning products. In addition to standard classification, the book discusses unsupervised learning, regression, image segmentation, and anomaly detection. The book also covers topics such as industry-strength evaluation, fairness, and responsible AI. Implementations using Python and Scikit-learn are available on the book's website.

The book plays a number of important roles. Foremost, it is meant to educate Machine Learning practitioners about the need for a thorough evaluation and the fact that evaluation is intimately linked to the application at hand. There are many evaluation tools appropriate for different applications, and each focuses on the specific goals the application is trying to achieve. Is the goal to diagnose patients as accurately as possible, like in a medical screening device? Is it to balance resources and needs in the fairest possible way, like in a public health program? Is it to minimize financial risk to make a business viable? Whatever the purpose of the evaluation, a particular set of techniques will be most appropriate to handle it, and the central purpose of the book is to expose the reader to these different tools so that, when the time comes, he or she will be able to select those most appropriate for the purpose of the application.

The second goal of the book is to sensitize the reader to the fact that neither Machine Learning nor its evaluation should be approached with certainty. Machine Learning can produce unexpected results and evaluation, while able to roughly assess the worth of a Machine Learning-based system, can miss some very important limitations of that system. Nevertheless, neither the learning algorithms nor their evaluation are the results of random calculations, and by considering the fascinating mechanisms used by both learning algorithms and evaluation mechanisms, we hope to give readers the tools necessary to either build sufficient trust in an algorithm or know when to reject one as not being reliable enough for deployment.

On the practical side, the third goal of the book is to provide tools that make the implementation of the evaluation process easier. Some of these tools already exist, and in such cases, we point the reader to them and illustrate their uses.

Finally, the fourth role filled by our book concerns the future. This is the beginning of Machine Learning deployment. The evaluation of Machine Learning systems is bound to evolve and improve as more and more approaches get deployed. For the time being, the certainty with which evaluation tools are assessed depends on statistical principles rather than customer reactions to deployed products.