Название: Open Source Observability: Disaggregating the Observability Stack for Scale, Cost, and Flexibility
Автор: Peter Corless, Neha Pawar
Издательство: O’Reilly Media, Inc.
Год: 2025-04-24
Язык: английский
Формат: pdf, epub
Размер: 10.1 MB

As modern applications grow in complexity, traditional observability solutions can be costly, rigid, and limiting. This report explores a new approach—disaggregated, modular observability stacks that offer flexibility, cost efficiency, and data autonomy. Authors Peter Corless and Neha Pawar provide engineering teams with a blueprint for designing tailored observability solutions using the best open source tools for data collection, storage, querying, and visualization.

This book is a vital guide for practitioners making critical choices for their own organizations, employing best-of-breed options at each and every layer of their observability stacks. The term observability has come to be used as an umbrella term for all software telemetry, encompassing monitoring, logging, exception handling, profiling, and more. But if you zoom out a bit, the linguistic shift from monitoring to observability over the past decade underlines an epistemic sea change in how we understand software itself.

Observability is no longer just about the signals we use to operate our code; it’s about how we develop our code. The sociotechnical systems that make up software delivery are all about feedback loops, and observability itself is the ur–feedback loop, the sense-making for how we build software. Software engineering can be such a creative, dynamic, collaborative, and fun career choice. We get paid to solve puzzles! All day! Yet so many of us get bogged down in roles that feel anything but fun. Observability can’t fix everything…but it can fix a lot of things. The joy of engineering is in creation. Engineers who can ship fast are usually happy engineers.

Observability, also referred to as O11y, is the ability to understand a system’s internal state by observing and analyzing the data it generates. It is an ever-evolving body of knowledge, policies, procedures, best practices, tools and technology. This data is used to detect and diagnose problems, ensure optimum performance, and maintain system security and reliability. Such data can and should also be used to drive continuous improvements in the observed code and systems architecture. Observability allows operators to know if systems are running well.

This is a big moment for open source in observability, as Open­Telemetry has overtaken Kubernetes as the most popular project by some metrics. And not a moment too late. Our lack of a shared substrate—shared semantic conventions—has held us back as an industry for years, maybe decades. It has allowed vendors to maintain their business by locking customers in, rather than by providing an excellent experience. It’s also a big moment in the industry, as AI-generated code is causing unprecedented anxiety over the future of software engineering. I for one am not worried. The hardest part of software engineering has never been writing code, and the heart of software engineering is not about writing code. Software engineering is about solving business problems with technology. The hardest part of software has always been making sense of it, creating shared understanding, and making wise decisions about how to govern it over its lifecycle, to build enduring value for users and businesses. Modern observability is about meeting that need, translating business needs into the language of software and vice versa.

Contents: