{"id":4697,"date":"2022-09-23T13:53:23","date_gmt":"2022-09-23T17:53:23","guid":{"rendered":"https:\/\/solutionsreview.com\/network-monitoring\/?p=4697"},"modified":"2022-09-23T14:45:35","modified_gmt":"2022-09-23T18:45:35","slug":"simplifying-devops-monitoring-with-opentelemetry","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/","title":{"rendered":"Simplifying DevOps Monitoring with OpenTelemetry"},"content":{"rendered":"

\"OpenTelemetry\"<\/p>\n

As part of Solutions Review\u2019s Premium Content Series\u2014a collection of contributed columns written by industry experts in maturing software categories\u2014 Dotan Horovits of Logz.io<\/a> provides an in-depth overview of OpenTelemetry and how it can help simplify DevOps monitoring.<\/strong><\/em><\/p>\n

\"SR\u201cMicroservices\u201d is the new norm for building products these days. It bears many advantages in accelerating development, but it also makes systems more complex to monitor. These systems are often polyglot, leveraging multiple programming languages, each having its own way of being monitored. Furthermore, today\u2019s systems make extensive use of third-party tools and frameworks to accelerate development and keep the engineering team focused on the business differentiators. These third parties could be open-source projects, proprietary tools, or cloud services. And while we didn\u2019t write these frameworks, we still need to effectively monitor them, to gain end-to-end observability across our system.<\/p>\n

Observability is the paradigm that enables us to monitor and understand our systems. The formal definition, taken from Control Theory, discusses \u201ca measure of how well internal states of a system can be inferred from knowledge of its external outputs.\u201d<\/p>\n

In this article, I\u2019d like to look into the various types of observability data we need to collect and how to simplify the collection process using the OpenTelemetry open-source project, a fascinating project under the Cloud Native Computing Foundation.<\/p>\n

\t\t\t

\"Download<\/a><\/p>\n<\/div>\n\t\t<\/div><\/div><\/p>\n

OpenTelemetry: DevOps Monitoring Simplified.<\/strong><\/h2>\n
\n

Observability Signal Types<\/strong><\/h3>\n

Every system emits different telemetry signals, which help us understand its internal state. The classic signs are logs, metrics, and traces, often referred to as \u201cthe three pillars of observability.\u201d Logs and metrics have been around for decades, while Distributed Tracing is younger in comparison but gaining strong momentum. Other signal types are emerging, such as continuous profiling, but these are much earlier in the adoption cycle. The observability data also stems from a large variety of data sources, both in the application and in the infrastructure. A typical system may consist of a Python frontend, a Java backend, some databases, and a messaging service such as Kafka (or its cloud service equivalents) – each emitting its own logs, metrics, and other telemetry data in need of ingestion.<\/p>\n

Collecting such heterogeneous data in a unified manner has been a challenge for many years. Proprietary solutions provided by vendors failed to keep up with the ever-expanding ecosystem of third-party tools and the advancements in programming languages. Furthermore, each tool and each vendor had its own proprietary APIs and SDKs for instrumenting the application code, proprietary agents to collect and process the data, and proprietary protocol for transmitting the telemetry to the analytics backend. That effectively created data silos that prevented full observability. We needed to take a different approach and look at observability as a data analytics problem.<\/p>\n

Unified Data Collection with OpenTelemetry<\/strong><\/h3>\n

The open-source community that brought us Kubernetes and the cloud native ecosystem has also delivered open-source tools and standards to monitor them. One important project under the Cloud Native Computing Foundation<\/em> (CNCF) is OpenTelemetry, an observability framework that assists in generating and collecting telemetry data from cloud-native software.<\/p>\n

Let\u2019s take a closer look at what the OpenTelemetry framework provides us. For each programming language, OpenTelemetry offers a single API and SDK (i.e., client library) for instrumenting the application. It also provides a unified Collector that can collect telemetry from multiple sources, whether your application or infrastructure components, over various protocols. The Collector then processes the telemetry data and exports it to any observability analysis backend tool or downstream system over multiple protocols. It\u2019s important to note that the OpenTelemetry project does not provide the analysis backend nor take any stake in it. Last but not least, OpenTelemetry offers a unified protocol, OTLP, for transmitting logs, metrics, and trace data.<\/p>\n

OpenTelemetry isn\u2019t yet another open-source project. In fact, it\u2019s the most active project in the entire CNCF after Kubernetes. All the monitoring and observability vendors, as well as the major cloud providers, have started to align with OpenTelemetry, even at the cost of sunsetting lucrative proprietary agents. This is a positive sign that OpenTelemetry is set out to become the new de-facto standard for telemetry collection. Gartner’s recent Hype Cycle for Emerging Technologies (2022) even listed OpenTelemetry within the Innovation Trigger, estimating that it will reach the plateau on Gartner\u2019s hype cycle within 2-5 years.<\/p>\n

OpenTelemetry is a relatively young project, but it is already generally available for use with Distributed Tracing telemetry data and is in the Release Candidate stage (the step before general availability) for use with Metrics data. The least evolved of the signals is Logs, which will not be generally available before 2023. OpenTelemetry\u2019s roadmap also looks into additional signals beyond the \u201cthree pillars,\u201d with Continuous Profiling being the first one.<\/p>\n

Getting Started with OpenTelemetry<\/strong><\/h3>\n

OpenTelemetry is not a single monolithic project but rather a collection of projects run by multiple working groups. This can make starting with OpenTelemetry quite confusing. When you start with OpenTelemetry, it\u2019s important to know your tech stack.<\/p>\n

Start with these four questions, which will direct you to the components relevant to your system:<\/p>\n

Which programming languages?<\/strong><\/h4>\n

Which programming languages do you use? That will determine the OpenTelemetry APIs and SDKs relevant for you, and potentially also auto-instrumentation agents. Go the extra mile and determine the programming frameworks you use with each language to see which integrations exist for them.<\/p>\n

Which signal types and protocols?<\/strong><\/h4>\n

Next, determine which observability signals you intend to ingest to determine the relevant Collector receivers you\u2019d be using. Start by figuring out which signal types are of interest to you, among traces, metrics, and logs. Also, see which telemetry protocols you should support. This is especially important with brownfield projects, having existing and potentially legacy components already emitting telemetry in certain protocols to which you need to adhere.<\/p>\n

Which infrastructure components?<\/strong><\/h4>\n

Next, list down which are the sources of the signals, that is, the components you monitor. Many infrastructure components have their own formats and use designated receivers to ingest them. This is true for open-source tools such as Kubernetes, Kafka, MySQL, and HTTPD. It is also true for cloud services such as AWS X-Ray and GCP pubsub, or even with existing telemetry collectors such as CollectD or StatsD.<\/p>\n

Which backend analytics tools?<\/strong><\/h4>\n

Lastly, you\u2019d need to define which tool stack you intend to use for running analytics on your telemetry data. It may be an open-source tool, a proprietary one, or a cloud service. It could also be another downstream system that receives the data and processes it. This will help you determine the relevant Collector exporters you\u2019d be using.<\/p>\n

\n

Endnote<\/strong><\/h3>\n

OpenTelemetry is a young but promising project that is set out to become a de-facto standard. It also bears the promise of unified observability on the side of generating and collecting the observability data. Having this project under the wings of the Cloud Native Computing Foundation, alongside Kubernetes, Prometheus, and other leading projects in the space will further facilitate the collaboration to ensure compatibility across the stack.<\/p>\n

\t\t\t

\"Download<\/a><\/p>\n<\/div>\n\t\t<\/div><\/div><\/p>\n","protected":false},"excerpt":{"rendered":"

As part of Solutions Review\u2019s Premium Content Series\u2014a collection of contributed columns written by industry experts in maturing software categories\u2014 Dotan Horovits of Logz.io provides an in-depth overview of OpenTelemetry and how it can help simplify DevOps monitoring. \u201cMicroservices\u201d is the new norm for building products these days. It bears many advantages in accelerating development, […]<\/p>\n","protected":false},"author":330,"featured_media":4698,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[1,21],"tags":[259,1252,92,364,29,1769,8,217,1768],"acf":[],"yoast_head":"\nSimplifying DevOps Monitoring with OpenTelemetry<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Simplifying DevOps Monitoring with OpenTelemetry\" \/>\n<meta property=\"og:description\" content=\"As part of Solutions Review\u2019s Premium Content Series\u2014a collection of contributed columns written by industry experts in maturing software categories\u2014 Dotan Horovits of Logz.io provides an in-depth overview of OpenTelemetry and how it can help simplify DevOps monitoring. \u201cMicroservices\u201d is the new norm for building products these days. It bears many advantages in accelerating development, […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/\" \/>\n<meta property=\"og:site_name\" content=\"Network Monitoring Tools & Performance Software | Solutions Review\" \/>\n<meta property=\"article:published_time\" content=\"2022-09-23T17:53:23+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-09-23T18:45:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/solutionsreview.com\/network-monitoring\/files\/2022\/09\/Simplifying-DevOps-Monitoring-with-OpenTelemetry.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"400\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Dotan Horovits\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dotan Horovits\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/\",\"url\":\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/\",\"name\":\"Simplifying DevOps Monitoring with OpenTelemetry\",\"isPartOf\":{\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/solutionsreview.com\/network-monitoring\/files\/2022\/09\/Simplifying-DevOps-Monitoring-with-OpenTelemetry.jpg\",\"datePublished\":\"2022-09-23T17:53:23+00:00\",\"dateModified\":\"2022-09-23T18:45:35+00:00\",\"author\":{\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/#\/schema\/person\/fc99599b87e8a93be2513558a32c2cf5\"},\"breadcrumb\":{\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#primaryimage\",\"url\":\"https:\/\/solutionsreview.com\/network-monitoring\/files\/2022\/09\/Simplifying-DevOps-Monitoring-with-OpenTelemetry.jpg\",\"contentUrl\":\"https:\/\/solutionsreview.com\/network-monitoring\/files\/2022\/09\/Simplifying-DevOps-Monitoring-with-OpenTelemetry.jpg\",\"width\":800,\"height\":400,\"caption\":\"OpenTelemetry\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/solutionsreview.com\/network-monitoring\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Simplifying DevOps Monitoring with OpenTelemetry\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/#website\",\"url\":\"https:\/\/solutionsreview.com\/network-monitoring\/\",\"name\":\"Network Monitoring Tools & Performance Software | Solutions Review\",\"description\":\"Evaluating Enterprise NPM Software, Network Observability & Packet Analysis Tools.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/solutionsreview.com\/network-monitoring\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/#\/schema\/person\/fc99599b87e8a93be2513558a32c2cf5\",\"name\":\"Dotan Horovits\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/solutionsreview.com\/network-monitoring\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/84252cdad8eb181fd39c720bb47544d2?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/84252cdad8eb181fd39c720bb47544d2?s=96&d=mm&r=g\",\"caption\":\"Dotan Horovits\"},\"description\":\"Dotan Horovits lives at the intersection of technology, product and innovation. With over 20 years in the hi-tech industry as a software developer, architect and a product manager, he brings a wealth of knowledge in cloud, big data, DevOps and more. Horovits is an avid advocate of open-source and communities. An organizer of the Cloud Native Computing Foundation local chapter in Tel-Aviv, a podcaster at OpenObservability Talks, a blogger and a public speaker. Working as the principal developer advocate at Logz.io, Horovits evangelizes on Observability in IT systems using popular open source projects such as Elasticsearch, Prometheus, Jaeger and OpenTelemetry.\",\"url\":\"https:\/\/solutionsreview.com\/network-monitoring\/author\/dhorovits\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Simplifying DevOps Monitoring with OpenTelemetry","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/","og_locale":"en_US","og_type":"article","og_title":"Simplifying DevOps Monitoring with OpenTelemetry","og_description":"As part of Solutions Review\u2019s Premium Content Series\u2014a collection of contributed columns written by industry experts in maturing software categories\u2014 Dotan Horovits of Logz.io provides an in-depth overview of OpenTelemetry and how it can help simplify DevOps monitoring. \u201cMicroservices\u201d is the new norm for building products these days. It bears many advantages in accelerating development, […]","og_url":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/","og_site_name":"Network Monitoring Tools & Performance Software | Solutions Review","article_published_time":"2022-09-23T17:53:23+00:00","article_modified_time":"2022-09-23T18:45:35+00:00","og_image":[{"width":800,"height":400,"url":"https:\/\/solutionsreview.com\/network-monitoring\/files\/2022\/09\/Simplifying-DevOps-Monitoring-with-OpenTelemetry.jpg","type":"image\/jpeg"}],"author":"Dotan Horovits","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Dotan Horovits","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/","url":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/","name":"Simplifying DevOps Monitoring with OpenTelemetry","isPartOf":{"@id":"https:\/\/solutionsreview.com\/network-monitoring\/#website"},"primaryImageOfPage":{"@id":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#primaryimage"},"image":{"@id":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#primaryimage"},"thumbnailUrl":"https:\/\/solutionsreview.com\/network-monitoring\/files\/2022\/09\/Simplifying-DevOps-Monitoring-with-OpenTelemetry.jpg","datePublished":"2022-09-23T17:53:23+00:00","dateModified":"2022-09-23T18:45:35+00:00","author":{"@id":"https:\/\/solutionsreview.com\/network-monitoring\/#\/schema\/person\/fc99599b87e8a93be2513558a32c2cf5"},"breadcrumb":{"@id":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#primaryimage","url":"https:\/\/solutionsreview.com\/network-monitoring\/files\/2022\/09\/Simplifying-DevOps-Monitoring-with-OpenTelemetry.jpg","contentUrl":"https:\/\/solutionsreview.com\/network-monitoring\/files\/2022\/09\/Simplifying-DevOps-Monitoring-with-OpenTelemetry.jpg","width":800,"height":400,"caption":"OpenTelemetry"},{"@type":"BreadcrumbList","@id":"https:\/\/solutionsreview.com\/network-monitoring\/simplifying-devops-monitoring-with-opentelemetry\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/solutionsreview.com\/network-monitoring\/"},{"@type":"ListItem","position":2,"name":"Simplifying DevOps Monitoring with OpenTelemetry"}]},{"@type":"WebSite","@id":"https:\/\/solutionsreview.com\/network-monitoring\/#website","url":"https:\/\/solutionsreview.com\/network-monitoring\/","name":"Network Monitoring Tools & Performance Software | Solutions Review","description":"Evaluating Enterprise NPM Software, Network Observability & Packet Analysis Tools.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/solutionsreview.com\/network-monitoring\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/solutionsreview.com\/network-monitoring\/#\/schema\/person\/fc99599b87e8a93be2513558a32c2cf5","name":"Dotan Horovits","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/solutionsreview.com\/network-monitoring\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/84252cdad8eb181fd39c720bb47544d2?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/84252cdad8eb181fd39c720bb47544d2?s=96&d=mm&r=g","caption":"Dotan Horovits"},"description":"Dotan Horovits lives at the intersection of technology, product and innovation. With over 20 years in the hi-tech industry as a software developer, architect and a product manager, he brings a wealth of knowledge in cloud, big data, DevOps and more. Horovits is an avid advocate of open-source and communities. An organizer of the Cloud Native Computing Foundation local chapter in Tel-Aviv, a podcaster at OpenObservability Talks, a blogger and a public speaker. Working as the principal developer advocate at Logz.io, Horovits evangelizes on Observability in IT systems using popular open source projects such as Elasticsearch, Prometheus, Jaeger and OpenTelemetry.","url":"https:\/\/solutionsreview.com\/network-monitoring\/author\/dhorovits\/"}]}},"_links":{"self":[{"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/posts\/4697"}],"collection":[{"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/users\/330"}],"replies":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/comments?post=4697"}],"version-history":[{"count":0,"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/posts\/4697\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/media\/4698"}],"wp:attachment":[{"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/media?parent=4697"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/categories?post=4697"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solutionsreview.com\/network-monitoring\/wp-json\/wp\/v2\/tags?post=4697"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}