{"id":4140,"date":"2024-11-07T08:00:00","date_gmt":"2024-11-07T08:00:00","guid":{"rendered":"http:\/\/www.sumologic.com\/blog\/ai-driven-low-noise-alerts"},"modified":"2025-11-07T16:29:30","modified_gmt":"2025-11-08T00:29:30","slug":"ai-driven-low-noise-alerts","status":"publish","type":"blog","link":"https:\/\/www.sumologic.com\/blog\/ai-driven-low-noise-alerts","title":{"rendered":"Reduce alert noise, automate incident response and keep coding with AI-driven alerting"},"content":{"rendered":"\n<section class=\"e-stn e-stn-0d652506f82b000a392973813b918ee25d5b4211 e-stn--glossary-inner-content e-stn--table-of-content\"><div class=\"container\">\n<div class=\"wp-block-b3rg-row e-row row\">\n<div class=\"wp-block-b3rg-column e-col e-col-1f7b3997080fc292474d26ff00c905d99d3520fa e-col--content-wrapper  col-sm-12 col-lg-12 col-xl-12\">\n<div class=\"e-div e-div-a1b32f66e1749758df41d5aea14f647cd10e362c e-div--card-btn-link\"><div class=\"e-img \">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"400\" src=\"http:\/\/www.sumologic.com\/wp-content\/uploads\/AI_Alerting_Blog_header_700x200.png\" alt=\"AI-driven alerting\" class=\"wp-image-4134\" title=\"\"><\/figure>\n<\/div>\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-2cbfa69228bced6886c8fbce9549594c\">Noisy monitors can lead to <a href=\"https:\/\/www.sumologic.com\/blog\/reducing-alert-fatigue-grouping-customization\/\">alert fatigue<\/a>, which frustrates engineers and hinders innovation.  With our patent-pending anomaly detection capabilities built on the power of AI, you can eliminate 60-90% of alerts. A unique differentiator, Sumo Logic\u2019s alerts can also trigger one or more playbooks to drive auto-diagnosis or remediation and accelerate time to recovery for application incidents. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-2de02e387b9ea6da32065617dc053787\">Faster issue remediation means engineers can focus more time on development and releasing software. The combination of next-generation anomaly detection and automation is part of our AI-driven alerting capabilities available to all Sumo Logic customers that will change how you troubleshoot. <\/p>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-c0917d098d677af7ccd374ab28e25999\" id=\"alerting_challenges\">Alerting challenges<\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-d014f05c992380d8243167433bc736dc\">The noisiest 5% of monitors on logs and metrics used by Sumo Logic trigger seven times per day, as in the graph below. Anecdotally, half of these noisy alerts trigger after regular work hours. These stats imply that the volume of alerts from modern applications can overwhelm on-call teams. <\/p>\n\n\n<div class=\"e-img \">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"808\" height=\"652\" src=\"http:\/\/www.sumologic.com\/wp-content\/uploads\/Alerting-challenges.png\" alt=\"Alerting challenges\" class=\"wp-image-4135\" title=\"\"><figcaption class=\"wp-element-caption\">Top 5% of noisiest monitors trigger seven times per day<\/figcaption><\/figure>\n<\/div>\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-04438f8ca8aeb67dcf1a1ee74c8cf7ea\">A substantial portion of these alerts are often irrelevant or insignificant, contributing to alert fatigue among operators. The image below shows a monitor for a Sumo Logic customer, which, over three days, generated two false positives (i.e. false alarm) and one false negative (i.e. alert was not triggered when it should have). False positives are a distraction that pulls engineers away from their focused work, while false negatives hide genuine problems that developers actually need to act on.<br><\/p>\n\n\n<div class=\"e-img \">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1096\" height=\"732\" src=\"http:\/\/www.sumologic.com\/wp-content\/uploads\/Customer-monitor.png\" alt=\"Customer monitor\" class=\"wp-image-4136\" title=\"\"><figcaption class=\"wp-element-caption\">Customer monitor with two false positives and one false negative<\/figcaption><\/figure>\n<\/div>\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-f7cce1213fb363f2124489bae8e44562\">For AI-driven alerting, we were convinced we could leverage real-time AI and <a href=\"https:\/\/www.sumologic.com\/glossary\/machine-learning\/\">ML<\/a> to drive up our accuracy and keep developers focused on the work they do best. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-e4fdb8934b72d7267b4ea22afdf3b4aa\">While reducing false positives helps developers with alert fatigue, when incidents do occur, you want to resolve them quickly. AI-driven alerting also features playbooks for automating incident diagnosis and if necessary, recovery.<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-0b7b14d5018a4cc4ab7e51667fc1321b\" id=\"why_are_monitors_noisy?\">Why are monitors noisy?<\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-aea6a2d15af319782524bb39f4a47124\">First-generation anomaly detection, such as Sumo Logic <a href=\"https:\/\/help.sumologic.com\/docs\/search\/search-query-language\/search-operators\/outlier\/\" target=\"_blank\" rel=\"noopener\">outlier<\/a>, learns dynamic baselines from recent data points and can avoid the problem of finding optimum static thresholds. However, these techniques can still generate false positives because they:<\/p>\n\n\n\n<ul>\n<li><strong>Don&#8217;t adjust for seasonality<\/strong>, especially longer range periodicity such as weekly periodicity or weekday\/weekend periodicity. As a result, some monitors based on first-generation anomaly detection might trip on weekends, as they do not factor the expected dip on weekends for most business apps, which is particularly annoying for false alarms.<\/li>\n<li><strong>Require tuning of lots of parameters <\/strong>based on periodic assessment of false positives and false negatives. During our customer previews, we assessed that many customer on-call engineers spend a lot of time tweaking monitor thresholds or parameters of an AI-driven alerting feature,<\/li>\n<li><strong>Unable to support contextual and dynamic thresholds<\/strong>. For example, for some signals in some contexts, you may want to wait for ten minutes of sustained degradation before triggering an alert, while in others you may want to trigger an alert even for a single anomalous data point.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-5a0e44a094b650efd9174dfce3e0862d\" id=\"the_benefits_of_ai-driven_alerting\">The benefits of AI-driven alerting<\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-c565928af42d618a8eec1f06fe8e5313\">AI-driven alerting addresses challenges with first-generation anomaly detection through the following strategies:<\/p>\n\n\n\n<ul>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-2a7c746bff9786bf9876abfe60c18f27\"><strong>Model-driven anomaly detection<\/strong>: AI-driven alerts use 60 days of historical data (when available) to train and test an ML model so that hourly, daily and weekly (especially, weekday\/weekend) seasonality are factored into baselines. An anomaly is an unusual datapoint compared to the baseline or expected value. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-87beeca7d71b4214d646cafd7e6b78b5\"><strong>AutoML:<\/strong> AI-driven alerts embed an AutoML framework where the analytics tune itself based on model performance on training datasets. Simply put, AutoML supports a \u201cset it and forget it\u201d experience with minimal user intervention.<\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-929cb668154f6754ff339d03eca8e778\"><strong>Model contextual and dynamic thresholds<\/strong>: AI-driven alerts have a sensitivity setting (low sensitivity for signals that are expected to be noisy and high sensitivity for critical indicators). Additionally, the user can configure the incident detector based on context. For example, in the Cluster detector, the user can specify how many data points in a detection window of say 5m need to be unusual before triggering an alert.<\/p>\n\n\n\n<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-4b3e28e68ad5f554bde521ddfeaa61f9\" id=\"ai-driven_alerting_case_study\">AI-driven alerting case study<\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-5a4d4fa8aee504a4def551311a996404\">One of our preview customers for AI-driven alerting is a B2C modern application company that had many first-generation Sumo Logic outliers that were noisy primarily because they missed the weekday\/weekend periodicity of their signals. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-0bc8161ebe3214e37d08db9813e8a5d5\">AI-driven alerting successfully modeled the periodicity in the data as indicated in the blue line in the chart below, while the red lines are the upper and lower bounds predicted by the ML model. With AI-driven alerting, false alarms were successfully mitigated while alerts were triggered on genuine issues. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-da0032710605e2efd5031e2aecd995b6\"><\/p>\n\n\n<div class=\"e-img \">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"720\" height=\"427\" src=\"http:\/\/www.sumologic.com\/wp-content\/uploads\/AI-driven-alerting-case-study.png\" alt=\"AI-driven alerting case study\" class=\"wp-image-4137\" title=\"\"><figcaption class=\"wp-element-caption\">AI-driven monitor successfully models weekend\/weekday periodicity<\/figcaption><\/figure>\n<\/div>\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-4398ab157b3fdf1342bf636b2b45aa07\" id=\"incident_response_automation\">Incident response automation<\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-f29f116f2af9931e8f918179f083d853\">When incidents are detected correctly via anomaly detection or otherwise, you want to resolve them quickly to minimize customer impact and lost revenue. Recovery time for production incidents is about 30 minutes, which is driven largely by the ad hoc nature of reading through text playbooks, contacting subject matter experts, collecting diagnostics, forming hypotheses and taking action. What if diagnosis and\/or recovery time could be reduced to five minutes through automation? <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-a98f018f2c666c366bd534f96940c11c\">Sumo Logic <a href=\"https:\/\/help.sumologic.com\/docs\/platform-services\/automation-service\/\" target=\"_blank\" rel=\"noopener\">Automation Service<\/a> is now integrated with monitors. Any logs or metrics monitor can be associated with one or more playbooks authored by subject matter experts in the Automation Service. When the monitor triggers, the playbooks execute and cut minutes and hours from the response. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-190c051facb6c8a1bac3c4a1d05854d2\"><br><\/p>\n\n\n<div class=\"e-img \">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"724\" height=\"626\" src=\"http:\/\/www.sumologic.com\/wp-content\/uploads\/Incident-response-automation.png\" alt=\"Incident response automation\" class=\"wp-image-4138\" title=\"\"><\/figure>\n<\/div>\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-ca1cfc30bfacb466a9e22832eaac9648\">Here is an example of an auto-diagnosis playbook, in response to a site-down alert, where the customer is running six log searches and one metrics search in parallel, collating the results and alerting an on-call user with a summary of the incident. In some cases, the root cause might be part of the summary; in other cases, the playbook helps eliminate known root causes so that the on-call engineer can begin an ad hoc investigation. Either way, this auto-diagnosis playbook reduced the recovery time.<\/p>\n\n\n<div class=\"e-img \">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"720\" height=\"356\" src=\"http:\/\/www.sumologic.com\/wp-content\/uploads\/Auto-diagnosis-playbook-example.png\" alt=\"Auto-diagnosis playbook example\" class=\"wp-image-4139\" title=\"\"><figcaption class=\"wp-element-caption\">Auto-diagnosis playbook example<\/figcaption><\/figure>\n<\/div>\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-fcabdfc525d153066fc3af8c7934e0af\">While these examples are related to application incidents, AI-driven alerting is also relevant for security alerts, by cutting noise and automating incident response through playbooks. Many <a rel=\"noreferrer noopener\" href=\"https:\/\/www.sumologic.com\/solutions\/cloud-siem\/\">Cloud SIEM<\/a> customers use playbooks already; with this release, playbooks can be attached to any logs or metrics-based security monitor.&nbsp;With <a rel=\"noreferrer noopener\" href=\"https:\/\/www.sumologic.com\/blog\/log-everything-zero-dollar-ingest\/\">Flex Licensing<\/a>, our aim is to cover 100% of your logs and use cases.<\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-edbf22b678b008e65b9cb12a875a30bb\">AI-driven alerting reduces alert noise and accelerates incident diagnosis and recovery time, changing how you troubleshoot and secure your applications and infrastructure. <a href=\"https:\/\/www.sumologic.com\/guides\/machine-data-analytics\/\">Learn more about AI and log analytics<\/a>, or <a href=\"https:\/\/www.sumologic.com\/sign-up\/\">start your free trial<\/a> and test it for yourself.<\/p>\n\n\n\n<div id=\"wistia-block_122173e9b2c98f964f023ec5160a3d2b\" class=\"wistia-video-block\" data-wistia-video=\"wistia_cwu3tr9d7o\">\n\n    <script src=\"https:\/\/fast.wistia.com\/player.js\" async><\/script>\n    <script src=\"https:\/\/fast.wistia.com\/embed\/cwu3tr9d7o.js\" async type=\"module\"><\/script>\n\n    <style>\n        wistia-player[media-id='cwu3tr9d7o']:not(:defined) {\n            background: center \/ contain no-repeat url('https:\/\/fast.wistia.com\/embed\/medias\/cwu3tr9d7o\/swatch');\n            display: block;\n            filter: blur(5px);\n            padding-top: 56.25%;\n        }\n    <\/style>\n\n    <wistia-player\n        media-id=\"cwu3tr9d7o\"\n        aspect=\"1.7777777777777777\">\n    <\/wistia-player>\n\n<\/div>\n\n<style>\n    div[data-wistia-video=\"wistia_cwu3tr9d7o\"] {\n        position: relative;\n        width: 100%;\n        padding-top: 56.25%;\n        background: center \/ cover no-repeat url('https:\/\/fast.wistia.com\/embed\/medias\/cwu3tr9d7o\/swatch');\n    }\n\n    div[data-wistia-video=\"wistia_cwu3tr9d7o\"] wistia-player {\n        position: absolute;\n        top: 0;\n        left: 0;\n        width: 100%;\n        height: 100%;\n        filter: none;\n    }\n<\/style>\n<\/div>\n<\/div>\n<\/div>\n<\/div><\/section>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":42,"featured_media":25118,"template":"","meta":{"_acf_changed":false,"show_custom_date":false,"custom_date":"","featured":false,"featured_image":0,"learn_more_label":"","image_alt_text":"","learn_more_type":"","show_popup":false,"learn_more_link_file":0,"event_date":false,"event_start_date":"","event_end_date":"","place_holder_image_url":"","post_reading_time":"4","notification_enabled":false,"notification_text":"","notification_logo":"","notification_expiration_time":0,"is_enable_transparent_header":false,"selected_taxonomy_terms":{"blog-category":[125,127],"blog-tag":[],"translation_priority":[]},"selected_primary_terms":[],"learn_more_link":[],"featured_page_list":[],"notification_enabled_post_list":[],"_gspb_post_css":"","_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"4668,71369,71176","_relevanssi_noindex_reason":"","inline_featured_image":false,"footnotes":""},"blog-category":[125,127],"blog-tag":[],"class_list":["post-4140","blog","type-blog","status-publish","has-post-thumbnail","hentry","blog-category-devops-it-operations","blog-category-secops-security"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4140","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/users\/42"}],"version-history":[{"count":5,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4140\/revisions"}],"predecessor-version":[{"id":59948,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4140\/revisions\/59948"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/media\/25118"}],"wp:attachment":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/media?parent=4140"}],"wp:term":[{"taxonomy":"blog-category","embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog-category?post=4140"},{"taxonomy":"blog-tag","embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog-tag?post=4140"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}