{"id":4789,"date":"2023-05-04T07:00:00","date_gmt":"2023-05-04T07:00:00","guid":{"rendered":"http:\/\/www.sumologic.com\/blog\/tuning-cloud-siem-machine-learning"},"modified":"2025-06-17T07:59:58","modified_gmt":"2025-06-17T15:59:58","slug":"tuning-cloud-siem-machine-learning","status":"publish","type":"blog","link":"https:\/\/www.sumologic.com\/blog\/tuning-cloud-siem-machine-learning","title":{"rendered":"Fine-tuning Cloud SIEM detections through machine learning"},"content":{"rendered":"\n<section class=\"e-stn e-stn-0d652506f82b000a392973813b918ee25d5b4211 e-stn--glossary-inner-content e-stn--table-of-content\"><div class=\"container\">\n<div class=\"wp-block-b3rg-row e-row row\">\n<div class=\"wp-block-b3rg-column e-col e-col-1f7b3997080fc292474d26ff00c905d99d3520fa e-col--content-wrapper  col-sm-12 col-lg-12 col-xl-12\">\n<div class=\"e-div e-div-a1b32f66e1749758df41d5aea14f647cd10e362c e-div--card-btn-link\"><div class=\"e-img \">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"400\" src=\"http:\/\/www.sumologic.com\/wp-content\/uploads\/blog-Fine-tuning-Cloud-SIEM-header-1.png\" alt=\"ML-powered SIEM tuning\" class=\"wp-image-4787\" title=\"\"><\/figure>\n<\/div>\n\n\n<p><\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-16f32df2ee34203e3d9d9a869bf0bc25\">Security engineering teams spend hours every week tuning their <a href=\"https:\/\/www.sumologic.com\/glossary\/siem\/\">security information and event management (SIEM)<\/a> systems to ensure that they are effective at detecting security threats and minimizing false positives. Such \u201ctuning tax\u201d is common as customers add new <a href=\"https:\/\/www.sumologic.com\/guides\/siem\" data-type=\"resource\" data-id=\"3026\">SIEM<\/a> rules to cope with rapidly changing threat landscape and attacker tactics and as their attack surface evolves through automated changes to their application and infrastructure stacks.<\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-8774f19995ab5e0dc0078600480c0cc9\">Just as you would engage a personal trainer to get the best results for time spent at the gym, Insight Trainer learns from your Signals and Insights data to identify Tuning Expressions and adjustments to rule severities to optimize your<a href=\"https:\/\/www.sumologic.com\/solutions\/cloud-siem-enterprise\/\"> Cloud SIEM<\/a> detections. With Insight Trainer, SOC teams can free themselves to focus on real threats by minimizing time spent investigating false positive Insights. <\/p>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-5126ee285e020c735569858eafa43235\" id=\"a_day_in_the_life_of_a_security_engineer\"><strong>A day in the life of a security engineer<\/strong><\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-5db7894a0cf42ca106fd75ea4079b477\">Many Sumo Logic customers divide responsibility between security\/detection engineers and the Security Operations Center (SOC) teams. The latter are the first-line responders for security incidents while the former administer and tune SIEM platforms like Sumo Logic for detection efficacy. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-fdf26f7485c5608dc3d5b030cd1a375d\">A typical process for tuning SIEMs involves the following steps:<br><\/p>\n\n\n\n<ol>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-7fc135f865599313608e21926d930aef\">Gather feedback from SOC teams. Many customers have weekly meetings between security engineers and SOC teams to assess the latest security findings including rules that seem to be noisy as they cause false alarms. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-809cee1fc8357bff220ea5c329d01915\">Refine detection rules. Security\/detection engineers analyze Insights and Signals from noisy rules within Cloud SIEM and assess if known and trusted users (e.g. system administrators) or machines (e.g. instances created by trusted automation scripts) dominate false positives. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-931893aa8ec2cd7cf9e848d8a14ae8cd\">Create Tuning Expressions that mute signals from these users or lower the severity or deactivate these rules. In any of these strategies, the impacted rules are expected to trigger fewer insights because Cloud SIEM\u2019s detection algorithm triggers Insights when the cumulative severity of rules triggered on an entity exceeds a threshold. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-e77fe5eba7b91c8fb466307df76fa69c\">Rinse and repeat the above process every week.<\/p>\n\n\n\n<\/li>\n<\/ol>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-ade11c15c8398af14e44732be4d1856a\">Security engineers also assess new and emerging threat patterns from penetration testing or other means and add new detection rules, initially as prototype rules. A prototype rule generates Signals, but those Signals will not contribute to Insights. Running the rule as a prototype for a while allows you to determine whether the rule is too noisy and fires too many Signals. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-9b3a50ca6c54557c3e974d972be299e4\">Many inefficiencies are apparent in this process including<br><\/p>\n\n\n\n<ul>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-eb20a74b652f795bb67df4dc0e93c5b6\">The tuning process is manual. There is only so much analysis security teams can do through log searches that it is unlikely for the tuning process to be comprehensive. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-4f69f54b09bafdfd8a97cfcb0e9af12d\">Finding optimal rule severities involves trial and error. Setting severities too low on high efficacy rules can miss real threats while setting them too high on low efficacy ones can result in too many false alarms. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-5c38f4a5e70ddf46e9803873ebf97302\">Given that security teams focus on false positives exclusively, they do not assess if certain rules are particularly helpful for detecting real threats or true positives. Increasing the severity of such rules can catch more threats sooner, given the way <a href=\"https:\/\/help.sumologic.com\/docs\/cse\/records-signals-entities-insights\/insight-generation-process\/#redundant-signal-suppression\" target=\"_blank\" rel=\"noopener\">algorithmic insights<\/a> are generated by Cloud SIEM. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-570057ab2c75d120e084f490fee0cd31\">Determining Tuning Expressions for noisy entities is also manual and unlikely to be comprehensive. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-5f3f46ebda7a19ef4039ad4e19ceca73\">New detection rules tend to be noisy. While Cloud SIEM\u2019s rule prototyping workflow allows experimentation before activating rules in production, that tuning process still involves trial and error. <\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-f9216dfc7d6f3c962b3d669228d81847\">Tuning has to be a continuous process as the threat, application and infrastructure landscape is constantly changing, especially with heavy automation through Infrastructure-as-code. Manual tuning processes can compound the \u201ctuning tax\u201d and consume valuable time from detection\/security engineering teams. <\/p>\n\n\n\n<\/li>\n<\/ul>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-b20456bb2782001f0d603bf85b5ada71\">Cloud SIEM Insight Trainer is designed to alleviate the manual tuning burden and maximize Cloud SIEM efficacy while still aligning with established security engineering processes as explained below. <\/p>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-3055761f0c5fbb191d29dfb161077bd8\" id=\"cloud_siem_insight_trainer\"><strong>Cloud SIEM Insight Trainer<\/strong><\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-0a26a6dca57e999f5552950cd577a13f\">Cloud SIEM Insight Trainer is a dashboard packaged within <a href=\"https:\/\/help.sumologic.com\/docs\/integrations\/sumo-apps\/cse\/\" target=\"_blank\" rel=\"noopener\">CSE<\/a> as shown in the screenshot. <\/p>\n\n\n<div class=\"e-img \">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1500\" height=\"1875\" src=\"http:\/\/www.sumologic.com\/wp-content\/uploads\/Cloud-SIEM-Insight-Trainer-1.png\" alt=\"Cloud SIEM Insight Trainer - dashboard\" class=\"wp-image-4788\" title=\"\"><\/figure>\n<\/div>\n\n\n<p><\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-0c84d382183843bb573a0dc632fdbf2b\">Insight Trainer learns from the history of your Cloud SIEM Insights and associated SOC team resolutions: true positive \/ resolved, false positive or no action. The algorithm calculates rule severities that accomplish the following goals<\/p>\n\n\n\n<ul>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-eb243287802ed75240a7f0f3662eb6af\">Preserve true positive \/ resolved counts<\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-7d735f1040ca5b92a0cac58a86cea4e1\">Minimize false positives<\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-78b456e6f027a515ca26d61813283aab\">Optionally, minimize no action. This option is appropriate for customers who use false positive and no action resolution states interchangeably.<\/p>\n\n\n\n<\/li>\n<\/ul>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-2948620d32ac77dd8f2c561403f108dd\">Both Sumo Logic Threat Labs and custom rules are included in the severity recommendations. While the details of our <a href=\"https:\/\/www.sumologic.com\/guides\/machine-data-analytics\/\">machine learning<\/a> algorithm are proprietary, all other things being equal, the algorithm recommends severity increases for rules that consistently contribute to true positive insights. Conversely, rules that consistently contribute to false positive insights are recommended for severity reduction. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-8b39903ecb884346560a46b22f499fb3\">In addition to rule severity recommendations, Insight Trainer analyzes the data to automatically identify noisy entities through a Tunability score. High Tunability score rules are those whose false positive (and optionally, no action) insights are caused by a small number of entities. Such rules would benefit from Tuning Expressions to suppress noisy entities, especially, if such entities are verified to be safe. Often, service accounts and administrative users may trigger Insights resolved as false positives or no-action insights by many customers. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-cc475687a4a66a2114a98e8d950d49cd\">To use Insight Trainer, we recommend the following workflow:<br><\/p>\n\n\n\n<ol>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-a98a53488dc6c8df57e96612ce541efd\">Review severity recommendations by rule<\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-5e4ab0689ca9541c72ea21d4c46ea530\">Assess dominant entities in false positives \/ no actions through rule Tunability scores<\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-8c3f895f1ef7ac9942bf7c7e21020753\">Evaluate and add tuning expressions for dominant entities, where possible.<\/p>\n\n\n\n<\/li>\n<li dir=\"ltr\">\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-ac27c320bcaac07a4112aa29b96bdace\">Adjust rule severities for other rules.<\/p>\n\n\n\n<\/li>\n<\/ol>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-3e58fbcdf9931bf743cfa118fe590b8c\">For best results, we also recommend security teams define and adhere to a standardized definition of true positive, false positive and no action resolution states. To avoid ambiguity during investigations, some Sumo Logic customers have used <a href=\"https:\/\/help.sumologic.com\/docs\/cse\/administration\/manage-custom-insight-resolutions\/#create-a-custom-sub-resolution\" target=\"_blank\" rel=\"noopener\">Insight sub-resolution states<\/a> to guide SOC responders along with upfront training on how to interpret and assign resolution states for common scenarios like admin activity, penetration testing or VPN-related Insight activity.<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-3ea10dd307c2453988205e0060a04d89\" id=\"insight_trainer_in_action\"><strong>Insight Trainer in action<\/strong><\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-0efa832a8fa3af5a92c2702935a91638\">We ran Insight Trainer over hundreds of Cloud SIEM customers and observed the following results. The average drop in false positives was 72% while no action insights was 74%. On average, only six or seven rules were recommended for severity decrease while zero or one rule was recommended for severity increase. In other words, tuning a few rules can result in dramatic reductions in False Positive \/ No Action Insights <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-751eb66fb33e60b9243236838a8da838\">Several Sumo Logic customers evaluated Cloud SIEM Insight Trainer over the past two months and were pleased with the results. A detections engineer from a financial services organization reported: <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-0b6b03b529728bb72265e622b849f943\"><em>\u201c&#8230;looking at these rules [severity recommendations], we recognize many common offenders\u2026changing severity by just 1 can reduce the false positive count [associated with rule] by 60% which is HUGE. Also, with growing AWS adoption alongside automation usage, we expect to see a lot of false positives for new rules. <\/em><\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-c2314a1f4c69475de9b703cd9d25739d\"><em>It is very difficult to tune these out through Tuning Expressions because service accounts used by automation are short-lived but are making lots of changes in a short amount of time &#8211; triggering a lot of insights. Such legitimate activity is the greatest contributor of false positives in an AWS environment and severity changes are a key lever to mitigate these.\u201d<\/em> <\/p>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-d377cbd8382a515927ae57740a5eac94\" id=\"chatgpt_envisions_insight_trainer\"><strong>ChatGPT envisions Insight Trainer<\/strong><\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-efb0e41dcc1f74026122ad4026b3cd6e\">As a fun exercise, we asked ChatGPT about typical false positive rates for SIEM systems and goals security engineering teams should aspire for, on this measure. Of course, ChatGPT is a limited tool without knowledge of the truth (whatever the truth even is!), but it can reflect fascinating patterns in existing data. Below is ChatGPT\u2019s response. <\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-b6921a268bc41850acffc065a38eb201\">\u201c<em>Generally, a high false positive rate can be an indicator that the SIEM system is not optimally configured or tuned. There is no one-size-fits-all answer to what a typical false positive rate for SIEMs should be. <\/em><\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-75f863106e9bf90e0427c7141d39c1bb\"><em>However, it is common to aim for a false positive rate of less than 5% to ensure that the security team can focus on genuine security incidents and reduce alert fatigue. It is important to note that reducing the false positive rate should not come at the expense of reducing the true positive rate, which measures how accurately the SIEM system identifies genuine security incidents. <\/em><\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-88cfdb1a518d60ce3626e132d007c031\"><em>The goal is to strike a balance between reducing false positives and ensuring that the SIEM system can accurately detect and respond to genuine security incidents<\/em>.\u201d<\/p>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-39a47a39b644305329af0085afcd5d00\">We could say that Insight Trainer would make ChatGPT happy if it were possible to make an AI language model happy!<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-eigengrau-color has-text-color has-link-color wp-elements-06ec3e7267a44fc35446071c0cf82608\" id=\"next_steps\"><strong>Next steps<\/strong><\/h2>\n\n\n\n<p class=\"has-delft-blue-color has-text-color has-link-color wp-elements-7cf2d6fe9aaed87c681dcdac6d09bfd2\">Cloud SIEM Insight Trainer is an essential tool for fine-tuning Cloud SIEM detections for your organization and helps focus SOC analysts\u2019 attention on high-risk, true positive insights. Start using <a href=\"https:\/\/help.sumologic.com\/docs\/cse\/rules\/insight-trainer\" target=\"_blank\" rel=\"noopener\">Insight Trainer<\/a> today, or learn how <a href=\"https:\/\/www.sumologic.com\/case-study\/roku\/\">Roku tunes its SIEM<\/a>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div><\/section>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":42,"featured_media":0,"template":"","meta":{"_acf_changed":false,"show_custom_date":false,"custom_date":"","featured":false,"featured_image":0,"learn_more_label":"","image_alt_text":"","learn_more_type":"","show_popup":false,"learn_more_link_file":0,"event_date":false,"event_start_date":"","event_end_date":"","place_holder_image_url":"","post_reading_time":"5","notification_enabled":false,"notification_text":"","notification_logo":"","notification_expiration_time":0,"is_enable_transparent_header":false,"selected_taxonomy_terms":{"blog-category":[126,127],"blog-tag":[]},"selected_primary_terms":[],"learn_more_link":[],"featured_page_list":[],"notification_enabled_post_list":[],"_gspb_post_css":"","_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"71176,71070,71043","_relevanssi_noindex_reason":"","inline_featured_image":false,"footnotes":""},"blog-category":[126,127],"blog-tag":[],"class_list":["post-4789","blog","type-blog","status-publish","hentry","blog-category-cloud-siem","blog-category-secops-security"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4789","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/users\/42"}],"version-history":[{"count":6,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4789\/revisions"}],"predecessor-version":[{"id":26811,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4789\/revisions\/26811"}],"wp:attachment":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/media?parent=4789"}],"wp:term":[{"taxonomy":"blog-category","embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog-category?post=4789"},{"taxonomy":"blog-tag","embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog-tag?post=4789"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}