From an open source perspective, although by default none of our solutions are exposed to the internet, some do leverage the impacted versions of Fluent Bit. We have released updated versions in those instances. See below for details.

• Tailing sidecar operator version 0.13.0 contains the fix.
• Kubernetes Collection Helm Chart v4.7.1 contains the above fix.
• Kubernetes Collection Helm Chart v3.19.3 contains the above fix and a separate upgrade to Fluent Bit.
• No upgrades available for Kubernetes Collection Helm Chart version 2. Support for this version ended on 2023-07-20. Customers are encouraged to upgrade to v4.7.1.
• No upgrades available for Helm Operator. An upgrade to version 4 of the Helm Chart (where there’s no Fluent Bit) is in progress.

We will continue to monitor and update as appropriate.

To our valued customers:

We want to provide you with an update on Sumo Logic’s recent security incident. We take the safety and reliability of our platform seriously. This is why we took immediate action to secure our platform as soon as we detected a potential security incident, including the recommendation to rotate all credentials.

We are grateful to share that the diligent investigation led by our security and engineering teams uncovered no proof of customer data impact and no threat of customer data impact present. These findings were verified by third-party forensic experts and the investigation of this incident is now complete and closed.

We remain committed to providing all of our customers with secure and reliable digital experience and are doing everything we can to emerge safer from this incident. To that end, we will be undertaking additional evaluation to learn from this incident and identify any measures or modifications to prevent future incidents.

As we have done throughout this process, we encourage you to keep an eye on the Security Response Center where we have posted pertinent updates. We also would like to provide you with additional tools and information.

FURTHER STEPS YOU CAN TAKE

While this investigation is complete and closed, and there is no proof of customer data impact and no threat of customer data impact present, Sumo Logic is providing the Indicators of Compromise (IOCs) and supporting documentation so customers can inspect their own environments, which we recommend.

Indicators of Compromise

34.201.113.45

66.225.222.68

159.223.118.253

54.183.5.235

34.207.95.146

44.203.0.45

45.154.98.33

18.189.57.10

185.220.101.56

34.238.239.207

185.220.101.58

45.76.10.28

How to Inspect Your Own Environments

As we formally close this investigation, we want to share some of the techniques we used internally, leveraging the Sumo Logic platform to further guide you in further inspecting your environments and ensuring your security posture.

We used our own solutions within Sumo Logic. In this particular instance, we used both our Cloud SIEM to search our environment for Indicators of Compromise (IOCs) and malicious IPs. In addition, we also used our Cloud Infrastructure Security (CIS) solution for enterprise audit activities such as access key creation, deletion and active vs. inactive. Below is a sample query you can run within your Sumo Logic instance. This will return any source categories where the IOCs are present and will show you where to do further investigation.

Example Search in Sumo Logic

_sourcecategory=* (“34.201.113.45” or “66.225.222.68” or “159.223.118.253” or “54.183.5.235” or “34.207.95.146” or “44.203.0.45” or “45.154.98.33” or “18.189.57.10” or “185.220.101.56” or “34.238.239.207” or “185.220.101.58” or “45.76.10.28”)
| timeslice 1m
| parse regex “(?<ioc_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})” multi
| where ioc_ip in (“34.201.113.45″,”66.225.222.68″,”159.223.118.253″,”54.183.5.235”, “34.207.95.146”, “44.203.0.45”, “45.154.98.33”, “18.189.57.10”, “185.220.101.56”, “34.238.239.207”, “185.220.101.58”, “45.76.10.28”)
| count _timeslice, _sourcecategory, ioc_ip

If you have questions about this guidance, don’t hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

Thank you for your patience and understanding throughout this process. We look forward to continuing to help our customers turn insights into action and deliver reliable and secure digital experiences.

• Sumo Logic API access keys

What you could also rotate as an additional precautionary measure:

• Third-party credentials that have been stored with Sumo as part of webhook connection configuration

If you have questions about steps to take, don’t hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

———

If you have questions about steps to take, don’t hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

———

If you have questions about steps to take, don’t hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

Specifically, we advise you to rotate immediately:

• Sumo Logic API access keys

What you could also rotate as an additional precautionary measure:

• Third-party credentials that have been stored with Sumo as part of webhook connection configuration

———

WE NO LONGER RECOMMEND THE FOLLOWING MEASURES FOR THIS INCIDENT:

• Sumo Logic installed collector credentials
• Third-party credentials that have been stored with Sumo for the purpose of data collection by the hosted collector (e.g., credentials for S3 access)
• User passwords to Sumo Logic accounts

———

If you have questions about steps to take, don’t hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

Here is a direct link to the playbook: link

The information contained in this website is provided “as is,” without any warranty of any kind, either express or implied. Users are solely responsible for adequate protection and backup of the data and equipment used in connection herewith. If Users require assistance, please contact our Support team via opening a ticket using our Support Console.

November 8, 2023

As an outcome of our ongoing investigation, we are reducing the scope of the additional precautionary measures mentioned in our November 7th message. Here is the updated recommendation:

What you could also rotate as an additional precautionary measure:

• Third-party credentials that have been stored with Sumo as part of webhook connection configuration

November 7, 2023

To Our Valued Customers:

At Sumo Logic, ensuring the security and reliability of our customers’ digital experience is our top priority. We have always placed great emphasis on protecting our customers against threats, and we understand and deeply value the trust our customers place in us.

To that end, we are writing to notify you, as a precautionary measure, of a possible security incident within our platform.

WHAT HAPPENED:

On Friday, November 3rd, 2023, Sumo Logic discovered evidence of a potential security incident. The activity identified used a compromised credential to access a Sumo Logic AWS account. We have not at this time discovered any impacts to our networks or systems, and customer data has been and remains encrypted.

WHAT HAVE WE DONE:

Immediately upon detection we locked down the exposed infrastructure and rotated every potentially exposed credential for our infrastructure out of an abundance of caution. We are continuing to thoroughly investigate the origin and extent of this incident. We have identified the potentially exposed credentials and have added extra security measures to further protect our systems. This includes improved monitoring and fixing any possible gaps to prevent any similar events and we are continuing to monitor our logs to look for further signs of malicious activity. We have taken actions to stop the threat to our infrastructure and are advising customers to rotate their credentials.

WHAT SHOULD YOU DO:

We recommend that customers rotate credentials that are either used to access Sumo Logic or that you have provided to Sumo Logic to access other systems. Specifically:

What we advise you rotate immediately:

• Sumo Logic API access keys (If you need assistance with this, please contact Sumo Support at https://support.sumologic.com/support/s/)

What you could also rotate as an additional precautionary measure:

• Sumo Logic installed collector credentials
• Third-party credentials that have been stored with Sumo for the purpose of data collection by the hosted collector (e.g., credentials for S3 access)
• Third-party credentials that have been stored with Sumo as part of webhook connection configuration
• User passwords to Sumo Logic accounts

If you have questions about steps to take, please do not hesitate to contact our customer support team at https://support.sumologic.com/support/s/

WHAT HAPPENS NEXT:

While the investigation into this incident is ongoing, we remain committed to doing everything we can to promote a safe and secure digital experience.

We will directly notify customers if evidence of malicious access to their Sumo Logic accounts is found. Customers may find updates at our Security Response Center.

Your security remains our top priority and we want to reiterate how much we value you putting your trust in us. Thank you for your understanding through this process.

We use a custom SumoLog4Layout library that never invokes custom lookups (as compared to Apache Log4j) so the Sumo Logic Service was never impacted.