{"id":4883,"date":"2023-03-28T07:00:00","date_gmt":"2023-03-28T07:00:00","guid":{"rendered":"http:\/\/www.sumologic.com\/blog\/chatgpt-cyber-defenders-ai"},"modified":"2025-05-08T19:11:01","modified_gmt":"2025-05-09T03:11:01","slug":"chatgpt-cyber-defenders-ai","status":"publish","type":"blog","link":"https:\/\/www.sumologic.com\/blog\/chatgpt-cyber-defenders-ai","title":{"rendered":"ChatGPT praise and trepidation – cyber defense in the age of AI"},"content":{"rendered":"\n
\n
\n
\n
\n
\"ChatGPT<\/figure>\n<\/div>\n\n\n

Authorship –
\nAI Contribution Score (ACS): 0%
\nHuman: 100%<\/p>\n\n\n\n

ChatGPT has taken the world by storm, so much so that we are all left guessing how far this will go. And it\u2019s not a trivial question, as it relates to the future of humanity itself. <\/p>\n\n\n\n

On one extreme, technology is increasing rapidly enough to synthesize some of the most fundamental parts of our existence\u2014communicating naturally with one another. That can be a scary thought. Many of us feel like Truman in his artificial world as he saw the button reading, “How’s it going to end?” and replied, “I’ve been wondering the same thing myself lately.” The only difference is we are willing participants in creating this artificial world. <\/p>\n\n\n\n

On the other extreme, we welcome new powerful tools that help us achieve real results in a short amount of time, giving us our only non-renewable resource \u2013 time. As with any new technology, all we can do is assess them as they develop and put in safety measures, or sometimes even countermeasures, and move forward. In the words of the CEO of OpenAI<\/a>, “We’ve got to be careful here. I think people should be happy that we are a little bit scared of this.” Nowhere is this more true than in the world of cybersecurity itself.<\/p>\n\n\n\n

One of the things that make large language models (LLM) so powerful is that a \u201clanguage\u201d is not limited to the spoken word but also includes programming languages. Because GPT is multilingual, it allows us to ask it a question in English and have it translate the answer into another language, such as Python. <\/p>\n\n\n\n

That is a game changer, but not new to ChatGPT. When Salesforce started exploring the possibilities of this over a year ago, Chief Scientist Silvio Savarese explained their project CodeGen<\/a>: <\/p>\n\n\n\n

\n

[With CodeGen] we envisioned a conversational system that is capable of having a discussion with the user to solve a coding problem or task. This discussion takes the form of an English text-based discourse between a human and a machine. The result is AI-generated code that solves the discussed problem.<\/p>\n<\/blockquote>\n\n\n\n

GitHub has Copilot and Copilot X which are built on GPT-3 and GPT-4 respectively and help developers write and debug their code, generate descriptions of pull requests, automatically tags them, and can provide AI-generated answers about documentation. Extrapolate this out, instead of a human doing the programming, machines are learning to program themselves, with the human providing only high-level guidance. What could possibly go wrong?<\/p>\n\n\n\n

Battle at the binary level<\/h2>\n\n\n\n

In cyber defense, adversaries are driven to discover new vulnerabilities and write exploits against those found and create new tools for breach, extortion, ransomware, remote access, and the rest. Exploit development is a very technical and challenging undertaking, and when exploits are created they are valuable. <\/p>\n\n\n\n

These tools are shared on black markets for selling and trading new zero-days, rootkits, and remote access trojans (RATs). Eventually, the defenders catch up and adversary tactics, techniques and procedures (TTPs) become part of the common body of knowledge of attack vectors and are entered into the common vulnerabilities and exposures (CVE) and MITRE ATT&CK hall of fame. Soon after, vendors push detections into their point solutions. <\/p>\n\n\n\n

As an example on the defending side, we actively fight to detect malicious binaries by their signatures, often in the form of their unique hash value or pattern matching of binary sequences. Once all the anti-virus vendors identify the malicious signatures, attackers have to find (or create) other tools that can go undetected.<\/p>\n\n\n\n

I recall experimenting with a little detection avoidance tool called DSplit. The concept was simple. Split binaries into two, run them both through the AV tool in which you\u2019re hoping to avoid detection, and see which file trips detection. Take the offending half, split it again and rinse and repeat. Eventually, you get down to a small enough chunk of bits that you can quite literally change a 0 to a 1, and then viola! \u2013 it passes undetected. <\/p>\n\n\n\n

That approach is somewhat shallow, and the reassembled binary may not run as expected, but it brings us back to ChatGPT. What if every malicious developer could create unique code at the outset with minimal coding skills? And that code itself could be polymorphic<\/a> in nature, always being unique with no static fingerprint or signature. Defenders and security vendors now have to refocus on behavior and anomaly detection. For a deep dive into the threat offensive AI poses to organizations, this paper<\/a> is worth a read.<\/p>\n\n\n\n

Again, this is nothing new, but yesterday’s detection strategies are now more stale than ever. It\u2019s critical that tools in the security stack are intelligent enough to correlate across the entire lifecycle of an attack and have an entity-centric view of activity in order to identify malicious patterns of behavior. Detecting the initial attack vector<\/a> may be ideal, but as it becomes more difficult, our tooling should also adopt proven approaches that rely on behavior-based anomalies. <\/p>\n\n\n\n

With the advent of ChatGPT, this type of detection becomes even more important.<\/p>\n\n\n\n

Of course, as the cat-and-mouse game levels up on both sides, machine learning, and AI can also be used in malware image classification at the binary level. In head-splitting terminologies, various machine learning approaches such as support vector machine, k-nearest neighbors, random forests, naive Bayes, and decision trees have been used to detect and classify known malware. <\/p>\n\n\n\n

But even more creative than these names are detection techniques where malware binaries have been converted into actual grayscale images, then using convolutional neural network (CNN) models for static malware classification by \u201ccomputer vision\u201d. In one test<\/a>, byte plot visualization of malware as grayscale images across 9,342 malware samples achieved 80-99% detection accuracy. <\/p>\n\n\n

\n
\"ChatGPT<\/figure>\n<\/div>\n\n\n

With the recent release of ChatGPT-4, it can \u201csee\u201d and understand images. And this may pave the way for unique ways for it to identify unique malicious code not seen (editor\u2019s note: hah!) before. Around and around we go with AI helping adversaries and defenders alike.<\/p>\n\n\n\n

Truth be told, I\u2019ve always been the guy in the back of the room rolling my eyes every time AI\/ML is mentioned by a security vendor. I\u2019m doing less eye-rolling these days, as AIOPs<\/a> have made leaps and bounds. Gartner\u2019s Market Guide for AIOps Platforms<\/a> detailed what features and use cases AI brings to the table. They defined AIOps as platforms that can analyze telemetry<\/a> and events, and identify meaningful patterns that provide insights to support proactive responses with five common characteristics:<\/p>\n\n\n\n

    \n
  1. \n\n\n\n

    Cross-domain data ingestion and analytics<\/p>\n\n\n\n<\/li>\n

  2. \n\n\n\n

    Topology assembly from implicit and explicit sources of asset relationship and dependency<\/p>\n\n\n\n<\/li>\n

  3. \n\n\n\n

    Correlation between related or redundant events associated with an incident<\/p>\n\n\n\n<\/li>\n

  4. \n\n\n\n

    Pattern recognition to detect incidents, their leading indicators or probable root cause<\/p>\n\n\n\n<\/li>\n

  5. \n\n\n\n

    Association of probable remediation<\/p>\n\n\n\n<\/li>\n<\/ol>\n\n\n\n

    I don\u2019t actually think the future will be in AIOps platforms that can ingest huge volumes of telemetry (logs, metrics, traces) and then identify malicious behaviors as they unfold. We already have tools doing this today in the form of security information and event management (SIEMs<\/a>), security orchestration, automation, and response (SOARs<\/a>) and user and entity behavior analytics (UEBA). <\/p>\n\n\n\n

    What I expect to see is fewer AI-dedicated \u201cplatform\u201d plays and more adoption of ML functionality across proven security solutions. Here at Sumo Logic, for example, our customers have found value in our ability to analyze global security signals in an obfuscated and anonymized way and then present back to customers a global view of attack vectors targeted against them as it compares to thousands of other similar customers. <\/p>\n\n\n\n

    These baselines help our customers optimize their security posture based on how unusual their security findings are compared to others. Because we can apply ML to huge data sets not available to siloed security teams, it\u2019s a value add that only comes with a multi-tenant cloud-native offering. Again, expect next-gen offerings to rapidly adopt these features, especially as Google, Apple, Meta, Baidu and Amazon release native services for developers to play with.<\/p>\n\n\n\n

    More than just that bird’s eye view, it\u2019s also very tactical for security operations center (SOC) analysts. Every security insight generated by Sumo Logic\u2019s Cloud SIEM<\/a> includes a Global Confidence Score baselines of how alerts were previously triaged (true or false positive) by the company’s SOC analysts and other customers’ SOC teams. The score is then presented on a scale of 0-100, where the higher the value, the more actionable the alert is. \u201cNo score\u201d would mean there is not enough data yet to make any prediction. Learn best practices for making the most of SIEM in our ultimate guide<\/a>.<\/p>\n\n\n\n

    We are also in the early stages of an ML-driven rule recommendation feature that will help security engineers determine what severity a security signal should have. As for ChatGPT\u2019s AI, we\u2019re still exploring ways, with trepidation, to incorporate it into our security solutions. <\/p>\n\n\n\n

    We see some of our customers leveraging it successfully for tasks in both analyst workflows and engineering\/development, driving that shift-left and DevSecOps<\/a> approach. I asked M. Scott Ford<\/a>, Co-Founder & Chief Code Whisperer at CorgiBytes, how he sees things unfold with ChatGPT and AI in modern app development. <\/p>\n\n\n\n

    \n

    Large language models, such as ChatGPT, have a huge potential to be another tool in an application developer’s toolbox. For example, members of the Legacy Code Rocks community have been experimenting with using ChatGPT to suggest refactoring improvements. I see the potential for a similar approach being used to detect common programming mistakes that can lead to security vulnerabilities. This can help short-circuit the cat-and-mouse game by helping sanitize the security surface as it’s being developed.<\/p>\n<\/blockquote>\n\n\n\n

    Lowest hanging fruit is still ripe for the picking<\/h2>\n\n\n\n

    How much of this technology works its way into our security stack is to be determined, but taking off our propeller head caps for a moment, it\u2019s important to realize that most attackers aren\u2019t \u201c1337\u201d, or at least don\u2019t need to be. Remember the human is still the weakest link, and the easiest way in may be a carefully crafted phishing email with an attachment. We\u2019ve been painfully trained on how to spot phishing attacks. Speaking of which, I think my mandatory training is overdue. <\/p>\n\n\n\n

    But the craft of social engineering and phishing is advancing. They are most effective when written by native English speakers and lethal when done using a little humint research against a target, making emails sound realistic and relevant. This means these attacks can\u2019t be done at scale. Until now, thanks to ChatGPT! <\/p>\n\n\n\n

    It happens to have perfect grammar, can research what a company does, and write extremely realistic emails. What\u2019s even more shocking, is that it has been used to carry on email correspondence with a victim, and only include a malicious attachment after some rapport has been established. And this phishing, vishing (voice phishing), and smishing (SMS text phishing) can all be done programmatically in a scalable way. <\/p>\n\n\n\n

    Do we need an AI countermeasure to evaluate each message and provide a \u201chuman vs AI\u201d rating and focus attention on the action it’s trying to lure us into?<\/p>\n\n\n\n

    Again, not sure who is the cat and who is the mouse anymore, but the double edge sword of AI has led to AI detecting AI. GPTZero for example, claims to be the world’s #1 AI detector with over one million users. The app, ironically, uses ChatGPT against itself, checking the text to determine how much \u201cAI involvement\u201d there has been by ChatGPT-like services. Perhaps this will be baked into future email protection solutions. <\/p>\n\n\n\n

    Other vendors already claim AI\/ML email threat detection, but then we’re back to the dsplit technique above. Can an attacker simply intentionally misspell<\/a> a few words and get some grammar incorrect to convince GPTZero the authorship was actually human? Can attackers ask ChatGPT to incorporate common spelling and grammar mistakes into natural language processing (NLP) output?<\/p>\n\n\n\n

    Although some guardrails are being created to prevent ChatGPT from writing phishing emails or malware, there have been workarounds. One observed use was Telegram bots that leveraged unbounded use of ChatGPT available on dark websites. You could get 20 free sample query options and after that, for every 100 queries, people are being charged $5.50. In some cases, scammers can now craft emails that are so convincing they can extort cash from victims without even relying on malware<\/a>.<\/p>\n\n\n\n

    Again, the net of all this is, our detection stack has to be more intelligent and \u201cassume breach\u201d. Track all activity across all entities and identify behavior that is unusual or potentially malicious. It\u2019s never been true that attackers only have to be right once, and defenders have to be right all the time. That only holds true for the initial access. After that, it\u2019s flipped upside down where attackers have to be stealthy all the time as one misstep will trip a wire in a good layered defense strategy. <\/p>\n\n\n\n

    Platforms like Sumo Logic include traditional detection rules like brute force attempts, reconnaissance detections etc, but also track each individual user and system dynamically creating Signals when \u201cfirst seen\u201d behavior<\/a> is observed. Has this user logged in from this location before? Has this machine been remotely accessed in this way previously? Has this user accessed these AWS Secrets before? <\/p>\n\n\n\n

    The kill-chain requires attackers to eventually create indicators of compromise, regardless of the initial infection vector. Thus, it\u2019s not more about shifting up, than it is about shifting left.<\/p>\n\n\n\n

    Shall we play a game?<\/h2>\n\n\n\n

    There is no doubt research is being done at all levels on how to leverage AI in securing our national infrastructure and leveling up our cyber defenders. Truly, there are many areas that ChatGPT can be woven into the defenders workflow. And once a security program is built on top of a platform with intelligent automation, we enter the art of the possible. <\/p>\n\n\n\n

    Eventually, we will be able to use ChatGPT to feed in a lot of detections to make some intelligent determinations\u2026 but we are not there yet. And the best approach is to take a use-case-driven approach to solve pain points individually and programmatically.<\/p>\n\n\n\n