Beyond ChatGPT: The rise of agentic AI and its implications for security

The emergence of generative artificial intelligence (genAI) large language models (LLMs) — such as ChatGPT — has created an earthquake of change that has rippled through every industry and every business. We have all felt the shocks. But these shocks have introduced new capabilities, efficiencies and possibilities. They have also shaken the existing structures, processes, governance and operational activities to the core.  

Most of us have been researching, using and incorporating genAI in some fashion in our organizations. But like all technology advancements, things move fast. Tools like ChatGPT or Microsoft Copilot are now almost commonplace. Many organizations continue to incorporate genAI into their applications while adding features and technologies that make the solutions more accurate and capable, such as model grounding or implementing retrieval augmented generation (RAG) patterns. Just as we are gaining an understanding of these capabilities, there is a growing trend that is obtaining more prominence: namely, agentic AI.  

Agentic AI is a new trend of using AI in an iterative workflow approach that contains agents that act autonomously to achieve specific goals. They can make decisions and act without the need for human intervention or a human in the loop. They are always online, listening, reacting and analyzing domain-specific data in real time, making decisions and acting on them.   

Gartner predicts that by 2028, one-third of human interactions with genAI will evolve from user-prompting LLMs to use interfacing directly with autonomous, intent-driven agents. This is a major jump ahead of the reactive AI assistants many users are now familiar with.  

While this mix of autonomy and automation provides more advantages for certain applications, especially those used in dynamic and complex environments, the need for security is even more crucial now.  

Like any new tool introduced into a system, it can also introduce new vulnerabilities. If, for instance, an agentic AI system is compromised, the decisions it makes autonomously could range from troublesome to disastrous and could include downstream effects.  

As with any new tool or technology, it’s important to examine the security considerations and areas that require additional security focus. You’ll also want to investigate tools and patterns that can be utilized to make the system more secure.  

Though you may not yet be aware of agentic AI, your development teams are likely already working with it. The time to understand agentic AI and implement security measures is now. 

A brief overview of agentic AI 

Let’s dive into agentic AI a bit deeper to understand and set the baseline of the security measures that will be needed.  

Agents, and agentic AI, are not the same as existing genAI services such as copilots or chatbots. Traditional genAI services accept a specific command or prompt and return a response. Agents work as part of a workflow process that acts based on the results returned from the agent.  

Agentic AI brings together a set of tools, frameworks and patterns to automate end-to-end business process workflows that enable AI and humans to work together. Utilizing agents, solutions can be built to deliver end-to-end results that can be pieced together to autonomously achieve the business objective. 

AI agents are the foundational units of work in this architecture that drive each automation task. Each agent is designed to perform a specific, unique, autonomous task and seamlessly integrate back into the broader workflow. There it will deliver information from LLMs, internal business systems, data sources or other AI services as well as data from external systems. These agents can interact with LLMs, allowing the creation of a variety of content. This includes generated code that can be executed by other agents further downstream in the workflow process.  

The workflow process that directs these agents is the controller and it calls the agents required, based on the unique sequence of activities established from the rules and decisions made with the data returned by each agent. Within this process, the workflows can select the right agents to interact with appropriate APIs, determine the right sequence and execute the processes to fulfill the business requirements. 

As the organization builds out solutions and creates agents, it draws upon the strength of agentic AI, which is the ability to integrate external agents that were not originally built on the platform or within the team. This allows teams to collaborate but also allows companies to innovate and incorporate new technologies and capabilities without disrupting the existing solution.  

Because of the autonomy and dynamic nature of the workflows and agents, there may or may not be human interaction in the end-to-end process (aka a human in the loop). However, there must always be the ability to control the process and operations (aka a human on the loop). We must be able to monitor the process, log what each agent is doing, log the data that each agent receives and returns from the workflow, and have the ability to shut the process down or override the operation.  

You may be reading this and thinking that agentic AI is very similar to microservices. You are partially correct. Agentic AI is different in that you are placing the LLM in the control flow and letting it dynamically decide which actions to take. This makes agentic AI leaps and bounds more powerful and dynamic than microservices. So, if you have already put in place security and governance measures for your microservices solutions, you can start there and expand. However, what has been created so far is not sufficient to cover what is required for AI…and especially agentic AI.  

Deciding how agentic AI will be integrated into your organization will require you to think about the complexities and touchpoints that must be covered. You will need to consider all the activities that need to be governed and monitored so that teams don’t create a black-box solution. You will need to ensure that there is no automation happening without direct human oversight and control. In addition, tools today are available that provide the ability for agents to be created and managed by non-developers, using low-code or no-code frameworks. This further drives our need to implement governance, security guidelines and expansive testing to mitigate risks. 

The risks of agentic AI  

There are several risks associated with agentic AI. By understanding the risks, we can begin to put a strategy together for mitigation.  

The major risks that we will focus on are: 

Unexpected behavior or problematic behavior 

AI systems are non-deterministic and behave unpredictably or even, at times, counterintuitively. Agentic AI autonomy increases this risk. Agents may carry out tasks in ways that weren’t anticipated. The decision-making and activity trail needs to be logged and transparent, otherwise it will be difficult for humans to understand and harder to control or reverse the behavior. 

Ethical concerns and dilemmas 

Safety and Ethical concerns must be at the forefront of the risk discussion. The autonomy of agentic AI agents raises the questions of potential misuse as well as unintended consequences. These questions need to be addressed to ensure we maintain trust and ensure the responsible use of these agents. 

Bias is also essential to be aware of and protect against. When agentic AI systems make decisions, we need to be able to detect if there are biases that would lead to unfair or discriminatory results. 

Responsible AI standards are essential not only for ethical concerns but also for transparency, explainability, responsibility and visibility. Businesses and users won’t move forward if they can’t trust the solution. 

Lack of human controls 

As discussed earlier, there must always be the ability to have control over the process and operations (human on the loop). People may think that introducing human controls will slow down performance and use that as an argument. We need to ensure that systems are being designed and implemented with the ability to monitor the process, log what agents are doing, what each agent receives and returns from the workflow, and shut the process down or override the operation. 

We need to ensure that we review operational metrics so that the system is performing within the goals and governance standards as well as remaining aligned with organizational goals. 

Security risks 

This is listed last because it is so important but also a catch-all to the risks that arise. If an agentic AI system gets hacked, serious consequences can arise. First, we need to consider the time it takes to realize a hack has occurred and identify that it is a hack. Even a small unintended change or manipulation can have large consequences. Second, because of the autonomous nature of the system, new security vulnerabilities are introduced, which need to be addressed. Lastly, the largest security risk is putting a system into production without monitoring, logging and controls. These are not bolt-on-afterward activities. 

We need to address specific challenges such as testing for hallucinations, prompt injection attacks and unfiltered user-provided text directly into prompts.  

These risks are important to understand and mitigate. It is important to be leading the activities around controls and governance. According to PR Newswire, 58% of organizations are concerned about the lack of visibility into the unsanctioned use of genAI. Organizations need to lay the foundation regarding the activities and usage, but even more so, they need to be vocal about the rules of use and processes to utilize genAI and agentic AI in production. Without a process in place, and an understanding of where genAI and agentic AI are used, leaders will be wary of the liability. ISMG’s Generative AI Study showed that 55% of leaders lack an understanding of how AI is and will be regulated and are seeking guidance.   

It is incumbent upon us to mitigate the risks, help leadership understand how the organization is using AI, what processes and controls are in place and how we are working with the development teams to ensure that they are implemented according to company rules. 

Strategies for securing agentic AI  

Security is everyone’s job, and we will need to take a multilayered approach to achieve the desired results. There will be activities that are manual tasks, based on a consistent checklist and other activities that we can automate. The goal will be to establish a balance. Especially knowing that we will need some manual activities to dive deeper into findings from the automated tasks. The balance will shift over time as more activities can be automated.  

The multilayered approach needs to include the traditional cybersecurity measures which have been implemented today as well as additional measures and protections for AI. This will include extending policy-based access controls, logging, monitoring, real-time alerts, and detection mechanisms for suspicious or malicious activities compared to a baseline and overall safety measures across the board. All of this needs to be carefully managed to address the challenges and risks. By adopting these additional measures now, before solutions deploy to production, you will be ready to harness the power of agentic AI, enhance your security posture and be able to protect against evolving threats.  

The following should be added to your existing measures:  

First, start with a gradual implementation. Identify the governance, security controls and requirements and put those into place in down-level environments. Work with the development teams to understand where those measures may miss the mark and need to be augmented. Depending on your environment, if the development team is already moving forward, you may have to slipstream measures into place and evolve the implementation over time.  

Second, identify where existing and traditional cybersecurity measures need to be modified. There will be specialized protections for AI and more so for agentic AI.   

Third, put in place, and require end-to-end monitoring. Ensure that teams are: 

• Logging and monitoring communication (inputs and outputs) from the LLM as well as all communication to and from each agent.   
• Include things such as a correlation ID, to be able to track a process through the complete lifecycle instance. Each implementation will also have custom output which needs to be understood by the security teams.  
• Identifying prompt injections, data leakage or unexpected behaviors. 
• Restricting, or placing strict validations on, user-provided prompts or user-provided text in prompts. 

As issues arise, you will need to be able to review decisions made by the agents and the associated workflow and review input and output to ensure that the system is transparent. You will then have the information required if, and when, an audit needs to take place.  

Fourth, look at LLM-specific threats. Put in place procedures to isolate agents from critical systems, limit the access the agent has to resources and evaluate and validate prompts before they get submitted to the LLM. Use tools such as OpenAI’s Moderation API to evaluate prompts and responses to ensure content filtering.   

In addition, as you review prompts and responses, utilize either the Evals Framework from OpenAI or the PromptFlow SDK from Microsoft. These tools provide the ability to evaluate the response with a set of ideal answers to compare against the response. You can either compare against specific benchmarks you create or when evaluating open-ended questions, you can have the model grade itself and provide statistics.  

Also, review the top 10 list of threats published by OWASP. This will help you identify other threats than what we have covered here, and help you plan for mitigating those threats.   

Fifth, incorporate automation frameworks that help testers, red team test groups and security teams to proactively uncover risks. You should be performing a red team test and testing for responsible AI simultaneously. You will need to have a group dedicated to red team activities and use a tool such as PyRIT. This tool helps to proactively uncover risks. 

Red teaming an agentic AI system is different from traditional systems. Agentic AI and traditional AI systems are non-deterministic, and scripts will need to be run multiple times. Each time the scripts are run the output will differ. You need to take this variability into account as you test each scenario. You also have to keep in mind that due to the agentic workflow logic, the LLM itself, the variability in prompts and the agent behavior, will result in more variability. You will also experience that executing the same task against the same scenario will respond differently, and you will need to run more tests and test scenarios to cover any potential blind spots. Have your development teams create a map of all rules and flow possibilities through the process.  

As with any tool, you won’t be able to, and shouldn’t always, automate everything. Use a tool such as PyRIT along with manual testing. Manual testing will allow testers to test specific trouble areas as well as perform deeper dives into any areas the automation testing uncovered.  

Make sure that you are also providing monitoring and logging of your automation tests. This will help test the process of tracing issues but also help as the team dives in deeper with their manual tests. Test the process of using the logged data to ensure transparency and auditability at this stage, instead of when an issue presents itself in production.  

Lastly, work with other cybersecurity experts to compare and contrast measures and practices. Continue to build out your governance framework and always add and refine your procedures. 

The future of agentic AI: Promising…and full of possibilities 

The wide range of benefits, capabilities and efficiencies that can be offered to the business make this the perfect time to explore this technology. However, the associated risks and security threats cannot be ignored. We must make sure that we are broadening the corporate culture so that security is everyone’s responsibility. It is incumbent upon teams to log all interactions, monitor the system and ensure that there are human controls in place. Tools must be incorporated into the end-to-end processes, to proactively find issues before they erode user and business confidence. Transparency, human oversight and AI safety must always be top of mind.  

Security teams need to outline controls and governance, security measures and rules. Development teams need to educate themselves, not only on these rules and requirements but also on the risks they will encounter and the mitigations they need to put in place. 

Stephen Kaufman serves as a chief architect in the Microsoft Customer Success Unit Office of the CTO focusing on AI and cloud computing. He brings more than 30 years of experience across some of the largest enterprise customers, helping them understand and utilize AI ranging from initial concepts to specific application architectures, design, development and delivery.   

This article was made possible by our partnership with the IASA Chief Architect Forum. The CAF’s purpose is to test, challenge and support the art and science of Business Technology Architecture and its evolution over time as well as grow the influence and leadership of chief architects both inside and outside the profession. The CAF is a leadership community of the IASA, the leading non-profit professional association for business technology architects.