Detecting unauthorized tool calls using Ollama and Splunk

Abstract

Organizations increasingly deploy self-hosted LLMs using frameworks like Ollama to get better productivity with tools that can execute actions or access sensitive resources. Such deployments may be internal or external and often ship without built-in security controls, making them vulnerable to risks such as prompt injection that can manipulate an LLM into invoking connected tools or accessing sensitive resources beyond its intended permissions. One can add guardrails to such models or leverage existing prompt monitoring solutions to gain visibility, but integrating LLM telemetry into existing security operations workflows remains challenging. During a security incident, timely detection and response are critical, which is where PromptMon helps. PromptMon is a lightweight Python middleware library that can be integrated alongside Ollama-hosted LLMs to add observability, prompt monitoring, and security telemetry without modifying the underlying model. PromptMon allows developers to integrate their own machine learning classifiers to evaluate prompts and assign confidence scores. The normalized log data is sent to a SIEM, opening a wide range of possibilities for response actions using SOAR or low code/no code platforms. Furthermore, it can be used for an audit trail during security incidents - add runtime prompt monitoring and SOC-ready observability in a few lines of code. This poster demonstrates the integration of PromptMon with an Ollama-based LLM application, illustrating its architecture, telemetry workflow, and Splunk integration, while outlining future work toward detection-and-response methodologies for agentic systems.