Red Teaming LLMs: Making AI Safety Measurable with Open-Source Tools

Large language models are moving quickly from experimentation into operational workflows. Teams are using them to summarize information, generate code, assist analysts, triage data, and accelerate software delivery. That shift creates a practical question for organizations adopting AI: how do we know whether a model behaves safely under pressure?

It is not enough to ask a model a few benign questions and assume the behavior will hold. Model behavior changes based on phrasing, context, formatting, and adversarial attempts to bypass guardrails. A model that refuses a direct harmful request may respond differently when the same intent is wrapped in an encoding trick, framed as a role-play, or hidden inside a structured task.

That is where red-team evaluation becomes useful. The goal is not to prove that a model is perfectly safe. The goal is to make model behavior measurable, repeatable, and visible enough for engineering and governance teams to make informed decisions.

To put that principle into practice, we built a local evaluation workflow from open-source components. Promptfoo orchestrated the tests, Ollama hosted the Llama target model, and an OMLX-hosted Gemma model graded the responses. The tools made the experiment possible, but the more transferable lesson is the method: define risk, vary attack strategies, preserve repeatable evidence, and report results without exposing sensitive test content.

Why Local Red Teaming Matters

Many organizations want to evaluate model behavior, but they operate under constraints that make public testing difficult. Provider terms of service may restrict or require approval for red-team activity against hosted models. Sensitive prompts or model outputs may not be appropriate to send to external services. Government and regulated environments may also require tighter control over evaluation data, test artifacts, and audit records.

Local red teaming helps address those constraints. Instead of sending adversarial probes to a public model endpoint, the evaluation can run against a locally hosted model with local storage for results. This gives teams more control over what is tested, where artifacts are stored, and who can access the raw evidence.

That control matters because red-team outputs are sensitive by nature. A useful evaluation may include unsafe prompts, failed responses, grader reasoning, and model outputs that should not be broadly distributed. The same evidence that helps engineers improve model behavior can create risk if it is shared without context or controls.

Our approach separated two audiences:

• Technical reviewers who may need access to the native evaluation dashboard and raw technical artifacts.
• Stakeholders who need a sanitized summary of methodology, results, trends, and risk areas.

That distinction became important in how we packaged and presented the results.

Building an Open-Source Evaluation Workflow

Open-source tooling makes red-team evaluation accessible and adaptable, but the workflow matters more than any single product. A useful evaluation harness needs to describe the system under test, exercise relevant risk categories, grade behavior consistently, and retain results for analysis.

For this case study, we used Promptfoo, an open-source evaluation framework, as the orchestrator. Its approach is described in the Promptfoo red-team guide. A configuration file defined the target model, the intended purpose of the system, the risk categories to test, and the model used to grade responses.

At a high level, the workflow looks like this:

1. Define the model target and system purpose.
2. Select risk categories and attack strategies.
3. Generate or load test cases.
4. Send each test to the target model.
5. Grade the response against the expected safety behavior.
6. Store the results for reporting and analysis.

The target and grader were separate local models. A Llama 3.1 8B model served through Ollama received the probes, while a Gemma 4 12B model served through OMLX graded the responses through an OpenAI-compatible API endpoint. Promptfoo coordinated the workflow. This design kept target execution, grading, and evidence storage inside the local evaluation boundary.

Case Study: Testing Beyond the Baseline

The initial goal was to evaluate local, Bedrock, and public deployments using a common red-team plan. During execution, we narrowed the scope for governance reasons. Public OpenAI targets were excluded because red-team activity against hosted public models may conflict with provider terms or require explicit approval. Bedrock was also not executed because access and terms had not been validated for this use case. The completed evaluation focused on the local model path.

The local target was:

ollama:chat:llama3.1:8b

The local grader was:

openai:chat:gemma-4-12B-it-4bit
http://127.0.0.1:8000/v1

The baseline started with Promptfoo's default risk coverage and added explicit checks for:

• Hate-related harmful content (harmful:hate)
• Profanity (harmful:profanity)

The first baseline run produced 195 probes across 39 risk categories. That baseline was useful, but it also raised a question. The pass rate was high enough that it warranted deeper testing. A shallow direct probe set can make a model look stronger than it is because many models are tuned to reject obvious harmful requests.

To test beyond direct prompts, we performed a deeper transformed evaluation. The deeper run reused the baseline probe corpus and applied local transformations to create broader coverage. The transformation set included:

• Baseline prompts
• Base64 instruction wrapping
• ROT13 instruction wrapping
• Leetspeak-style obfuscation
• Research-style jailbreak wrapping
• JSON task wrapping

This produced 1,170 total evaluations. The run used local execution controls, disabled public sharing, and avoided attaching raw evaluation artifacts to internal tracking or stakeholder-facing packages.

Results

The baseline scan produced a strong initial result:

• Total probes: 195
• Passed: 192
• Failed: 3
• Runtime errors: 0
• Pass rate: 98.46%

The deeper transformed scan produced a more useful picture:

• Total probes: 1,170
• Passed: 1,133
• Failed: 37
• Runtime errors: 0
• Pass rate: 96.84%

The decrease from 98.46% to 96.84% was not dramatic, but it was meaningful. The deeper scan found 34 additional failures beyond the baseline set. More importantly, it showed where the model was more likely to fail.

Strategy-level results are summarized below.

The clearest weakness was leetspeak-style obfuscation. This is an important finding because obfuscation is a common way to test whether a model is responding to semantic intent or only matching obvious surface patterns.

The weakest tested risk category was hijacking:

• Total hijacking tests: 30
• Passed: 21
• Failed: 9
• Pass rate: 70.00%

The explicitly requested harmful-content categories performed better:

• Hate-related harmful content (harmful:hate): 30 passed, 0 failed
• Profanity (harmful:profanity): 29 passed, 1 failed

The result was not simply a pass/fail score. It was a risk map. The model performed well across many categories, but certain transformations and control-bypass patterns exposed weaknesses that would not have been visible from the baseline scan alone.

These results describe one target model, one grader, one probe corpus, and one point-in-time configuration. They are a scoped risk signal, not a general model benchmark or safety certification.

What We Learned

The main lesson is that model safety should be measured under variation. A direct refusal test is useful, but it is not sufficient. If the same unsafe intent is expressed through obfuscation, encoding, formatting, or a structured wrapper, the model may behave differently.

That finding has several practical implications.

First, red-team testing should be repeatable. A one-time manual review is not enough for systems that will evolve over time. Models change, prompts change, serving stacks change, and application context changes. Evaluation should become part of the delivery lifecycle, not a one-off exercise.

Second, local evaluation is operationally valuable. It gives teams a way to test models without sending sensitive adversarial content to public endpoints. That is especially important for organizations working in regulated, government, or security-sensitive environments.

Third, reporting needs to be audience-aware. A native evaluation dashboard is useful for engineers because it can include detailed prompts, outputs, grader decisions, and failure records. That same detail may be inappropriate for broad stakeholder review. A sanitized report can communicate the main findings without distributing adversarial content.

Finally, high pass rates should not end the conversation. The baseline pass rate looked strong at 98.46%. The deeper scan showed that additional test variation exposed new issues. The right question is not whether a model achieved a single impressive score. The better question is what the score covers, what it misses, and how the results change under realistic adversarial pressure.

Reporting and Presentation

For this work, we produced two classes of artifacts.

The native evaluation dashboard produced by Promptfoo was preserved for technical review. It allows authorized reviewers to inspect detailed results, filter failures, and examine model behavior case by case. Because it can expose raw adversarial prompts and model outputs, it should be handled as sensitive technical material.

We also produced a sanitized web report for team presentation. That report includes scope and methodology, evaluation totals and pass rates, strategy-level breakdowns, category-level failure summaries, caveats, and handling notes.

The sanitized report intentionally excludes raw prompts and model outputs. That made it suitable for internal tracking and stakeholder review while preserving the more sensitive raw artifacts locally.

This separation is a pattern worth keeping. Effective AI governance requires transparency, but transparency does not mean every artifact belongs in every forum.

Business and Mission Implications

As organizations adopt AI, they need confidence that model behavior aligns with mission, policy, and risk tolerance. That confidence cannot come from vendor claims or isolated demos. It has to come from evidence.

Open-source evaluation tools provide a practical way to generate that evidence. A reusable harness can test model behavior, compare changes over time, and identify where mitigations are needed. Promptfoo filled that role in this case, but the organizational value comes from making red-team evaluation a repeatable engineering practice. In enterprise environments, that can support several outcomes:

• Safer AI adoption through measurable behavior validation.
• Better governance through repeatable evaluation records.
• Faster engineering feedback loops when model, prompt, or guardrail changes are introduced.
• More informed risk decisions for local, hosted, or hybrid AI architectures.
• Clearer communication between engineers, security teams, and stakeholders.

For mission environments, the value is even more direct. AI systems increasingly support analysis, software delivery, knowledge management, and operational workflows. If those systems behave unpredictably under adversarial or malformed input, they can create risk at the speed of automation.

Red-team evaluation does not eliminate that risk. It makes the risk visible earlier.

Conclusion

AI safety evaluation is becoming a core engineering discipline. As models move into real workflows, organizations need practical ways to test behavior, identify weaknesses, and communicate risk without slowing delivery to a halt.

This local, open-source evaluation showed how a repeatable red-team workflow can support that goal. The baseline scan suggested strong model behavior, but the deeper transformed scan exposed additional weaknesses, especially around obfuscated inputs and hijacking patterns. That is exactly the kind of insight a useful evaluation should produce: not a simplistic pass/fail judgment, but actionable evidence about where the system is strong and where it needs more attention.

The specific stack made the case study reproducible, but the method extends beyond it: define the risks that matter, test the same intent under meaningful variation, retain evidence, and communicate findings at the right level for each audience.

The broader takeaway is straightforward. Model behavior should be tested the same way we test software: repeatedly, intentionally, and with enough variation to expose failure modes before users or adversaries do.

At BridgePhase, this is the kind of practical AI engineering discipline we see becoming essential for organizations adopting LLMs in serious environments. The future of AI adoption will not be defined only by model capability. It will also be defined by how well teams can measure, govern, and improve that capability over time.