DarkMoon and Strix: Evidence and Evaluation

Reviewed: September 9, 2026

Editorial decision: Integrate a bounded architecture comparison and privacy case study.

Relevance score: 3/5 for this guide. This scores educational value, not security effectiveness.

Canonical page: Agentic Pentesting

Review scope

The review inspected DarkMoon’s prompt plugin, privacy gateway, global result sanitizer, deployment configuration, and README; Strix’s README, local-provider documentation, SDK dependency, report validator, deduplication code, and telemetry documentation. A local sanitation function was exercised with synthetic data as recorded below. No pentest, Docker deployment, provider-traffic capture, or head-to-head benchmark was executed for this documentation change. Reading implementation code is not an end-to-end behavior test.

Sources: DarkMoon repository and Strix repository.

The implementation links below identify the reviewed source. Published benchmarks are assessed separately.

Findings supported by the inspected sources

Finding	Source	Evidence boundary
DarkMoon’s prompt protection is an OpenCode plugin	Plugin hooks and fallback	Registration code inspected; runtime coverage not traced
DarkMoon can continue with raw results after a sanitation exception	Global sanitizer	Explicit exception branch returns value; no exception injected into a running campaign
Command restoration and result sanitation have separate policies	Gateway implementation	DEGRADE versus STRICT concerns command handling; neither establishes complete recognition coverage
Strix provides a Claude Code integration route	Agent skills section	Documented integration, not exercised here
Strix supports configuration for local inference	Local-provider documentation	Contradicts the cloud-only row in DarkMoon’s README; does not establish offline behavior
Strix enforces a report structure with evidence and counterevidence	Validation functions	Nonempty text and metadata checks do not execute an exploit
Strix’s deduplication is another model-data route	Judge and provider resolution	Can use a separate endpoint; errors can return a nonduplicate result
Strix documents telemetry and an opt-out	Telemetry policy	Documentation inspected, network behavior not captured
DarkMoon’s default deployment grants substantial host access	Compose configuration	Root, host network, Docker socket and capabilities visible in configuration; effective deployment not measured
Strix’s Docker path also requires a host-access review	Docker capabilities and source mounts	Added network capabilities and writable source mounts inspected; effective deployment and optional backends not tested

The raw-result exception path prevents a universal statement that no sensitive data can reach the model. Successful recognition of IP addresses or registered values would not prove coverage of previously unseen secrets in other formats. The guide describes the mechanism and the verification needed, without treating tokenization as complete anonymization.

Synthetic sanitation check

Executed on September 9, 2026 against the reviewed DarkMoon source. Every fixture gets a fresh vault so a previously registered value cannot hide a recognition gap. The value below is invented test data. From a clone of the DarkMoon repository, install its Python dependencies and run this command at the repository root:

PYTHONPATH=./mcp/src python3 - <<'PY'
import json
from privacy import CommandGateway, PrivacyVault

secret = "DemoSecret987!"
fixtures = {
    "plain": f"password={secret}",
    "json_text": json.dumps({"password": secret}),
    "mapping": {"password": secret},
}
for name, value in fixtures.items():
    result = CommandGateway().sanitize_result(
        value, PrivacyVault(session_id=f"fixture-{name}")
    )
    print(f"{name}: synthetic_value_present={secret in json.dumps(result)}")
PY

Observed output:

plain: synthetic_value_present=False
json_text: synthetic_value_present=True
mapping: synthetic_value_present=True

This demonstrates a format-dependent recognition gap in the tested function. It does not show what an entire OpenCode session sent to a provider, and it does not cover values registered earlier in a session.

Published benchmarks and unresolved claims

DarkMoon’s pinned README benchmark reports 57 Juice Shop vulnerabilities using local inference. This review does not certify that count as 57 independent, reproduced defects. The exact model, target snapshot, edition, and report deduplication need a campaign manifest and export.

Strix’s XBEN publication, consulted September 9, 2026, reports 100/104 solved challenges for v0.4.0 with gemini-3-pro-preview. Its documented success criterion is flag extraction. This is a publisher-reported benchmark, not a reproduction with the newer inspected source revision. A solved-challenge ratio cannot be compared numerically with a finding count from one application.

DarkMoon’s Juice Shop article describes seven findings, four classified as exploited. Its privacy article explicitly describes a single LLM-endpoint run on a sealed build, reporting zero real target-address occurrences across approximately 3.5 MB of model traffic. These publisher claims were checked against saved copies of the linked articles. They describe different campaign evidence and do not verify the README’s 57-finding run or the inspected open-source deployment.

The demo report remains a source for a separate export review. This documentation pass did not reopen or independently reproduce its individual findings; it does not certify its deduplicated count or exploitation labels.

Evidence required for a stronger recommendation

Priority	Requested artifact	Acceptance criterion
P1	Campaign manifest	Maps edition, source commit, image digests, target snapshot, exact model and all provider routes to one run
P1	Privacy coverage and failure tests	Synthetic known and unseen values in text, JSON, errors and encoded output; forced sanitizer failure; expected and actual handling recorded
P1	Outgoing model-request capture	Captures primary and secondary inference requests, with a documented boundary and canary search; vault logs serve as supporting evidence
P2	Machine report export	Reconciles the advertised count with distinct root causes, reproduction evidence, rejected findings and manual decisions
P2	Shared laboratory evaluation	Same target and acceptance rules, recorded budgets and human intervention, repeat runs, false positives and missed defects
P2	Deployment verification	Confirms effective mounts, privileges, network reachability, target enforcement, stop behavior and cleanup

The first three items are necessary before a claim about reduced exposure on a measured remote-model run. The report export is necessary before repeating a validated defect count. A shared evaluation is necessary before ranking detection performance.

Editorial challenge and decision

Reason to include: The pair makes the distinction between model, agent runtime, integration skill, tool execution, and privacy controls concrete. DarkMoon’s separate prompt and output paths make a useful example of why a security claim needs an explicit boundary. Strix’s structured reports demonstrate the difference between requiring evidence fields and validating their contents.

Reason to limit coverage: Neither system has been operated in a common laboratory for this guide. Integrations documented by a maintainer, a Docker image, and a benchmark table do not establish reliable operation on the reader’s application.

Decision: Publish the architecture comparison, with short pointers from privacy and hardening guidance. Keep installation walkthroughs and operational recommendations conditional on the evidence above. Do not label either platform a native Claude Code security feature or declare a performance winner.