NVIDIA SkillSpector: “Safe” Verdicts Don’t Mean The Skill Is Safe

NVIDIA has created a very useful scanner for AI agent/chatbot Skills: SkillSpector. This tool walks a directory containing the Skills files, and performs a lightweight static analysis combined with YARA rules and a lightweight LLM of its own to categorize a Skill’s content as “Safe”, “Caution”, or “Do Not Install”. But a “Safe” verdict, despite the name, doesn’t really mean the skill you just scanned is actually safe. Our research into SkillSpector’s approach, led by Alon Lerner, uncovered three ways malicious Skills can get a “Safe” verdict, at least in SkillSpector versions through 2.2.3. We reported these to NVIDIA; however, since they don’t consider these to be evidence of vulnerability in SkillSpector, it’s outside their threat model – which means it’s inside yours.

A quick SkillSpector overview

Skills are a quasi-standard way to give new capabilities to agents (for simplicity, we include chatbots, code assistants, and other similar AI systems when we use the word “agent”, even if they aren’t truly agentic). They’re a directory of files; the `SKILL.md` file contains some metadata describing the skill and a comprehensive prompt, and is the core of the Skill. Skills can also contain other things:

• Additional documents, usually also in markdown (.md) format; these aren’t read until the prompts in the SKILL.md file direct the agent to do so.

• Scripts, usually in bash (which agents often execute directly in a shell), Python, or JavaScript. Different agents have different strategies for sandboxing these scripts, with varying degrees of capability and safety
• Templates that the Skill can fill in rather than having to generate outputs from scratch. For example, a Skill that produces a report as a Word file might have a Word template that defines the styles and layout.
• Assets that the Skill can include as-is in its output. For example, a Skill that produces a branded image for social media might have a logo asset that the Skill can add into the generated image

SkillSpector focuses on the places of highest risk: the prompts in SKILL.md (because they are input into the LLM) and the bundled scripts (because they will execute where the agent is running). It scans these, at a high level, by:

1. Finding relevant files in the Skill directory; only the files marked as relevant are fed to the next steps
2. Running a suite of tests for known risky patterns, using a mix of regular expressions (regex), simple static analysis (looking for things like running local commands), data flow analysis, and a YARA run to identify malicious behavior
3. Feeding the relevant files to an LLM. There’s a discovery phase and a meta-analyzer phase that confirms findings

Each finding identified is assigned risk points, and if the total points exceed certain thresholds, SkillSpector will report a “CAUTION” or “DO_NOT_INSTALL” flag, which you can implement so that it blocks Skill installation or startup.

Problem 1: Non-determinism

The first challenge for SkillSpector verdicts is that the LLM outputs aren’t deterministic: you can scan a skill twice, with no changes, and get different verdicts. Alon’s testing was able to surface cases where a byte-for-byte identical Skill could raise CAUTION flags in some scans, while being marked SAFE in others.

This doesn’t mean SkillSpector is useless, but it does mean it has a degree of unreliability. This is a risk whether or not a Skill author is trying to evade SkillSpector, since it can result in both false positives (a safe Skill being blocked) and false negatives (a malicious Skill being approved). In CI/CD environments, build gates that key off of SkillSpector’s outputs could correctly block a Skill during one build, but permit it in the next.

Problem 2: You can hide malicious things in stuff SkillSpector ignores

Remember how SkillSpector’s first step is to build a list of relevant files? This list explicitly excludes dotfiles (like .envrc) and vendored dependency directories (like node_modules and venv Python environments). This is a sensible architectural design on the surface: SkillSpector intends to scan what Skill authors write, not third-party code found in those sources. This saves time on the static scan, and time and tokens during the LLM stage.

Unfortunately, that makes these excluded paths ideal places to hide malware. Alon was able to slip malicious code into node_modules/devtools-core/index.js, essentially pretending the malicious code was a legitimate dependency. SkillSpector gave a SAFE verdict, with a score of only 6 (while CAUTION requires at least a score of 50 out of a total possible 100) – because SkillSpector never even examined the malicious code.

At runtime though, the AI agent loads that code into its Node environment and happily executes it, allowing an attacker to execute arbitrary commands wherever the agent runs.

This risk exists because of a mismatch between what the real-world agent executes and what SkillSpector is willing to analyze, which means attackers can bypass both the static and LLM parts of SkillSpector

Problem 3: You can obfuscate some sketchy stuff

SkillSpector’s deterministic half identifies dangerous functions by name; so if you can hide the real names, you can hide your code from this half of the scanner. Hiding names is pretty easy: a call to os.system.cmd() would get flagged, but a call to getattr(os, "system")(cmd) does the same thing and looks completely fine to the scanner (or at least it did; as of version v2.5.0 this seems to have been fixed since then). Alon was able to reliably hide exactly that type of call from SkillSpector, once again allowing arbitrary execution of commands.

Fortunately, the LLM half isn’t so easily tricked. So, unlike Problem 2 above, it’s not a complete bypass of SkillSpector. However, it is a very good reason to make sure that you’re not tempted to run SkillSpector in “static only” mode. It might save time, but it is also very easy to trick.

Disclosure and NVIDIA’s response

Checkmarx Zero follows coordinated disclosure practices, so we of course reported these issues to NVIDIA as soon as we were confident we had a clear and repeatable set of issues. NVIDIA responded promptly to our reports, and ultimately determined that these issues are not within the scope of their threat model, and so aren’t considered vulnerabilities from their point of view.

Which ultimately means these problems are up to you, as an adopter of SkillSpector, to understand and address.

Though the NVIDIA security team didn’t accept the report, it seems someone on their development team was thinking along similar lines.

Lesson: no single control keeps you safe

SkillSpector is a valuable tool in a defender’s arsenal; used with care, it can reduce an organization’s exposure to malicious and dangerous AI Skills. But it’s definitely not perfect, and it’s up to security teams to keep it in perspective.

The limited scope of the scans leaves the door open for a clever adversary to hide dangerous code in a Skill that SkillSpector will mark SAFE. The Hybrid deterministic-plus-LLM approach has a lot of value (we do the same thing in our updated SAST tool, because it really is more accurate than either approach alone). But the single-LLM approach creates opportunities for error and inconsistency. As we learned in developing our Fusion scanner, multiple complementary models and controls make for a much better system.

So certainly use SkillSpector if it makes sense for your organization; but don’t rely on it alone, especially in critical or sensitive workflows. For example:

• Use SkillSpector as a “Fail Fast” control: if it says it’s “Do Not Install”, it’s probably right. And you can get there fast.
• For “Safe” verdicts, use additional static analysis tools that can examine the parts SkillSpector ignores or misses before finally approving.
• As with all AI-related risks, make sure your endpoint controls are well-configured to reduce the impact and “splash damage” should a malicious Skill sneak by the above controls

As always, security is ultimately about thoughtful, interlocking layers of defense.