SkillEvaluator
Open-source multi-tier framework from NVIDIA for checking AI agent skills for quality and security and measuring their impact on agent task results.
At a Glance
SkillEvaluator core CLI/framework is licensed under Apache License 2.0 and free to use, modify and distribute. External model/provider and sandbox usage may incur separate costs.
Engagement
Available On
Alternatives
Listed Nov 2026
About SkillEvaluator
SkillEvaluator is an open-source framework from NVIDIA that checks AI agent skills and plugins for quality and security, then measures whether a skill helps an agent complete real tasks. It ships as a Python CLI and is part of the NVIDIA Verified Skills pipeline. The documentation labels it experimental, with community best-effort support and no SLA.
What It Is
SkillEvaluator evaluates agent skills, which are folders of instructions and supporting files defined by the Agent Skills specification. It combines deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation. Each of its three tiers can run independently, so a user can start with a keyless static check and add deeper evaluation later.
The Three Tiers
- Tier 1, Validation and Harness Check: Tier 1 checks whether a skill is well-formed and safe, and integrates SkillSpector for security scanning. The harness check reports whether agents such as Codex, Claude Code, OpenCode, Hermes, OpenClaw and Copilot CLI will load the skill. It needs no model key or agent run.
- Tier 2, Deduplication: Uses embeddings and chat models to detect repeated or overlapping guidance, optionally against a skill catalog.
- Tier 3, Live Agent Evaluation: Runs the same tasks with and without the skill in a sandbox (Docker by default, powered by Harbor) and produces a report comparing results.
Ways to Run It
The quality-check command prints a score out of 100 with a grade and findings, with no API key, GPU or Docker required. The validate command runs the combined pipeline. A CI template for GitHub or GitLab produces quality and security reports when skills change, and the SkillEvaluator skill can be installed into an agent so it can run evaluations on request. Supported providers include NVIDIA Build endpoints for prototyping.
Setup Path
Install with pip or uv on Python 3.12 to 3.14. Full static coverage needs external scanners, and Tier 3 needs provider access, a supported agent and a running sandbox. Model and managed-sandbox usage can incur charges. The project is Apache-2.0 licensed and has an accompanying paper and NVIDIA Developer blog post.
Community Discussions
Be the first to start a conversation about SkillEvaluator
Share your experience with SkillEvaluator, ask questions, or help others learn from your insights.
Pricing
Open Source (Apache-2.0)
SkillEvaluator core CLI/framework is licensed under Apache License 2.0 and free to use, modify and distribute. External model/provider and sandbox usage may incur separate costs.
- Apache License 2.0
- Tier 1: Validation and Harness Check
- Tier 2: Deduplication
- Tier 3: Live Agent Evaluation
- Keyless quality-check needs no API key, GPU or Docker
Capabilities
Key Features
- Keyless static quality check with score out of 100
- Tier 1 validation with SkillSpector security scanning
- Harness compatibility check across agents
- Tier 2 semantic deduplication and overlap detection
- Synthetic eval dataset generation
- Tier 3 live agent evaluation with and without the skill
- GitHub and GitLab CI templates
- Agent skill for evaluating from within an agent
- Reports, performance dashboard and metrics
- Custom graders and tasks
Integrations
Demo Video

