EveryDev.ai
Subscribe
Home
Tools

5,901+ AI tools

  • Featured
  • Gems
  • Trending
  • New
  • Rate tools
  • Compare
  • Arena
Categories
  • Agents2972
  • Coding2248
  • Infrastructure1067
  • Projects818
  • MCP794
  • Research744
  • Analytics565
  • Marketing527
  • Design488
  • Testing476
  • Data442
  • Security412
  • Integration288
  • Communication284
  • Prompts283
  • Extensions238
  • Voice213
  • DevOps190
  • Commerce183
  • Web128
  • Learning123
  • Finance44
AI Tools by Topic
  • AI Coding Assistants
  • Agent Frameworks
  • MCP Servers
  • AI Prompt Tools
  • Vibe Coding Tools
  • AI Design Tools
  • AI Database Tools
  • AI Website Builders
  • AI Testing Tools
  • LLM Evaluations
Follow Us
  • X / Twitter
  • LinkedIn
  • Reddit
  • Discord
  • Threads
  • Bluesky
  • Mastodon
  • YouTube
  • GitHub
  • Instagram
Get Started
  • Users
  • Rate Tools
  • About
  • Editorial Standards
  • Corrections & Disclosures
  • Community Guidelines
  • Advertise
  • Contact Us
  • Newsletter
  • Submit a Tool
  • Start a Discussion
  • Write A Blog
  • Share A Build
  • Terms of Service
  • Privacy Policy
Explore with AI
  • ChatGPT
  • Gemini
  • Claude
  • Grok
  • Perplexity
Agent Experience
  • llms.txt
Theme
With AI, Everyone is a Dev. EveryDev.ai ©
    1. Home
    2. Tools
    3. SkillEvaluator
    SkillEvaluator icon

    SkillEvaluator

    Code Security
    Featured

    Open-source multi-tier framework from NVIDIA for checking AI agent skills for quality and security and measuring their impact on agent task results.

    Visit Website

    At a Glance

    Pricing
    Open Source

    SkillEvaluator core CLI/framework is licensed under Apache License 2.0 and free to use, modify and distribute. External model/provider and sandbox usage may incur separate costs.

    Engagement

    Available On

    Linux
    Web
    API
    CLI

    Resources

    WebsiteDocsGitHubllms.txt

    Topics

    Code SecurityLLM EvaluationsAgent Skill Registries

    Alternatives

    agent-skills-evalPurple LlamaCoze Loop
    Developer
    NVIDIASanta ClaraEst. 1993

    Listed Nov 2026

    About SkillEvaluator

    SkillEvaluator is an open-source framework from NVIDIA that checks AI agent skills and plugins for quality and security, then measures whether a skill helps an agent complete real tasks. It ships as a Python CLI and is part of the NVIDIA Verified Skills pipeline. The documentation labels it experimental, with community best-effort support and no SLA.

    What It Is

    SkillEvaluator evaluates agent skills, which are folders of instructions and supporting files defined by the Agent Skills specification. It combines deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation. Each of its three tiers can run independently, so a user can start with a keyless static check and add deeper evaluation later.

    The Three Tiers

    • Tier 1, Validation and Harness Check: Tier 1 checks whether a skill is well-formed and safe, and integrates SkillSpector for security scanning. The harness check reports whether agents such as Codex, Claude Code, OpenCode, Hermes, OpenClaw and Copilot CLI will load the skill. It needs no model key or agent run.
    • Tier 2, Deduplication: Uses embeddings and chat models to detect repeated or overlapping guidance, optionally against a skill catalog.
    • Tier 3, Live Agent Evaluation: Runs the same tasks with and without the skill in a sandbox (Docker by default, powered by Harbor) and produces a report comparing results.

    Ways to Run It

    The quality-check command prints a score out of 100 with a grade and findings, with no API key, GPU or Docker required. The validate command runs the combined pipeline. A CI template for GitHub or GitLab produces quality and security reports when skills change, and the SkillEvaluator skill can be installed into an agent so it can run evaluations on request. Supported providers include NVIDIA Build endpoints for prototyping.

    Setup Path

    Install with pip or uv on Python 3.12 to 3.14. Full static coverage needs external scanners, and Tier 3 needs provider access, a supported agent and a running sandbox. Model and managed-sandbox usage can incur charges. The project is Apache-2.0 licensed and has an accompanying paper and NVIDIA Developer blog post.

    SkillEvaluator - 1

    Community Discussions

    Be the first to start a conversation about SkillEvaluator

    Share your experience with SkillEvaluator, ask questions, or help others learn from your insights.

    Pricing

    OPEN SOURCE

    Open Source (Apache-2.0)

    SkillEvaluator core CLI/framework is licensed under Apache License 2.0 and free to use, modify and distribute. External model/provider and sandbox usage may incur separate costs.

    • Apache License 2.0
    • Tier 1: Validation and Harness Check
    • Tier 2: Deduplication
    • Tier 3: Live Agent Evaluation
    • Keyless quality-check needs no API key, GPU or Docker

    Capabilities

    Key Features

    • Keyless static quality check with score out of 100
    • Tier 1 validation with SkillSpector security scanning
    • Harness compatibility check across agents
    • Tier 2 semantic deduplication and overlap detection
    • Synthetic eval dataset generation
    • Tier 3 live agent evaluation with and without the skill
    • GitHub and GitLab CI templates
    • Agent skill for evaluating from within an agent
    • Reports, performance dashboard and metrics
    • Custom graders and tasks

    Integrations

    SkillSpector
    Harbor
    Docker
    GitHub Actions
    GitLab CI/CD
    NVIDIA Build
    Claude Code
    Codex
    OpenCode
    OpenClaw
    Hermes
    Copilot CLI
    API Available
    View Docs

    Demo Video

    SkillEvaluator Demo Video
    Watch on YouTube

    Ratings & Reviews

    No ratings yet

    Be the first to rate SkillEvaluator and help others make informed decisions.

    Rate other tools you’ve used

    Developer

    NVIDIA

    NVIDIA builds the computing platform powering modern AI, from data center GPUs and networking to developer SDKs and open-source tooling. The company develops hardware, software, and frameworks that accelerate AI training, inference, and deployment at every scale. NVIDIA's open-source projects — including OpenShell — extend its platform into agent runtimes, safety tooling, and developer workflows. With deep roots in GPU architecture and a growing focus on AI infrastructure, NVIDIA ships tools used by researchers, enterprises, and individual developers worldwide.

    Founded 1993
    2788 San Tomas Expressway
    42,000 employees

    Used by

    OpenAI
    Microsoft Azure
    Amazon Web Services
    Google Cloud
    +15 more
    Read more about NVIDIA
    WebsiteGitHubLinkedInX / Twitter
    17 tools in directory

    Similar Tools

    agent-skills-eval icon

    agent-skills-eval

    A test runner for Agent Skills that evaluates whether your SKILL.md actually improves model performance by running evals with and without the skill loaded.

    Purple Llama icon

    Purple Llama

    Meta's umbrella project of tools and evaluations for assessing and improving the security and safety of LLMs and generative AI applications.

    Coze Loop icon

    Coze Loop

    Open-source platform for developing, debugging, evaluating, and monitoring AI agents across their full lifecycle.

    Browse all tools

    Related Topics

    Code Security

    Tools that analyze code for security vulnerabilities and issues.

    110 tools

    LLM Evaluations

    Platforms and frameworks for evaluating, testing, and benchmarking LLM systems and AI applications. These tools provide evaluators and evaluation models to score AI outputs, measure hallucinations, assess RAG quality, detect failures, and optimize model performance. Features include automated testing with LLM-as-a-judge metrics, component-level evaluation with tracing, regression testing in CI/CD pipelines, custom evaluator creation, dataset curation, and real-time monitoring of production systems. Teams use these solutions to validate prompt effectiveness, compare models side-by-side, ensure answer correctness and relevance, identify bias and toxicity, prevent PII leakage, and continuously improve AI product quality through experiments, benchmarks, and performance analytics.

    248 tools

    Agent Skill Registries

    Registries and directories that catalog agent capabilities, skills, and competencies, enabling discovery and composition of agent abilities across platforms.

    219 tools
    Browse all topics
    Back to all toolsSuggest an edit
    ratings
    discussions