800 fake skills later, 'did anyone check this' is the only question left

On July 20, 2026, Island security researchers published an analysis of roughly 7,600 malicious GitHub repositories. More than 800 of them were dressed up as AI skills and MCP servers, and over 600 of those listings had already spread across public AI registries including LobeHub, Glama, MCP.so and MCP Market (Island Security Research, July 2026).

The technique was not sophisticated. Copy a real project. Register a username one character off a real developer's. Write a convincing README. Point the download button at a zip file.

The part worth your attention is what happened next. Island tested whether AI coding assistants would find these repositories on their own, without being handed a link. They did. In one run, an assistant read the attacker's README and repeated its installation steps back to the user as guidance.

That is the actual state of the ecosystem in mid-2026. Claude skills are trivially easy to install, genuinely useful when they work, and close to impossible to evaluate by looking at them. Which raises a question worth answering properly: what would it even mean to check one?

What a Claude skill is, and why it is hard to judge

A skill is a set of instructions that teaches an AI coding tool how to do a specific job. It is not a program in the traditional sense. It does not compile, it usually does not run in a sandbox you control, and there is often nothing to inspect except text files and a manifest. If you want the fuller breakdown of how skills differ from plugins and MCP servers, we wrote that up separately.

That format is exactly why skills spread so fast, and exactly why they are hard to vet. A skill is small enough to install in seconds and complex enough to do something you did not ask for. The gap between those two facts is where the whole problem lives.

Three things people tend to conflate:

It installed. The file landed in the right directory. This tells you almost nothing.

It runs. The platform actually picks it up and fires it when it should. Plenty of published skills fail here silently, with no error, because one field in the manifest does not match the platform's contract.

Someone checked it. A human or a system looked at what the skill does when it runs, and confirmed it matches what the description says.

Most public listings verify none of the three and let you assume all three. Star counts, download numbers and verification badges measure attention, not review. Island's own recommendation after the AgentBaiting research was blunt: treat a registry listing as no signal of legitimacy, and verify the publisher rather than the project.

The four layers, and what each one catches

Here is what our certification pass actually does. There are four layers, one gate each, and a skill has to clear every layer on every supported platform before it lists. There is no partial pass.

Each check returns pass, warn or fail, and the gate is per-skill and per-platform. A warn is a note. A fail is a fail.

Layer 1: Structure

Does the file exist, does it have shape, and is there anything actually in it?

The checks here are unglamorous on purpose. Does the adapter file exist at all for this platform. Does it clear a minimum byte floor, because a file under a few hundred bytes is a stub wearing a filename. Does it have a top-level heading. Does it have enough real sections to be a document rather than a title and a shrug (three is the pass line, two gets flagged, fewer fails).

Slugs get validated against a strict pattern, so identifiers stay clean and predictable rather than becoming whatever the author typed that day.

There is also a check most people would not think to write: the shipped file must not contain our internal branding or internal codenames. Customer artifacts should be about the skill, not about the pipeline that produced them or the internal names we use for our own tooling. If any of that leaks into a packaged file, the skill fails and gets fixed before it ships.

This layer exists because a surprising share of published skills are skeletons. Placeholder sections, a TODO, a heading with nothing under it. They install cleanly and do nothing, and you do not find out until the moment you needed them to work.

Layer 2: Platform fit

Does each adapter actually satisfy the platform it claims to run on?

Four platforms are supported: Claude Code, Codex, OpenClaw and Google Antigravity. Each gets its own adapter, and each adapter is checked against that platform's current contract, not a generic one. Each platform loads and triggers skills differently, which our install guides cover per platform.

Concretely, that means format-level verification per platform. Is this the current expected file layout rather than a deprecated one we have since moved off. Does the adapter carry the frontmatter that platform requires, with the specific fields that platform requires present in it. Are the tool references it declares actually valid.

The frontmatter requirement is enforced as its own CI gate, and critically it validates the packaged download, not just the source tree. Plenty of things are correct in a repo and wrong in the zip a customer actually receives. Checking the artifact you ship rather than the code you wrote is the difference between believing it works and knowing it does.

Two rules make this layer strict rather than advisory. All four adapters have to pass independently, so a skill with three solid adapters and one half-baked one does not ship at all. And a concept that only works on one platform is not eligible for the catalog, no matter how good it is.

This layer catches the most frustrating failure mode in the ecosystem: the skill installs, throws no error, and simply never fires. A skill that works in one tool and quietly does nothing in another is not certified. It is lucky.

Layer 3: Quality

Does it say anything specific, and does it tell the model enough to behave?

This is the layer that most catalogs do not have at all, and it is the one that decides whether a skill is worth installing rather than merely safe to.

Generic-template detection. We maintain a registry of the exact filler phrasings that LLM-generated skills produce when nobody edited them. You have read these without noticing: gather the information, produce the deliverable, structure the output clearly, verify it meets the requirements. A skill built out of those sentences will run and will produce nothing you could not have gotten by asking directly. One instance is a warning. More than one fails.

Boilerplate-process detection, which is separate. There is a specific four-step scaffold that generated skills fall into: understand the request, gather information, produce the deliverable, review quality. Matching that skeleton is an automatic fail, not a warning, because it means the skill has no actual method in it. It is a shape where a process should be.

Output template required. Does the skill define the structure of what it produces, or does it just hope? A skill without an output contract gives you something different every run, which makes it useless for anything repeated.

Constraint depth. Skills need a dedicated constraints or rules section, and it needs real rules in it, not one line. Three or more passes. One or two gets flagged. This is the section that tells the model what not to do, and it is the difference between a skill that stays in its lane and one that improvises.

Content depth. A hard word-count floor for production quality, currently 400 words, with a warn band below it. Not because length equals quality, but because below a certain point there is provably not enough instruction present to constrain a model's behavior.

Underneath all of that sits a similarity gate that compares every adapter against its nearest peer in the catalog and fails anything above a strict duplicate threshold. A handful of legacy adapters are grandfathered through a tracked exceptions ledger, each one recorded individually. New skills are never added to that ledger. They get rewritten below the threshold or they do not ship. That is the mechanism that stops a catalog from quietly filling with eleven versions of the same thing, which is the failure mode of every marketplace that optimizes for listing count.

Layer 4: The security scan

This is a separate 7-dimension content-security scan, and it is worth being precise about how it differs from the layers above. The first three layers are pass/fail checks. This one is seven scanners, each hunting a category of problem and returning findings rather than a verdict.

The seven dimensions, and what each one is actually looking for:

1. Prompt injection, including hidden-Unicode variants. Instructions buried in a skill file that the model reads as though they came from you. Folded into this dimension is an unsafe-markup check, since smuggled active content is the same class of problem: script and iframe tags, javascript: URLs, inline event handlers.
2. Hardcoded secrets. Live payment keys, webhook signing keys, email-service keys, PEM private keys, JWTs, and connection strings with the password sitting in them. The scanner is placeholder-aware, so an obvious dummy value in an example does not trigger a false alarm, which matters because a scanner people learn to ignore is worse than none.
3. Shell and command injection. Two things specifically. Pipe-to-shell one-liners, the curl something | sh pattern that hands an unreviewed remote script your permissions. And dynamic execution sinks invoked with anything other than a static literal, meaning the command is being assembled at runtime out of something the skill does not control.
4. Outbound URLs. Every http and https literal in the file is extracted and its host checked against an allowlist of expected destinations. An unknown host is a finding. This is the exfiltration check: not "is this domain evil," but "why is this talking to somewhere nobody expected."
5. Tool-use safety. Language that instructs unscoped capability. Run any command. Wildcard filesystem access. Unrestricted shell. Disable safety checks. Proceed without confirmation on something destructive. These widen the blast radius past what certification can meaningfully reason about.
6. Output schema. Whether an executable declares and validates an output contract, so whatever consumes it downstream gets a stable typed surface instead of a surprise.
7. Supply-chain provenance. Where the components came from and whether that chain holds up.

Findings are severity-ranked, and the ranking is the part that gives the layer teeth:

• A serious (P1) finding always blocks. There is no allowlist for it, no exception process, no override. This is deliberate.
• A medium (P2) finding blocks any new or changed skill, and blocks everything under the strict-mode runs CI uses.
• A small legacy backlog of P2s is downgraded to warnings through an explicit ledger, and every entry carries a written one-line justification for why it is acceptable. Not a blanket exemption. A list you can read.

So "every certified skill passed the security scan" is a sentence with a mechanism behind it rather than a marketing claim.

Ninety-six checks, and why we do not call it 103

The first three layers add up to 96 deterministic checks per skill, aggregated across all four platform adapters. The security layer is 7 scanners, not 7 checks.

We keep those numbers separate on purpose, and it is worth explaining why, because most vendors would just add them.

The 96 are discrete pass/fail. Each one either passes or it does not. The seven security dimensions are a different unit entirely: each runs many internal patterns and returns zero or more findings per file. Adding a scanner to a check produces a number that cannot be reconciled if anyone ever asks for the breakdown. If we said 103, and a partner or an enterprise security team asked us to enumerate the 103, the math would not hold.

So it stays 96 checks plus a separate 7-dimension security scan. If you want one round number, "100+ checks and scans" is defensible.

A number by itself is a vanity stat anyway. The explanation is what makes it useful, which is why this post exists.

What to check yourself, before you install anything

Certification is what we do so you do not have to. But most of what people install does not come from a certified catalog, so here are the checks that matter most when you are pulling from the open ecosystem. None of these take long.

Look at the shape of the package, and let a tool do the looking. This is the single fastest tell from the AgentBaiting research. A real skill or MCP server is source code with a manifest. If the install instructions say download the zip, extract it, and run the application, that is not a skill, and anything that installs through a downloaded executable should be rejected on sight no matter how good the README looks.

The good news is that none of this needs your eyeballs. Four commands, all free, all on tools you either have or can install in a minute:

List the archive without extracting it. On macOS or Linux, unzip -l package.zip or tar -tzf package.tar.gz. On Windows, 7-Zip's 7z l package.zip, or in PowerShell [IO.Compression.ZipFile]::OpenRead("$PWD\package.zip").Entries | Select FullName, Length. You get the full file list without a single byte executing. If you see a .exe, .cmd, .bat, .dll or a bundled runtime in something claiming to be a skill, you are done. Close it.

Check what files actually are, not what they are named. This is the one that catches the specific trick in this campaign. The malicious packages hid a roughly 300 KB obfuscated program inside a file with a .txt extension, next to a renamed runtime, so a directory listing looked harmless. On macOS or Linux, file * reads each file's actual header and tells you what it really is. Run it across an extracted skill directory and a text file that is not a text file has nowhere to hide. On Windows, Get-ChildItem -Recurse | Format-Table Name, Length will at least flag the size anomaly, since nobody's README is 300 KB.

Scan for secrets and suspicious patterns. Gitleaks and TruffleHog are both open source, both run locally, and both will find committed credentials in seconds. Semgrep has a free tier and open-source rulesets, and it is the right tool if you want to scan skill and server source for patterns like shell construction from untrusted input. These are the same categories our security layer covers, and you can run them yourself on anything you pull from the open ecosystem.

Hash it before you trust it. shasum -a 256 file on macOS or Linux, Get-FileHash file on Windows, then paste the hash into VirusTotal. Free, no account needed, and known-bad packages surface immediately. Island published SHA-256 hashes for the campaign packages, which is exactly what that list is for.

None of that replaces reading the thing. It does mean the first pass is a machine's job, and the machine is faster than you and does not get bored on the fortieth file.

Verify the publisher, not the project. The campaign built credibility from one-character-off usernames and copied profiles. Check the account history, not the star count on the repo. Registries reproduce the publisher's own README, so a listing carries the attacker's instructions onto another platform without adding any scrutiny.

Read what it does when it runs. Open the file. Look at what tools it asks the model to invoke and where it sends anything. This is the step everybody skips, and it is the one that catches the most.

Evaluate new capabilities somewhere isolated first. No browser sessions, no cloud credentials, no SSH keys, no production data. If a skill misbehaves, you want it misbehaving somewhere that does not matter.

Keep an inventory. Source repository, commit, version. When a campaign gets identified publicly, an inventory is the difference between knowing in minutes and never knowing at all. If you cannot list the AI tooling your team is running, no policy document fixes that. We have written more about how to run this kind of self-audit if you want a checklist.

Where this goes

The install step is not the risky moment anymore. Discovery is. When an assistant can search for a capability, find a repository, read a stranger's README and hand you the setup instructions, the decision about what to trust has already been made somewhere upstream of you.

That is the case for a curated catalog, and it is not only our case. It was Island's own first recommendation coming out of the AgentBaiting research: build a curated catalog of reviewed skills, MCP servers and agent plugins, because the campaign depends on open discovery and a reviewed catalog removes its audience.

We run one. Every skill in the certified catalog clears 96 checks across structure, platform fit and quality, plus a separate 7-dimension security scan, on all four platforms, before it lists. It holds 200+ certified skills and grows every month. If you would rather see how we pick the ones worth listing, that is a separate read.

We put our name on what is in there. That is the whole product.

Sources

• AgentBaiting: How 800+ Fake AI Skills and MCP Servers Delivered Malware, Island Security Research, July 20, 2026.
• Update on Exposed MCP Servers: The Threat Widens to the Cloud, Trend Micro, April 28, 2026.