Swift Accessibility Agent

Building a skill that can be evaluated

Good AI-agent instructions need more than a persuasive prompt. They need a way to establish which claims are supported, identify where an agent goes wrong, and test whether revised guidance actually helps.

Swift Accessibility Agent is my exploration of that engineering discipline, applied to Apple-platform accessibility. It combines a released guidance skill with a repository of executable examples, preserved failure histories, and controlled evaluation tooling. The goal is practical: help coding agents preserve working interfaces, identify real accessibility defects, and make justified changes.

Turn agent mistakes into testable questions

The project starts with specific failure modes: replacing a working control unnecessarily, hiding an ordinary action behind an accessibility shortcut, retaining a stale callback, or treating configured metadata as proof of behavior nobody observed.

Each proposed improvement follows a traceable sequence:

1. Establish the claim. Identify the relevant Apple documentation and SDK declarations, separating framework behavior from application policy.
2. Build an executable contract. Write a focused Swift example and assertions for the exact behavior under investigation. Introduce a targeted defect where useful to check that the assertion can detect it.
3. Preserve the difficulty. Keep the initial source, diagnostic output, environment, and repair history. Distinguish implementation bugs from faulty test expectations and harness problems.
4. Create a fair developer task. Write a realistic question and a separate rubric that accepts valid alternatives without leaking the solution into the agent’s input.
5. Evaluate before promotion. Compare frozen guidance conditions on the same cases, then review the evidence before updating the released skill.

For example, a checkout contract protects an existing button’s identity, typography, and action. A quantity-control example preserves ordinary plus/minus buttons alongside accessibility adjustment. A motion example updates final status independently of optional animation. The task is to preserve useful behavior—not simply add more accessibility modifiers.

Give the agent the right context, not the entire repository

The released skill selects guidance by platform and topic. Its contract asks agents to disclose the guidance actually consulted and distinguish sources from assumptions. This makes the reasoning easier to inspect without filling every prompt with the whole reference library.

The repository companion adds on-demand implementation detail: Swift source, tests, contract manifests, validation records, and difficulty histories. A candidate retrieval workflow for AppKit/macOS and UIKit/tvOS starts with one or two guidelines, adds source when an example is requested, and includes tests and manifests when a contract is requested. It records file hashes and bytes so retrieval can be examined rather than inferred from a model naming a document.

That companion-routing candidate is not yet part of the released skill. Keeping experimental tooling separate from the installed contract is an intentional release boundary.

Measure instruction quality and retrieval separately

The evaluation harness uses promptfoo and repository tooling to distinguish four questions:

Correctness and task success are scored separately from branding, routing, and disclosure. Naming a reference does not prove it was read; retrieval claims need traces. Comparisons track unsupported claims, API availability mistakes, preservation of existing behavior, tokens, reference bytes, tool calls, latency, and available cost data.

The primary target is frontier Swift coding agents. Smaller local-model pilots provide diagnostic observations, not a substitute for evaluation against that target. Existing development incidents remain development cases rather than being relabeled as fresh holdouts.

Executable evidence with explicit limits

The November 6, 2026 README checkpoint reports these companion results:

These are version-scoped compile and unit-test results, not coverage percentages, measured improvements in agent answers, or proof of delivered VoiceOver navigation. SwiftUI and visionOS guidance is available in the skill, but these companion results cover the three lanes listed above. The UIKit campaign records SDK 27.0 separately from its iOS 26.5 runtime; compiling against an SDK does not establish behavior on that OS version.

A passing example establishes a bounded implementation behavior. A better model answer requires separate paired-evaluation evidence. That distinction is central to the project.

Keep sources and observations traceable

The current source model distinguishes Apple documentation, declarations, and Apple-authored examples from implementation commentary and discovery or user-experience research. Project-authored tests supplement these sources with version-specific observations; they do not become a new authority tier.

Why this is an agent-engineering project

Accessibility is the domain; the broader contribution is the method for developing a specialized AI skill. Focused context, executable contracts, negative controls, preserved failures, independent rubrics, and measured release gates make proposed improvements inspectable and challengeable.

This approach treats a skill as an engineering artifact that needs maintenance and evidence—not a static collection of tips or a promise that an agent will always be correct.

Use it and inspect the work

The released skill is 0.2.2 at the November 6 checkpoint. New companion findings and retrieval changes await review and evaluation before promotion into a future release. The repository is public under the MIT license.

After installation, ask a normal development question—for example, review a table cell’s accessibility while preserving its existing buttons and actions. Use the repository companion when you need to inspect or rerun a specific implementation contract with the relevant Apple tooling.

Install the skill · Explore the methodology · Inspect the evaluation harness