⛵ Voyager

While Fathom provides fast heuristic and single-skill probabilistic checks, Voyager is a dedicated integration testing suite for your agent skills. It simulates an entire agent loop to verify that the LLM uses the correct sequence of tools to reach the expected end state.

Voyager reads your active agent berths, constructs JSON Schema tool definitions, and passes mock context payloads to ensure your "fleet" is ready for real-world deployment.

If Voyager discovers unmanaged local skills while preparing that fleet, it can also surface them as ghost-docking candidates. For the conceptual model behind that—and for the new primary skill-harbor ghosts workflow—see Ghosts.

At a high level, Voyager:

loads active skills from berths
exposes them as tools
runs a model loop
validates expected tool usage sequence

⚓ Voyager TL;DR

# Run a specific integration test file
skill-harbor voyager -f harbor-voyager-test.yaml

# Run an ad-hoc query simulation
skill-harbor voyager "Check if the codebase is portable."

🛠️ Defining a Voyager Test

Voyager tests are defined in YAML files (typically harbor-voyager-test.yaml). This allows you to define the user query, specify which tools must be invoked, and provide mock responses for those tools.

Test Structure Example

# harbor-voyager-test.yaml
query: "Check if the codebase is portable and then generate a report on any hidden skills."
expected_tools: 
  - Scryer
  - Fathom
mocks:
  Scryer: "Portable issues found: None. The codebase looks clean."
  Fathom: "Hidden skills report: 2 ghost skills found."

Key Parameters:

query: The initial prompt sent to the agent.
expected_tools: A list of tool names that the agent must call (in any order) to pass the test.
mocks: A mapping of tool names to their simulated return values. This prevents the agent from actually executing destructive commands during the test.

🧪 Benchmark Packs

Voyager now supports a Harbor-native benchmark-pack format for deterministic, fixture-driven scenario evaluation in local and CI environments. This complements the existing live single-scenario flow without replacing it.

Legacy scenario files remain the current single-test YAML shape (query, expected_tools, mocks, assertions).
Benchmark-pack files use a versioned Harbor-native root with kind, version, pack, and scenarios.
Pack execution is offline and API-key-free in v1: it evaluates scenario outcomes from fixtures rather than running the live provider loop.

Product boundary

Voyager owns empirical scenario evaluation: traces, branch outcomes, assertions, uplift/regression.
Fathom owns predictive and audit-style analysis: heuristics, token/context analysis, routing confidence, fleet recommendations.

That means benchmark packs still belong in Voyager: they make scenario evaluation reproducible, but they do not compute Fathom-style usefulness heuristics.

Benchmark-pack example

kind: harbor.voyager.benchmark-pack
version: 1
pack:
  id: sample-benchmark-pack
  name: Sample Voyager Benchmark Pack
scenarios:
  - id: tool-uplift
    query: "Check if the codebase is portable and then generate a report on any hidden skills."
    fixtures:
      with_skills: ...
      without_skills: ...
    assertions:
      with_skills: ...
      without_skills: ...
      delta: ...

Use a checked-in pack such as harbor-voyager-benchmark-pack.yaml to run deterministic benchmark-style evaluations in CI.

🛡️ Governance & CI/CD

Voyager is designed to be a Pull Request Gate. If the agent deviates from the expected_tools or fails to reach a terminal state, Voyager will exit with process code 1, blocking your CI/CD pipeline.

--file <path>: Provide a custom test definition file.
--model <name>: Override the model used for the simulation (e.g., gpt-4o-mini).
--baseUrl <url>: Point to a different LLM provider (Groq, Ollama, etc.).

⚓ Why Use Voyager?

Regression Testing: Ensure that adding a new skill doesn't break the routing logic for existing skills.
Chaining Validation: Verify that the output of one tool is correctly utilized by the next tool in a multi-step journey.
Mocking Destructive Actions: Test agents that use non-idempotent tools (like rm or git commit) without actually modifying your environment.
Team Standards: Commit your voyager tests to Git so every developer can verify the fleet's integrity before merging.

# Run Voyager with a specific model override
skill-harbor voyager -f harbor-voyager-test.yaml --model gpt-4o

Reference: Li, X., et al. (2026). SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. https://www.skillsbench.ai/skillsbench.pdf

For the broader alignment story, see How does Skill Harbor align with SkillsBench?.