Test Suites & Leaderboard

The Test Suites module allows users to configure, manage, execute, and compare evaluation scenarios for LLM-powered systems. It provides a structured way to test performance across multiple categories and maintain versioned evaluation records. Before creating or running a Test Suite, at least one LLM endpoint must be configured from Endpoints Page.

---

Create, Update, Clone, and Delete Test Suites

The Test Suites section (under the System parent node) is where all existing test configurations are stored and managed. Selecting this option opens the Test Suite page, which displays all test suites accessible to the user.

Create a Test Suite

1. Navigate to the Test Suite page.
2. Click the Create button to open the Create Test Config form.
3. Provide the following details:
   • Name (mandatory)
   • Description (optional)
   • Select the LLM
   • Specify the number of instances
4. Select the required test categories using the checkboxes on the right side of the screen.
5. Click Save to complete the configuration.

Once saved, the Test Suite becomes available for execution.

Update a Test Suite

1. Go to the Test Suite page.
2. Click the kebab menu (three dots) next to the desired Test Suite.
3. Select Update Test Config.
4. Make the necessary changes and save.

Clone a Test Suite

Cloning allows users to duplicate an existing configuration, which is useful when creating variations of a test setup.

1. Click the kebab menu next to the Test Suite.
2. Select Clone Test Suite. This will clone the configuration of the existing test suite and create a new one.

Delete a Test Suite

1. Click the kebab menu next to the Test Suite.
2. Select Delete.

Caution

Deleting a Test Suite will also delete all associated test runs permanently.

Publish Scores to Model Card

Users can publish Test Suite scores to the Model Card:

1. Click the kebab menu next to the Test Suite.
2. Select Publish in Model Card.

Note

After publishing, users must regenerate the Model Card to view the updated scores.

---

Run and Compare Tests

After a Test Suite is created, users can execute evaluations against the configured LLM.

Run a Test Suite

• Click the Evaluate button to start a test run.
• A new Test Run ID is generated for each execution.
• Test Suites can be executed multiple times.

Each run is tracked independently, enabling performance comparison over time.

Monitor and Manage Test Runs

The Test Suite page displays the status and progress of each test run. Users can monitor running evaluations, review completed results, and manage test execution as needed.

• Click on the Run ID to view the details of a test run. For running evaluations, the page displays the current progress. For completed runs, it displays the scores, metrics, and execution results.
• Users can stop an entire test run when the evaluation is no longer required or needs to be interrupted.
• Users can also stop an individual category within a running test run without stopping the remaining categories.
• Failed tasks can be reviewed individually, and users can manually rerun specific failed tasks without rerunning the entire test run.

These options allow users to monitor test execution and take action on individual runs, categories, or failed tasks without unnecessarily restarting the complete evaluation.

Export Test Results

Users can export individual test run results and their details in JSON or CSV format.

1. Open the relevant Test Suite and select an individual test run to view its results.
2. Click the Export button.
3. Select the desired export format: JSON or CSV.
4. Click Export to download the test run results and details in the selected format.

Compare Test Runs

Users can compare results across multiple runs:

1. Expand the relevant Test Suite.
2. Select the test runs to compare.
3. Click the Compare button at the top-right of the screen.

This comparison view highlights differences in metrics across selected runs.

---

Leaderboard

The System Leaderboard provides a consolidated view of the evaluation results for the active system. It allows users to review and compare scores across test categories and subcategories based on the Test Suite evaluations performed for the system.

Note

This Leaderboard is scoped to the currently selected System only. Unlike the general Leaderboard, it does not provide visibility across other Systems or test runs in Pacific AI. To learn more about the application level leaderboard and compare results across all Systems and model versions, see the Leaderboard.

The Leaderboard provides the following capabilities:

• Displays the evaluation score for each individual test subcategory.
• Uses the latest run when multiple test runs are available within a Test Suite.
• Allows users to filter the displayed results to focus on specific evaluation criteria.
• Provides separate category tabs so users can navigate directly to the evaluation category they want to review.
• Allows users to select an individual subcategory to view additional information about its evaluation results.
• Highlights the highest score among the available test runs in bold green for quick identification.

The System Leaderboard provides a centralized view of evaluation performance, helping users review results across categories, identify areas of strong or weak performance, and investigate individual subcategory results when additional information is required.