API Integration with Pacific AI

Pacific AI provides API integration capabilities that allow organizations to automate AI evaluation workflows and integrate them into existing enterprise systems and CI/CD pipelines.

This enables continuous validation of AI systems running in production environments, especially for applications exposed through internal or external APIs.

Typical integration scenarios include:

• Continuous model evaluation
• Automated testing pipelines
• Compliance monitoring
• Incident management workflows
• Integration with enterprise dashboards and monitoring systems

API Integration Requirements

To integrate an application endpoint with Pacific AI, the target service must expose:

• A well-defined JSON request structure
• A consistent response schema
• Stable and documented API behavior

The API does not need to follow a Pacific AI–specific format, but request and response structures must remain predictable to ensure reliable evaluation execution.

This allows Pacific AI to interact with external AI applications consistently during automated testing.

---

GitHub Actions Integration

Pacific AI supports automated evaluation orchestration through GitHub Actions.

Organizations can integrate testing directly into CI/CD pipelines to:

• Trigger evaluations automatically
• Run scheduled assessments
• Validate deployments before release
• Generate reports programmatically

Example: GitHub Workflow Integration

This example demonstrates how Pacific AI can evaluate a chatbot application powered by an LLM.

Example Scenario

Assume your organization operates a chatbot application exposed through an API endpoint.

Users communicate with the chatbot through your application, and Pacific AI is used to evaluate the quality of responses in areas such as:

• Safety
• Reliability

In this example:

• MyChatBotEndpoint refers to the chatbot API endpoint
• MyChatBotCompany refers to the organization name
• MyChatBotTest refers to the system created within Pacific AI

Workflow Capabilities

The GitHub workflow automatically performs the following actions:

• Creates a temporary System in Pacific AI
• Creates a Dataset
• Configures an LLM Endpoint
• Creates a Run Configuration
• Starts evaluation runs
• Monitors execution progress
• Downloads evaluation reports (HTML and JSON)
• Cleans up created resources after completion

This approach enables fully automated evaluation pipelines without manual setup steps.

Configure GitHub Secrets

Before running the workflow, configure the required GitHub repository secrets.

Navigate to:

Repository → Settings → Secrets and Variables → Actions

Required Repository Secrets

Secret Name	Description
PLATFORM_URL	Pacific AI platform URL
PLATFORM_USER	Pacific AI username
PLATFORM_PASSWORD	Pacific AI password
MyChatBotCompany_CHAT_ENDPOINT	Chatbot API endpoint
MyChatBotEndpoint_USER	Endpoint username
MyChatBotEndpoint_PASSWORD	Endpoint password

Configure Dataset Variable

For custom datasets, define the following variable:

Variable	Description
DATASET_PATH	Path to the dataset CSV file within the GitHub repository

The dataset file must exist inside the repository used by the workflow.

Workflow Scope Options

The workflow supports multiple evaluation scopes depending on the required test coverage.

Basic

Runs a lightweight validation using a small sample dataset.

Example dataset:

TestCases_MyChatBotCompany_sample5Q.csv

Used for:

• Fast validation checks
• Smoke testing
• Pipeline verification

Full

Runs a complete evaluation using a large dataset.

Example dataset:

TestCases_MyChatBotCompany_766Q.csv

Used for:

• Comprehensive evaluation
• Pre-release testing
• Detailed reporting

Custom

Uses a custom dataset defined through DATASET_PATH.

The CSV file must contain the following headers:

Required CSV Headers
question
expected_answer
evaluation_points
must_do
must_avoid

Workflow Output

After execution completes:

• Temporary Pacific AI resources are deleted automatically
• Evaluation reports are generated
• Reports are stored as GitHub Artifacts

Generated reports include:

• HTML Report
• JSON Report

---

Running the Workflow Locally

The same evaluation workflow can also be executed locally using the provided Python script.

Example command:

python init_evaluation.py \
 -b "PACIFIC_AI_API_URL" \
 -u admin \
 -p "PASSWORD" \
 -q /path/to/TestCases_MyChatBotCompany_sample5Q.csv \
 --MyChatBotEndpoint https://MyChatBotEndpoint/aiplatform/api/v1/chat \
 --MyChatBotCompany-user "USERNAME" \
 --MyChatBotCompany-password "PASSWORD"

Local Execution Results

After local execution:

• Temporary Pacific AI resources are removed automatically
• HTML and JSON reports are saved locally

This approach is useful for:

• Local testing
• Development validation
• Debugging evaluation workflows

---

Important Considerations

The workflow automatically creates datasets using predefined Category and Subcategory values.

Pacific AI does not support multiple datasets with identical Category/Subcategory combinations simultaneously.

Because of this:

• Dataset creation may fail if cleanup was not completed successfully
• Concurrent workflow executions using the same dataset configuration may conflict

To avoid conflicts, ensure that:

• Previous runs are cleaned up properly
• Parallel workflows use unique dataset categories or configurations

API integration enables organizations to incorporate AI evaluation directly into operational workflows, helping teams continuously validate system quality, reliability, and governance controls throughout the deployment lifecycle.