AWS Marketplace Installation Guide

Pacific AI : AWS Marketplace Installation Guide

This guide walks you through deploying Pacific AI on Amazon EKS manually, step by step. Each section provides the exact commands to run in your terminal. Total time is 35–55 minutes (cluster creation accounts for approximately 15–20 minutes of that).

---

Table of Contents

1. Overview
2. Before You Begin
   • Required Tools
   • IAM Permissions
   • Configure AWS Credentials
3. Step 1 : Set Configuration Variables
4. Step 2 : Create or Connect an EKS Cluster
   • Option A : Create a new cluster
   • Option B : Use an existing cluster
5. Step 3 : Configure IMDS on Cluster Nodes
6. Step 4 : Create a Kubernetes Namespace
7. Step 5 : Set Up Amazon EFS Storage
8. Step 6 : Pull the Helm Chart from Marketplace ECR
9. Step 7 : Install the AWS Load Balancer Controller
10. Step 8 : Generate Credentials and Install Pacific AI
    • Generate passwords
    • Run Helm install : HTTP only
    • Run Helm install : Traefik TLS
    • Run Helm install : AWS ALB
11. Step 9 : Attach the Marketplace IAM Service Account
12. Step 10 : Determine the Application URL
13. After Installation
14. Sensitive Information : Locations and Handling
15. Data Encryption
16. Credential and Key Rotation
17. Monitoring Application Health
18. TLS/SSL Reference
19. How the Application Accesses AWS
20. Troubleshooting
21. Uninstalling

---

1. Overview

Pacific AI is deployed onto Amazon EKS using Helm. The installation creates the following AWS resources in your account:

Resource	Purpose
EKS cluster + managed node group	Kubernetes runtime for all application pods
IAM OIDC provider	Enables IAM Roles for Service Accounts (IRSA)
Amazon EFS filesystem + mount targets	Shared persistent storage for application data
EBS CSI driver (IRSA service account)	Provisions EBS volumes for MongoDB
EFS CSI driver	Mounts EFS volumes into pods
StorageClass pacific-ai-efs	Default storage class backed by EFS
IRSA role for pacific-ai-pacific-ai	Grants pods access to Marketplace Metering and License Manager
Network or Application Load Balancer	Routes external traffic into the cluster

How AWS credentials work during installation: You run each command below using credentials already configured in your environment (aws configure, an instance profile, SSO, or environment variables). No credentials are entered into any form or prompt during the installation itself.

How the application accesses AWS at runtime: Once installed, pods use IAM Roles for Service Accounts (IRSA) exclusively. No static AWS credentials are stored in the cluster, in environment variables, or in Kubernetes Secrets. See section 22 for details.

---

2. Before You Begin

2.1 Required Tools

Install all four tools on the machine where you will run these commands.

Tool	Minimum version	Install guide
AWS CLI	2.15.0	https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
kubectl	any recent	https://kubernetes.io/docs/tasks/tools/
Helm	3.7.1	https://helm.sh/docs/intro/install/
eksctl	any recent	https://eksctl.io/installation/

Verify before proceeding:

aws --version        # must be 2.15.0+
kubectl version --client
helm version --short  # must be 3.7.1+
eksctl version

2.2 IAM Permissions

The AWS identity running these commands needs the following permissions.

AWS-managed policies : attach all three to your IAM user or role:

arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
arn:aws:iam::aws:policy/AmazonEC2FullAccess
arn:aws:iam::aws:policy/AWSCloudFormationFullAccess

Customer-managed policy : create this in IAM → Policies → Create policy → JSON, then attach it to the same user or role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EKSManagement",
      "Effect": "Allow",
      "Action": ["eks:*"],
      "Resource": "*"
    },
    {
      "Sid": "IAMForEksctl",
      "Effect": "Allow",
      "Action": [
        "iam:CreateRole", "iam:DeleteRole", "iam:GetRole", "iam:ListRoles",
        "iam:AttachRolePolicy", "iam:DetachRolePolicy",
        "iam:ListAttachedRolePolicies", "iam:ListRolePolicies",
        "iam:PutRolePolicy", "iam:DeleteRolePolicy", "iam:GetRolePolicy",
        "iam:PassRole", "iam:CreateOpenIDConnectProvider",
        "iam:DeleteOpenIDConnectProvider", "iam:GetOpenIDConnectProvider",
        "iam:ListOpenIDConnectProviders", "iam:TagOpenIDConnectProvider",
        "iam:CreateServiceLinkedRole", "iam:TagRole", "iam:UntagRole"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ECRMarketplaceLogin",
      "Effect": "Allow",
      "Action": ["ecr:GetAuthorizationToken"],
      "Resource": "*"
    },
    {
      "Sid": "ECRPull",
      "Effect": "Allow",
      "Action": [
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchCheckLayerAvailability"
      ],
      "Resource": "arn:aws:ecr:*:709825985650:repository/pacific-ai/*"
    },
    {
      "Sid": "AutoScaling",
      "Effect": "Allow",
      "Action": [
        "autoscaling:CreateAutoScalingGroup", "autoscaling:DeleteAutoScalingGroup",
        "autoscaling:DescribeAutoScalingGroups", "autoscaling:UpdateAutoScalingGroup",
        "autoscaling:CreateLaunchConfiguration", "autoscaling:DeleteLaunchConfiguration",
        "autoscaling:DescribeLaunchConfigurations", "autoscaling:CreateOrUpdateTags",
        "autoscaling:EnableMetricsCollection",
        "ec2:CreateLaunchTemplate", "ec2:DeleteLaunchTemplate",
        "ec2:DescribeLaunchTemplates", "ec2:DescribeLaunchTemplateVersions"
      ],
      "Resource": "*"
    },
    {
      "Sid": "SSMForNodeAMIs",
      "Effect": "Allow",
      "Action": ["ssm:GetParameter", "ssm:GetParameters"],
      "Resource": "arn:aws:ssm:*:*:parameter/aws/service/eks/*"
    },
    {
      "Sid": "EFSManagement",
      "Effect": "Allow",
      "Action": [
        "elasticfilesystem:CreateFileSystem", "elasticfilesystem:DeleteFileSystem",
        "elasticfilesystem:DescribeFileSystems", "elasticfilesystem:CreateMountTarget",
        "elasticfilesystem:DeleteMountTarget", "elasticfilesystem:DescribeMountTargets",
        "elasticfilesystem:TagResource", "elasticfilesystem:CreateTags"
      ],
      "Resource": "*"
    },
    {
      "Sid": "STSCallerIdentity",
      "Effect": "Allow",
      "Action": ["sts:GetCallerIdentity"],
      "Resource": "*"
    }
  ]
}

Tip : existing cluster: If deploying onto an existing cluster, omit AmazonEC2FullAccess, AWSCloudFormationFullAccess, EKSManagement, AutoScaling, and SSMForNodeAMIs.

2.3 Configure AWS Credentials

Configure credentials using any standard AWS method before running the commands below.

Option A : AWS CLI configuration (recommended for local machines):

aws configure

Option B : IAM Identity Center (SSO):

aws configure sso
aws sso login --profile <your-profile>
export AWS_PROFILE=<your-profile>

Option C : Environment variables:

export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...   # only for temporary/assumed-role credentials

Option D : EC2 instance profile: Attach an IAM role to your EC2 instance or Cloud9 environment. No credential configuration needed.

Verify your credentials before proceeding:

aws sts get-caller-identity

Confirm the account ID and ARN are correct before continuing.

---

3. Step 1 : Set Configuration Variables

Set these shell variables once. All subsequent commands reference them so you only need to substitute your values here.

export CLUSTER_NAME="pacific-ai"          # EKS cluster name
export REGION="us-east-1"                 # AWS region
export NAMESPACE="pacific-ai"             # Kubernetes namespace
export NODE_TYPE="m4.4xlarge"             # EC2 instance type for worker nodes
export NODE_COUNT="2"                     # Number of worker nodes
export K8S_VERSION="1.35"                 # Kubernetes version (1.33, 1.34, or 1.35)
export CHART_VERSION="1.1.0"              # From your AWS Marketplace subscription

export AWS_DEFAULT_REGION="$REGION"

Where to find your chart version: Log in to the AWS Marketplace portal, go to your Pacific AI subscription, and note the version listed under Container images.

---

4. Step 2 : Create or Connect an EKS Cluster

Choose Option A to create a new cluster, or Option B if you already have one.

Option A : Create a new cluster

Create an eksctl configuration file and provision the cluster. The iam.withOIDC: true field creates the OIDC identity provider required for IRSA.

cat > /tmp/pacific-ai-cluster.yaml <<EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ${CLUSTER_NAME}
  region: ${REGION}
  version: "${K8S_VERSION}"

iam:
  withOIDC: true

autoModeConfig:
  enabled: false

managedNodeGroups:
  - name: pacific-ai-nodes
    instanceType: ${NODE_TYPE}
    desiredCapacity: ${NODE_COUNT}
    minSize: 1
    maxSize: $((NODE_COUNT * 2))
EOF

eksctl create cluster -f /tmp/pacific-ai-cluster.yaml
rm /tmp/pacific-ai-cluster.yaml

This takes approximately 15–20 minutes. When complete, eksctl automatically updates your kubeconfig.

Verify the cluster is ready:

kubectl get nodes

All nodes should show STATUS: Ready.

Option B : Use an existing cluster

Update your kubeconfig and ensure the OIDC provider is associated (required for IRSA; this is a no-op if already configured):

aws eks update-kubeconfig \
  --name "$CLUSTER_NAME" \
  --region "$REGION"

eksctl utils associate-iam-oidc-provider \
  --cluster "$CLUSTER_NAME" \
  --region "$REGION" \
  --approve

Verify connectivity:

kubectl get nodes

Existing cluster requirements: Kubernetes version 1.33–1.35, at least 2 worker nodes with m4.2xlarge or larger, and your kubectl must have network access to the API server.

---

5. Step 3 : Configure IMDS on Cluster Nodes

Skip this step if you used Option B (existing cluster). This applies only to clusters created in Step 2 Option A.

The IMDS hop limit must be set to 2 so that pods can reach the instance metadata service : this is required for IRSA tokens to be delivered to containers. Run the following for each worker node:

INSTANCE_IDS=$(aws ec2 describe-instances \
  --region "$REGION" \
  --filters \
    "Name=tag:eks:cluster-name,Values=${CLUSTER_NAME}" \
    "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].InstanceId' \
  --output text)

for INSTANCE_ID in $INSTANCE_IDS; do
  aws ec2 modify-instance-metadata-options \
    --instance-id "$INSTANCE_ID" \
    --http-tokens optional \
    --http-put-response-hop-limit 2 \
    --region "$REGION"
  echo "Updated IMDS on $INSTANCE_ID"
done

Also update the node group’s launch template so replacement nodes inherit this setting:

NG_ASG=$(aws autoscaling describe-auto-scaling-groups \
  --region "$REGION" \
  --query "AutoScalingGroups[?Tags[?Key=='eks:cluster-name' && Value=='${CLUSTER_NAME}']].AutoScalingGroupName | [0]" \
  --output text)

LT_ID=$(aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names "$NG_ASG" \
  --region "$REGION" \
  --query 'AutoScalingGroups[0].LaunchTemplate.LaunchTemplateId' \
  --output text)

NEW_VER=$(aws ec2 create-launch-template-version \
  --launch-template-id "$LT_ID" \
  --source-version '$Latest' \
  --launch-template-data '{"MetadataOptions":{"HttpTokens":"optional","HttpPutResponseHopLimit":2}}' \
  --region "$REGION" \
  --query 'LaunchTemplateVersion.VersionNumber' \
  --output text)

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name "$NG_ASG" \
  --launch-template "LaunchTemplateId=${LT_ID},Version=${NEW_VER}" \
  --region "$REGION"

echo "Launch template updated to version ${NEW_VER}"

---

6. Step 4 : Create a Kubernetes Namespace

kubectl create namespace "$NAMESPACE"
kubectl config set-context --current --namespace="$NAMESPACE"

Verify:

kubectl get namespace "$NAMESPACE"

---

7. Step 5 : Set Up Amazon EFS Storage

This step creates the EFS filesystem for shared persistent storage and installs the EBS and EFS CSI drivers.

7.1 Gather cluster network information

VPC_ID=$(aws eks describe-cluster \
  --name "$CLUSTER_NAME" --region "$REGION" \
  --query 'cluster.resourcesVpcConfig.vpcId' --output text)

SUBNET_IDS=$(aws eks describe-cluster \
  --name "$CLUSTER_NAME" --region "$REGION" \
  --query 'cluster.resourcesVpcConfig.subnetIds[]' --output text)

VPC_CIDR=$(aws ec2 describe-vpcs \
  --vpc-ids "$VPC_ID" --region "$REGION" \
  --query 'Vpcs[0].CidrBlock' --output text)

echo "VPC:     $VPC_ID"
echo "CIDR:    $VPC_CIDR"
echo "Subnets: $SUBNET_IDS"

7.2 Create the EFS security group

EFS_SG_ID=$(aws ec2 create-security-group \
  --group-name "${CLUSTER_NAME}-efs-sg" \
  --description "EFS security group for Pacific AI cluster ${CLUSTER_NAME}" \
  --vpc-id "$VPC_ID" \
  --region "$REGION" \
  --query 'GroupId' --output text)

aws ec2 authorize-security-group-ingress \
  --group-id "$EFS_SG_ID" \
  --protocol tcp --port 2049 \
  --cidr "$VPC_CIDR" \
  --region "$REGION"

aws ec2 create-tags \
  --resources "$EFS_SG_ID" \
  --tags "Key=Name,Value=${CLUSTER_NAME}-efs-sg" \
  --region "$REGION"

echo "EFS security group: $EFS_SG_ID"

7.3 Create the EFS filesystem

EFS_ID=$(aws efs create-file-system \
  --performance-mode generalPurpose \
  --throughput-mode bursting \
  --encrypted \
  --region "$REGION" \
  --tags "Key=Name,Value=${CLUSTER_NAME}-efs" \
  --query 'FileSystemId' --output text)

echo "EFS filesystem: $EFS_ID"

Wait for the filesystem to become available before creating mount targets:

echo "Waiting for EFS filesystem to become available..."
while true; do
  STATE=$(aws efs describe-file-systems \
    --file-system-id "$EFS_ID" --region "$REGION" \
    --query 'FileSystems[0].LifeCycleState' --output text)
  [ "$STATE" = "available" ] && echo "EFS is available." && break
  echo "  State: $STATE : waiting..."
  sleep 5
done

7.4 Create mount targets

for SUBNET in $SUBNET_IDS; do
  aws efs create-mount-target \
    --file-system-id "$EFS_ID" \
    --subnet-id "$SUBNET" \
    --security-groups "$EFS_SG_ID" \
    --region "$REGION" \
  && echo "Mount target created in $SUBNET" \
  || echo "Mount target in $SUBNET may already exist : skipping."
done

7.5 Install the EBS CSI driver

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster "$CLUSTER_NAME" \
  --region "$REGION" \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --override-existing-serviceaccounts

EBS_ROLE_ARN=$(kubectl get sa ebs-csi-controller-sa -n kube-system \
  -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}')

aws eks create-addon \
  --cluster-name "$CLUSTER_NAME" \
  --region "$REGION" \
  --addon-name aws-ebs-csi-driver \
  --service-account-role-arn "$EBS_ROLE_ARN" \
  --resolve-conflicts OVERWRITE

echo "Waiting for EBS CSI driver to become active..."
aws eks wait addon-active \
  --cluster-name "$CLUSTER_NAME" \
  --region "$REGION" \
  --addon-name aws-ebs-csi-driver

echo "EBS CSI driver is ready."

7.6 Install the EFS CSI driver

eksctl create addon \
  --name aws-efs-csi-driver \
  --cluster "$CLUSTER_NAME" \
  --region "$REGION" \
  --force

echo "Waiting for EFS CSI controller deployment..."
until kubectl get deployment efs-csi-controller -n kube-system >/dev/null 2>&1; do
  sleep 5
done

kubectl rollout status deployment efs-csi-controller -n kube-system --timeout=120s
echo "EFS CSI driver is ready."

7.7 Create the StorageClass

kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pacific-ai-efs
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
parameters:
  provisioningMode: efs-ap
  fileSystemId: "${EFS_ID}"
  directoryPerms: "777"
  uid: "0"
  gid: "0"
  basePath: "/pacific-ai"
provisioner: efs.csi.aws.com
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: Immediate
EOF

echo "StorageClass 'pacific-ai-efs' created."

---

8. Step 6 : Pull the Helm Chart from Marketplace ECR

Authenticate to the Marketplace ECR registry using your active AWS credentials:

aws ecr get-login-password --region us-east-1 \
  | helm registry login \
      --username AWS \
      --password-stdin \
      709825985650.dkr.ecr.us-east-1.amazonaws.com

echo "Helm registry login successful."

Pull and extract the chart:

rm -rf awsmp-chart
mkdir -p awsmp-chart

helm pull oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/pacific-ai/pacific-ai \
  --version "$CHART_VERSION" \
  --destination awsmp-chart

cd awsmp-chart && tar xf ./*.tgz && rm ./*.tgz && cd ..

CHART_DIR="awsmp-chart/pacific-ai"
echo "Chart extracted to: $CHART_DIR"

---

9. Step 7 : Install the AWS Load Balancer Controller

Skip this step if you are using HTTP-only or Traefik TLS mode. This is required only for AWS ALB TLS mode.

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

VPC_ID=$(aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" \
  --query "cluster.resourcesVpcConfig.vpcId" --output text)

# Download the IAM policy document (pinned to LBC v2.8.2)
curl -fsSL \
  "https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.8.2/docs/install/iam_policy.json" \
  -o /tmp/aws-lbc-iam-policy.json

POLICY_NAME="AWSLoadBalancerControllerIAMPolicy-${CLUSTER_NAME}"
POLICY_ARN="arn:aws:iam::${ACCOUNT_ID}:policy/${POLICY_NAME}"

aws iam create-policy \
  --policy-name "$POLICY_NAME" \
  --policy-document file:///tmp/aws-lbc-iam-policy.json 2>/dev/null \
  || echo "IAM policy already exists : continuing."

rm /tmp/aws-lbc-iam-policy.json

eksctl create iamserviceaccount \
  --cluster="$CLUSTER_NAME" \
  --region="$REGION" \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name="AmazonEKSLoadBalancerControllerRole-${CLUSTER_NAME}" \
  --attach-policy-arn="$POLICY_ARN" \
  --override-existing-serviceaccounts \
  --approve

helm repo add eks https://aws.github.io/eks-charts 2>/dev/null || true
helm repo update eks

helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --namespace kube-system \
  --version 1.8.2 \
  --set "clusterName=${CLUSTER_NAME}" \
  --set "serviceAccount.create=false" \
  --set "serviceAccount.name=aws-load-balancer-controller" \
  --set "region=${REGION}" \
  --set "vpcId=${VPC_ID}" \
  --set "enableServiceMutatorWebhook=false"

kubectl rollout status deployment aws-load-balancer-controller \
  -n kube-system --timeout=3m

echo "AWS Load Balancer Controller is ready."

---

10. Step 8 : Generate Credentials and Install Pacific AI

Generate passwords

Run this block to generate all service credentials. The values are stored in shell variables and written to a credentials file.

KC_ADMIN_PW=$(openssl rand -hex 16)
KC_USER_ADMIN=$(openssl rand -hex 16)
KC_USER_MEMBER=$(openssl rand -hex 16)
KC_USER_GOV=$(openssl rand -hex 16)
KC_USER_POLICY=$(openssl rand -hex 16)
KC_USER_VENDOR=$(openssl rand -hex 16)
KC_USER_COMPLIANCE=$(openssl rand -hex 16)
KC_USER_RISK=$(openssl rand -hex 16)
KC_USER_DEVELOPER=$(openssl rand -hex 16)
KC_CLIENT_SEC=$(printf '%s-%s-%s-%s-%s' \
  "$(openssl rand -hex 4)" "$(openssl rand -hex 2)" \
  "$(openssl rand -hex 2)" "$(openssl rand -hex 2)" \
  "$(openssl rand -hex 6)")
KC_PG_PW=$(openssl rand -hex 16)
KC_PG_ROOT_PW=$(openssl rand -hex 16)
PG_PW=$(openssl rand -hex 16)
REDIS_PW=$(openssl rand -hex 16)
MONGO_ROOT_PW=$(openssl rand -hex 16)
MONGO_PW=$(openssl rand -hex 16)
LLM_GW_KEY="sk-$(openssl rand -hex 16)"
LLM_GW_REDIS_PW=$(openssl rand -hex 16)
INTERNAL_API_KEY=$(openssl rand -hex 16)

# Save credentials to a local file (permissions 600 : readable only by you)
cat > ./pacific-ai-credentials.txt <<EOF
Pacific AI : Installation Credentials
Generated: $(date)

  Cluster:   ${CLUSTER_NAME}
  Namespace: ${NAMESPACE}
  Region:    ${REGION}

Application Users:
  admin               password: ${KC_USER_ADMIN}
  member              password: ${KC_USER_MEMBER}
  governance_officer  password: ${KC_USER_GOV}
  policy_manager      password: ${KC_USER_POLICY}
  vendor_manager      password: ${KC_USER_VENDOR}
  compliance_officer  password: ${KC_USER_COMPLIANCE}
  risk_manager        password: ${KC_USER_RISK}
  developer           password: ${KC_USER_DEVELOPER}

Infrastructure (internal use):
  Keycloak admin password: ${KC_ADMIN_PW}

IMPORTANT: Copy these credentials to a secure vault, then delete this file.
           Never share or commit this file.
EOF
chmod 600 ./pacific-ai-credentials.txt
echo "Credentials saved to ./pacific-ai-credentials.txt"

Important: Do not close your terminal until the installation is complete. The password variables are used in the Helm install commands below and are not recoverable from the credentials file alone for infrastructure passwords.

---

Define common Helm flags

All three install variants share the same set of credential flags. Define them once as a bash array so each mode command stays concise.

HELM_COMMON=(
  --namespace "$NAMESPACE"
  --create-namespace
  --timeout 15m
  --set "keycloak.auth.adminPassword=${KC_ADMIN_PW}"
  --set "keycloakconfig.config.users.passwords.admin=${KC_USER_ADMIN}"
  --set "keycloakconfig.config.users.passwords.member=${KC_USER_MEMBER}"
  --set "keycloakconfig.config.users.passwords.governance_officer=${KC_USER_GOV}"
  --set "keycloakconfig.config.users.passwords.policy_manager=${KC_USER_POLICY}"
  --set "keycloakconfig.config.users.passwords.vendor_manager=${KC_USER_VENDOR}"
  --set "keycloakconfig.config.users.passwords.compliance_officer=${KC_USER_COMPLIANCE}"
  --set "keycloakconfig.config.users.passwords.risk_manager=${KC_USER_RISK}"
  --set "keycloakconfig.config.users.passwords.developer=${KC_USER_DEVELOPER}"
  --set "keycloak.config.clientSecret=${KC_CLIENT_SEC}"
  --set "keycloak.postgresql.auth.password=${KC_PG_PW}"
  --set "keycloak.postgresql.auth.postgresPassword=${KC_PG_ROOT_PW}"
  --set "postgresql.auth.password=${PG_PW}"
  --set "redis.auth.existingSecretPassword=${REDIS_PW}"
  --set "mongodb.auth.rootPassword=${MONGO_ROOT_PW}"
  --set "mongodb.auth.password=${MONGO_PW}"
  --set "mongodb.persistence.storageClass=gp2"
  --set "llm-gateway.masterkey=${LLM_GW_KEY}"
  --set "llm-gateway.redis.auth.existingSecretPassword=${LLM_GW_REDIS_PW}"
  --set "pacific.configuration.internalApiKey=${INTERNAL_API_KEY}"
)

Then run one of the three helm install blocks below based on your chosen TLS mode.

---

Run Helm install : HTTP only

Leave BASE_URL empty to auto-detect the LoadBalancer hostname after install, or set it to your known URL.

BASE_URL=""   # leave empty to auto-detect, or set e.g. http://pacific-ai.mycompany.com

helm install pacific-ai "$CHART_DIR" "${HELM_COMMON[@]}" \
  --set "global.baseUrl=${BASE_URL:-http://pending}"

---

Run Helm install : Traefik TLS

Set your domain, pick one certificate sub-option, then run the install command.

TLS_DOMAIN="pacific-ai.mycompany.com"   # replace with your FQDN
BASE_URL="https://${TLS_DOMAIN}"

TLS_ARGS=(
  --set "ingress.tls.enabled=true"
  --set "ingress.tls.domain=${TLS_DOMAIN}"
  --set "ingress.tls.mode=traefik"
)

Sub-option A : PEM certificate files:

TLS_CERT_FILE="/path/to/cert.pem"
TLS_KEY_FILE="/path/to/key.pem"

TLS_ARGS+=(
  --set "ingress.tls.certificate.source=pem"
  --set-file "ingress.tls.certificate.pem.cert=${TLS_CERT_FILE}"
  --set-file "ingress.tls.certificate.pem.key=${TLS_KEY_FILE}"
)

Sub-option B : Existing Kubernetes TLS Secret:

TLS_SECRET_NAME="my-tls-secret"   # must already exist in $NAMESPACE

TLS_ARGS+=(
  --set "ingress.tls.certificate.source=secret"
  --set "ingress.tls.certificate.secret.name=${TLS_SECRET_NAME}"
)

Sub-option C : cert-manager + Let’s Encrypt:

ACME_EMAIL="admin@mycompany.com"
CLUSTER_ISSUER="letsencrypt-prod"

TLS_ARGS+=(
  --set "certManager.enabled=true"
  --set "ingress.tls.certificate.source=cert-manager"
  --set "ingress.tls.certificate.certManager.issuerRef.name=${CLUSTER_ISSUER}"
  --set "ingress.tls.certificate.certManager.issuerRef.kind=ClusterIssuer"
  --set "certManager.letsEncrypt.enabled=true"
  --set "certManager.letsEncrypt.email=${ACME_EMAIL}"
  --set "certManager.letsEncrypt.issuerName=${CLUSTER_ISSUER}"
  --set "certManager.letsEncrypt.server=https://acme-v02.api.letsencrypt.org/directory"
)

Run install (same command for all three sub-options):

helm install pacific-ai "$CHART_DIR" "${HELM_COMMON[@]}" \
  "${TLS_ARGS[@]}" \
  --set "global.baseUrl=${BASE_URL}"

---

Run Helm install : AWS ALB

Complete Step 7 first.

TLS_DOMAIN="pacific-ai.mycompany.com"                               # replace
ACM_CERT_ARN="arn:aws:acm:${REGION}:123456789012:certificate/..."  # replace
BASE_URL="https://${TLS_DOMAIN}"

helm install pacific-ai "$CHART_DIR" "${HELM_COMMON[@]}" \
  --set "global.baseUrl=${BASE_URL}" \
  --set "ingress.tls.enabled=true" \
  --set "ingress.tls.domain=${TLS_DOMAIN}" \
  --set "ingress.tls.mode=external" \
  --set "ingress.tls.certificate.source=external" \
  --set "global.enableProxySSL=true" \
  --set "traefik.service.type=ClusterIP" \
  --set "traefik.service.annotations=null"

Then provision the ALB Ingress:

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pacific-ai-alb
  namespace: ${NAMESPACE}
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: ${ACM_CERT_ARN}
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  ingressClassName: alb
  rules:
    - host: ${TLS_DOMAIN}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pacific-ai-traefik
                port:
                  number: 80
EOF

Wait for the ALB to receive a hostname (up to 10 minutes):

echo "Waiting for ALB hostname..."
until kubectl get ingress pacific-ai-alb -n "$NAMESPACE" \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' 2>/dev/null | grep -q .; do
  sleep 10
done

ALB_HOSTNAME=$(kubectl get ingress pacific-ai-alb -n "$NAMESPACE" \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "ALB hostname: $ALB_HOSTNAME"
echo "Create a DNS CNAME record:"
echo "  ${TLS_DOMAIN}  CNAME  ${ALB_HOSTNAME}"

---

11. Step 9 : Attach the Marketplace IAM Service Account

This step creates an IAM role with the required AWS Marketplace policies and attaches it to the application’s Kubernetes service account via IRSA. Run this after helm install completes : Helm must have created the pacific-ai-pacific-ai service account first.

eksctl create iamserviceaccount \
  --name pacific-ai-pacific-ai \
  --namespace "$NAMESPACE" \
  --cluster "$CLUSTER_NAME" \
  --region "$REGION" \
  --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess \
  --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy \
  --approve \
  --override-existing-serviceaccounts

Restart the application pods so they pick up the new IRSA annotation:

kubectl rollout restart deployment -n "$NAMESPACE"
kubectl rollout status deployment pacific-ai -n "$NAMESPACE" --timeout=3m

Verify the annotation was applied:

kubectl get serviceaccount pacific-ai-pacific-ai -n "$NAMESPACE" \
  -o jsonpath='{.metadata.annotations}' | python3 -m json.tool

You should see "eks.amazonaws.com/role-arn" with a valid IAM role ARN.

---

12. Step 10 : Determine the Application URL

HTTP-only mode (auto-detect)

If you left BASE_URL empty during Helm install, detect the LoadBalancer hostname and update the release:

echo "Waiting for LoadBalancer hostname (up to 10 min)..."
until kubectl get svc pacific-ai-traefik -n "$NAMESPACE" \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' 2>/dev/null | grep -q .; do
  sleep 10
done

LB_HOSTNAME=$(kubectl get svc pacific-ai-traefik -n "$NAMESPACE" \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
BASE_URL="http://${LB_HOSTNAME}"

echo "LoadBalancer URL: $BASE_URL"

# Update the Helm release with the real URL
helm upgrade pacific-ai "$CHART_DIR" \
  --namespace "$NAMESPACE" \
  --reuse-values \
  --set "global.baseUrl=${BASE_URL}"

# Restart Keycloak and the app so they pick up the new URL
kubectl rollout restart statefulset pacific-ai-keycloak -n "$NAMESPACE"
kubectl rollout restart deployment pacific-ai -n "$NAMESPACE"
kubectl rollout status statefulset pacific-ai-keycloak -n "$NAMESPACE" --timeout=3m
kubectl rollout status deployment pacific-ai -n "$NAMESPACE" --timeout=3m

# Append the URL to the credentials file
echo "" >> ./pacific-ai-credentials.txt
echo "  URL: ${BASE_URL}" >> ./pacific-ai-credentials.txt
echo "URL appended to ./pacific-ai-credentials.txt"

Traefik TLS mode

Get the LoadBalancer hostname to create your DNS CNAME:

LB_HOSTNAME=$(kubectl get svc pacific-ai-traefik -n "$NAMESPACE" \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "Create a DNS CNAME record:"
echo "  ${TLS_DOMAIN}  CNAME  ${LB_HOSTNAME}"

ALB mode

The DNS CNAME was printed at the end of the ALB provisioning step above.

---

13. After Installation

Save your credentials

./pacific-ai-credentials.txt is the only record of the auto-generated passwords. Copy it to a password manager or secrets vault immediately, then delete it:

cat ./pacific-ai-credentials.txt
rm ./pacific-ai-credentials.txt

Verify the deployment

# All pods should be Running
kubectl get pods -n "$NAMESPACE"

# Check Helm release status
helm list -n "$NAMESPACE"

# Watch pods in real time
kubectl get pods -n "$NAMESPACE" -w

First login

Important: Your initial login credentials were auto-generated during installation and saved to ./pacific-ai-credentials.txt. This file is the only record of those passwords. Keep it in a secure location (e.g. a password manager or secrets vault) and delete it from disk as soon as you have saved its contents elsewhere.

1. Open the application URL in a browser.
2. Retrieve the admin password from ./pacific-ai-credentials.txt.
3. Log in with username admin and that password.
4. You will be prompted to change your password immediately after the first login. Enter a new password and confirm it.
5. Save the new password in your secrets vault. If ./pacific-ai-credentials.txt has not yet been deleted, update or delete it now — the initial password is no longer valid.
6. Complete the onboarding wizard.

---

14. Sensitive Information : Locations and Handling

Location	Contents	Action required
./pacific-ai-credentials.txt on the installer machine	8 application user passwords, Keycloak admin password	Copy to a vault, then rm ./pacific-ai-credentials.txt
Kubernetes Secrets in <namespace>	PostgreSQL, Redis, MongoDB, OAuth2 client secret, LLM gateway key, internal API key	Remain in-cluster; access with kubectl get secrets -n <namespace>
eks.amazonaws.com/role-arn annotation on the pacific-ai-pacific-ai ServiceAccount	IAM role ARN for IRSA	Not sensitive; identifies which role pods can assume
AWS CloudFormation outputs (eksctl stacks)	IRSA role ARNs, OIDC provider URL	Visible in the CloudFormation console; not sensitive

No AWS credentials (Access Key ID or Secret Access Key) are stored anywhere in the cluster. The application uses IRSA exclusively for all AWS API calls at runtime.

Credentials file

# View the generated credentials
cat ./pacific-ai-credentials.txt

# Delete after saving to a vault
rm ./pacific-ai-credentials.txt

The file is created with permissions 600 (owner read/write only). It is never transmitted over the network. If lost, passwords must be rotated individually (see section 16).

Kubernetes Secrets

# List all secrets in the namespace
kubectl get secrets -n <namespace>

# Decode a specific value (example: PostgreSQL password)
kubectl get secret pacific-ai-postgresql -n <namespace> \
  -o jsonpath='{.data.password}' | base64 -d

Kubernetes Secrets are base64-encoded at rest by default. To enable KMS envelope encryption, see section 15.

---

15. Data Encryption

Encryption at rest

Amazon EFS: Created with AES-256 server-side encryption (--encrypted), managed by AWS KMS using the default EFS-managed key (aws/elasticfilesystem). To use a customer-managed key, create the EFS filesystem manually before Step 5 with --kms-key-id <key-arn>, then skip steps 7.3–7.4 and use your existing EFS_ID.

Amazon EBS (MongoDB volumes): EBS encryption is not enabled by the installer. To encrypt all new EBS volumes in the region:

1. Open the EC2 console.
2. Choose Settings → Data Encryption → Manage.
3. Enable Always encrypt new EBS volumes and select your KMS key.

Volumes created before enabling this setting are not retroactively encrypted. To encrypt an existing volume, take a snapshot, copy it with encryption enabled, and restore it.

Kubernetes Secrets (etcd): Amazon EKS encrypts etcd at rest using AES-256. To add envelope encryption with a customer-managed KMS key:

aws eks associate-encryption-config \
  --cluster-name <cluster-name> \
  --region <region> \
  --encryption-config '[{
    "resources": ["secrets"],
    "provider": {"keyArn": "arn:aws:kms:<region>:<account-id>:key/<key-id>"}
  }]'

This operation is irreversible. Ensure the KMS key policy grants the EKS service principal kms:GenerateDataKey and kms:Decrypt.

Encryption in transit

Kubernetes control plane: TLS 1.2+ between kubectl, the EKS API server, and kubelets : managed by Amazon EKS.

Application traffic: HTTP by default. Enable HTTPS by choosing Traefik TLS or AWS ALB mode during Helm install (Step 8). When TLS is enabled, Pacific AI enforces HTTPS-only access via an HTTP-to-HTTPS redirect.

EFS: NFS over TCP/2049, accessible only within the cluster VPC.

---

16. Credential and Key Rotation

Application user passwords

Managed by Keycloak:

1. Log in to Pacific AI as admin.
2. Navigate to Settings → Users.
3. Select the user, choose Reset Password, and confirm.

Keycloak admin password

helm upgrade pacific-ai <chart-dir> \
  --namespace <namespace> \
  --reuse-values \
  --set keycloak.auth.adminPassword=<new-password>

kubectl rollout restart statefulset pacific-ai-keycloak -n <namespace>
kubectl rollout status statefulset pacific-ai-keycloak -n <namespace> --timeout=3m

Service-to-service credentials (PostgreSQL, Redis, MongoDB)

# 1. Generate a new password
NEW_PW=$(openssl rand -hex 16)

# 2. Update the Kubernetes Secret
kubectl patch secret <secret-name> -n <namespace> \
  --type='json' \
  -p="[{\"op\":\"replace\",\"path\":\"/data/<key>\",\"value\":\"$(echo -n "$NEW_PW" | base64)\"}]"

# 3. Restart affected workloads
kubectl rollout restart deployment -n <namespace>
kubectl rollout restart statefulset -n <namespace>

IRSA tokens

IRSA tokens are short-lived and rotated automatically by EKS (default TTL: 24 hours). No manual rotation is required. To rotate the underlying IAM role, re-run the eksctl create iamserviceaccount command from Step 9 and restart deployments.

TLS certificates

Traefik PEM mode:

kubectl create secret tls <secret-name> \
  --cert=<new-cert.pem> --key=<new-key.pem> \
  --namespace <namespace> --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deployment pacific-ai-traefik -n <namespace>

cert-manager + Let’s Encrypt: Renewed automatically 30 days before expiry. Force a renewal:

kubectl delete certificate <cert-name> -n <namespace>
# cert-manager re-issues within minutes

AWS ALB (ACM): ACM certificates renew automatically. To swap the certificate ARN:

aws elbv2 modify-listener \
  --listener-arn <listener-arn> \
  --certificates CertificateArn=<new-acm-cert-arn>

---

17. Monitoring Application Health

1 : Verify all pods are running

kubectl get pods -n <namespace>

All pods should show STATUS: Running with all containers ready (e.g. 1/1). Investigate failures:

kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous

2 : Verify the application responds

Open the application URL in a browser : you should see the Pacific AI login page. If not:

kubectl get svc -n <namespace> pacific-ai-traefik

EXTERNAL-IP should show a hostname. If <pending>, the load balancer is still provisioning (allow 3–5 min).

3 : Verify Keycloak is healthy

kubectl get pods -n <namespace> -l app.kubernetes.io/name=keycloak
kubectl logs -n <namespace> -l app.kubernetes.io/name=keycloak --tail=50

Keycloak must be Running for users to log in.

4 : Verify persistent storage is bound

kubectl get pvc -n <namespace>

All PersistentVolumeClaims should show STATUS: Bound. A Pending PVC indicates a storage driver issue:

kubectl get pods -n kube-system -l app=efs-csi-controller

5 : Verify the IRSA role is attached

kubectl get serviceaccount pacific-ai-pacific-ai -n <namespace> \
  -o jsonpath='{.metadata.annotations}' | python3 -m json.tool

Must contain "eks.amazonaws.com/role-arn". If missing, re-run Step 9.

6 : Check the Helm release status

helm list -n <namespace>
helm status pacific-ai -n <namespace>

STATUS should be deployed. Review history on failure:

helm history pacific-ai -n <namespace>

---

18. TLS/SSL Reference

Mode	Who terminates TLS	Certificate source	DNS required
No TLS (HTTP)	None	None	No (URL auto-detected)
Traefik : PEM files	Traefik (in-cluster)	Your cert/key files	Yes
Traefik : existing Secret	Traefik (in-cluster)	K8s TLS Secret	Yes
Traefik : cert-manager	Traefik (in-cluster)	Let’s Encrypt (auto-renewed)	Yes (public DNS, port 80 open)
AWS ALB	ALB (in front of cluster)	ACM certificate (pre-validated)	Yes

Which mode to choose:

• No TLS : internal networks, testing, or when TLS is handled upstream
• Traefik + cert-manager : simplest production setup; certificates are free and renew automatically
• Traefik + PEM : when you manage your own certificates or use a private CA
• AWS ALB : when you need WAF, centralised certificate management, or TLS termination outside the cluster

---

19. How the Application Accesses AWS

Pacific AI uses IAM Roles for Service Accounts (IRSA) for all AWS API calls at runtime. No static credentials are stored in the cluster.

1. OIDC provider : created during cluster setup (iam.withOIDC: true). Links the Kubernetes API server to AWS IAM.
2. IAM role with trust policy : created by eksctl create iamserviceaccount in Step 9. The trust policy is scoped to the specific cluster OIDC provider, namespace, and service account name.
3. Token injection : the EKS Pod Identity webhook mounts a short-lived, auto-rotating projected volume token into each pod using the annotated service account. No action is required by the application.
4. SDK credential chain : AWS SDKs in the application discover the projected token automatically. No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY environment variables are present in any pod.

IAM roles created during installation:

Service account	Namespace	IAM policies	Purpose
ebs-csi-controller-sa	kube-system	AmazonEBSCSIDriverPolicy	EBS volume provisioning
aws-load-balancer-controller	kube-system	Custom LBC policy	ALB/NLB provisioning (ALB mode only)
pacific-ai-pacific-ai	your namespace	Marketplace Metering, License Manager	Application AWS API calls

---

20. Troubleshooting

Helm install times out

kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>

Common causes:

• Pending pods : insufficient node capacity. Check with kubectl describe nodes.
• ImagePullBackOff : verify ecr:BatchGetImage permission and active Marketplace subscription.
• CrashLoopBackOff : application startup error; check pod logs.

Resume after resolving the underlying issue:

helm upgrade pacific-ai "$CHART_DIR" \
  --namespace <namespace> --reuse-values --timeout 15m

AWS credentials not configured

error: You must be logged in to the server (Unauthorized)

or

Unable to locate credentials.

Run aws sts get-caller-identity to verify credentials are active. See section 2.3.

Cluster creation fails

Check the CloudFormation stack named eksctl-<cluster-name>-cluster in the AWS Console for error events. Common causes: service quota limits (VPC, EIP, EC2 instances) or insufficient IAM permissions.

Clean up a partially-created cluster:

eksctl delete cluster --name "$CLUSTER_NAME" --region "$REGION"

EFS mount targets already exist

An error occurred (MountTargetConflict): ...

This warning is harmless if you are re-running the setup. The || in the mount target loop skips existing targets automatically.

LoadBalancer hostname not appearing

kubectl get svc -n <namespace> pacific-ai-traefik

If EXTERNAL-IP is <pending> after 5 minutes, check the EC2 console under Load Balancers. Once the hostname appears, update the Helm release:

helm upgrade pacific-ai "$CHART_DIR" \
  --namespace <namespace> --reuse-values \
  --set "global.baseUrl=http://<hostname>"

kubectl rollout restart statefulset pacific-ai-keycloak -n <namespace>
kubectl rollout restart deployment pacific-ai -n <namespace>

IRSA annotation missing from service account

Re-run the eksctl create iamserviceaccount command from Step 9, then restart the deployments.

OIDC provider association fails

Ensure your IAM identity has iam:CreateOpenIDConnectProvider and iam:GetOpenIDConnectProvider. These are in the custom policy in section 2.2.

---

21. Uninstalling

Run these commands to remove all resources created during installation.

1 : Uninstall the Helm release

helm uninstall pacific-ai -n <namespace>

2 : Delete PersistentVolumeClaims

kubectl delete pvc --all -n <namespace>

3 : Delete the namespace

kubectl delete namespace <namespace>

4 : Remove EFS resources

# Delete mount targets
MT_IDS=$(aws efs describe-mount-targets \
  --file-system-id <efs-id> --region <region> \
  --query 'MountTargets[].MountTargetId' --output text)

for MT_ID in $MT_IDS; do
  aws efs delete-mount-target --mount-target-id "$MT_ID" --region <region>
done

# Wait for deletion, then delete the filesystem
aws efs delete-file-system --file-system-id <efs-id> --region <region>

# Delete the security group
aws ec2 delete-security-group --group-id <efs-sg-id> --region <region>

5 : Remove IRSA service accounts and IAM roles

eksctl delete iamserviceaccount \
  --name pacific-ai-pacific-ai \
  --namespace <namespace> \
  --cluster <cluster-name> \
  --region <region>

6 : (Optional) Delete the EKS cluster

Warning: This is irreversible and removes all data in the cluster’s persistent volumes. Back up your data first.

eksctl delete cluster --name <cluster-name> --region <region>

This takes approximately 15 minutes and removes the cluster, VPC, node groups, and all associated CloudFormation stacks.