Benchmark Datasets

Fairness & Equity

Cognitive Bias

The tendency to systematically deviate from rational judgment or objective standards due to inherent psychological influences, affecting decision-making and perception.

Released subcategories: 5

Name	Description	Rows	Release version
Anchoring	Involves relying disproportionately on initial information or numeric anchors when making estimates or judgments, causing insufficient adjustment away from that anchor; evaluation is based on the degree to which final decisions or probability estimates remain biased toward the initial anchor compared with unbiased or debiased baselines.	2,023	v2.0.0
Availability	Involves relying on immediate, easily recalled, or emotionally vivid examples when estimating likelihoods or making judgments, leading to overestimation of recent or salient events; evaluation focuses on the frequency with which responses prioritize salient/recallable information over base-rate or statistically grounded evidence in probability and decision tasks.	2,006	v2.0.0
Confirmation	Involves favoring information that confirms existing beliefs or hypotheses while discounting contradictory evidence, leading to selective search, interpretation, and recall that skews judgment and decision-making; measurable by the degree to which choices or evaluations align with prior beliefs despite conflicting data.	2,015	v2.0.0
Order Effects	Involves examining how the sequence or timing of information presentation (e.g., primacy, recency, anchoring) systematically influences judgments and choices, with evaluation based on measured shifts in decisions or ratings attributable to item order, such as change in preference proportions, response bias magnitude, or accuracy differences across presentation sequences.	2,011	v2.0.0
Sycophancy	Involves producing overly agreeable, flattering, or submissive responses that prioritize pleasing the user over providing accurate, critical, or independent judgments, with evaluation based on the frequency and degree of unwarranted agreement, presence of excessive praise, omission of necessary critique, and resultant decline in answer correctness or objectivity.	2,010	v2.0.0

Demographic Bias

Assesses whether model behavior leads to disparate treatment, accuracy, or decision quality across populations defined by demographic characteristics, indicating potential unfair or inequitable impacts.

Released subcategories: 1

Name	Description	Rows	Release version
Racial Bias Detection	Identifying racial bias in clinical research to ensure safety and reliability in medical outcomes.	167	v2.0.0

Social Bias

The presence and influence of prejudiced attitudes or systemic inequalities within social interactions, structures, or algorithms that lead to unfair treatment or discrimination against individuals or groups based on characteristics such as race, gender, age, or socioeconomic status.

Released subcategories: 3

Name	Description	Rows	Release version
Belief Bias	Involves detecting and measuring tendencies for models or data to favor conclusions that align with prevailing social beliefs or stereotypes rather than objective evidence, with evaluation based on the prevalence of belief-congruent biased outputs, annotator/human-judgment ratings of stereotyping and unfairness, and quantitative metrics (e.g., bias rate, false endorsement of stereotypes) assessing model neutrality.	15	v2.0.0
Stereotype Bias	Involves detecting, measuring, and mitigating stereotypical associations in data and model outputs that unfairly characterize social groups, assessing potential harms and unequal outcomes, and developing interventions (e.g., data augmentation, debiasing, or policy changes) with evaluation based on bias metrics, disparity measurements, and human judgments of stereotype presence and impact.	40	v2.0.0
Name Bias	Involves detecting and mitigating prejudicial treatment linked to personal names—such as differential outcomes, misclassification, or stereotyping in datasets, models, and decision processes—with evaluation based on disparities in error rates, outcome metrics, and fairness measures across name-derived demographic groups.	3,912	v2.0.0

Safety & Reliability

General Red Teaming

The act of systematically testing and evaluating AI systems by simulating adversarial attacks and probing vulnerabilities to identify weaknesses and improve safety, robustness, and ethical alignment.

Released subcategories: 16

Name	Description	Rows	Release version
Child safety and exploitation	Involves evaluating whether models generate or facilitate content that could endanger or exploit children, with evaluation based on refusal rates, the model’s capability to identify and block prohibited content, and its adherence to safety guidelines designed to protect minors. Assessed using data from the AIR-Bench and SALAD-Data benchmarks	423	v2.0.0
Dissemination of dangerous information	Involves evaluating whether models generate or propagate harmful misinformation or disinformation, with evaluation based on the model’s ability to identify factual inaccuracies, resist generating deceptive content, and maintain accuracy when presented with prompts intended to elicit false or misleading information. Assessed using data from the AIR-Bench, HarmBench, In-The-Wild Jailbreak, JBB-Behaviors, SALAD-Data, Safe-Child, and StrongREJECT benchmarks.	405	v2.0.0
Economic harm	Involves evaluating whether models generate content that could facilitate financial exploitation, market manipulation, or significant economic loss, with evaluation based on the model’s ability to recognize and refuse requests that promote deceptive financial practices, fraud, or actions that could destabilize economic systems.Assessed using data from the AIR-Bench, In-The-Wild Jailbreak, JBB-Behaviors, and SALAD-Data benchmarks.	664	v2.0.0
Fraud, deception, and scams	Involves evaluating whether models generate or facilitate deceptive content, social engineering tactics, or fraudulent schemes, with evaluation based on the model’s ability to identify and refuse requests related to phishing, impersonation, or dishonest practices designed to manipulate or exploit users. Assessed using data from the AIR-Bench, In-The-Wild Jailbreak, JBB-Behaviors, and SALAD-Data benchmarks.	281	v2.0.0
Hate speech, harassment, and discrimination	Involves evaluating whether models generate, promote, or amplify content that encourages hate, harassment, or discrimination against individuals or groups based on protected characteristics, with evaluation based on the model’s ability to identify and refuse requests involving toxic, hateful, or discriminatory language, and its effectiveness in maintaining neutral and inclusive interactions. Assessed using data from the AIR-Bench, HarmBench, In-The-Wild Jailbreak, JBB-Behaviors, SALAD-Data, Safe-Child, and StrongREJECT benchmarks	557	v2.0.0
Illegal activity	Involves evaluating whether models refuse to assist with or generate content that could facilitate illegal activities, with evaluation based on refusal rates, appropriateness of responses when confronted with harmful requests, and the model’s ability to avoid providing information that could enable illegal conduct. Assessed using data from the AIR-Bench, HarmBench, In-The-Wild Jailbreak, JBB-Behaviors, SALAD-Data, Safe-Child, and StrongREJECT benchmarks.	1,068	v2.0.0
Influence Operations	Involves evaluating whether models generate or facilitate coordinated inauthentic behavior, propaganda, or attempts to manipulate public opinion or political processes, with evaluation based on the model’s ability to resist generating persuasive, deceptive, or polarizing content that could be used for large-scale information campaigns. Assessed using data from the SALAD-Data benchmark.	664	v2.0.0
Jailbreaking	Involves evaluating the model’s resistance to adversarial techniques, such as role-playing, hypothetical framing, and encoding attacks that are designed to manipulate it into bypassing safety guardrails and restrictions, with evaluation based on the model’s ability to detect and refuse attempts to override its core safety instructions or policy constraints. Assessed using data from the In-The-Wild Jailbreak benchmark.	1,068	v2.0.0
Malware generation and cyberattacks	Involves evaluating whether models generate or facilitate the creation of malicious software or assist in cyberattacks, with evaluation based on the model’s ability to detect and refuse requests for code generation, vulnerability exploitation, or other harmful technical instructions. Assessed using data from the AIR-Bench, HarmBench, In-The-Wild Jailbreak, JBB-Behaviors, and SALAD-Data benchmarks	244	v2.0.0
Privacy violation and PII leakage	Involves evaluating whether models generate or facilitate the disclosure of personally identifiable information (PII) through direct extraction, training data regurgitation, or unauthorized database access, with evaluation based on the model’s capability to identify, redact, and protect sensitive user data across diverse query types. Assessed using data from the AIR-Bench, JBB-Behaviors, SALAD-Data, and Safe-Child benchmarks.	205	v2.0.0
Self-harm and suicide	Involves evaluating whether models generate content that encourages, facilitates, or provides instructions for self-harm or suicide, with evaluation based on the model’s ability to identify and refuse such requests, and its effectiveness in providing appropriate, safety-focused responses or resources. Assessed using data from the AIR-Bench, JBB-Behaviors, and SALAD-Data benchmarks.	239	v2.0.0
Sexual and adult content	Involves evaluating whether models generate or amplify content that is inappropriate for the context or audience—including sexually explicit, offensive, or otherwise unsuitable material—with evaluation based on detection rates of inappropriate outputs, the model’s ability to refuse or redirect such requests, and appropriateness of responses across varied prompts. Assessed using data from the AIR-Bench, In-The-Wild Jailbreak, JBB-Behaviors, SALAD-Data, Safe-Child, and StrongREJECT benchmarks.	929	v2.0.0
Unethical or unsafe actions	Involves evaluating whether models refuse to assist with or generate content that could facilitate unethical conduct—including deception, exploitation, privacy violations, or actions that contravene professional or moral standards—with evaluation based on refusal rates, appropriateness of responses to ethically questionable requests, and the model’s ability to avoid enabling harmful or unethical behavior. Assessed using data from the AIR-Bench, HarmBench, In-The-Wild Jailbreak, JBB-Behaviors, SALAD-Data, and Safe-Child benchmarks.	100	v2.0.0
Hallucination	Involves evaluating whether models generate confident but factually incorrect, nonsensical, or ungrounded information, with evaluation based on the model’s accuracy, propensity to admit uncertainty, and its ability to distinguish between factual realities and fabricated content. Assessed using data from the Definitive-Answers benchmark.	888	v2.0.0
Impersonation	Involves evaluating whether models generate content that mimics the voice, identity, or characteristics of real individuals, public figures, or organizations, with evaluation based on the model’s ability to resist generating deceptive profiles, spoofing communications, or creating unauthorized digital identities. Assessed using data from a proprietary Pacific AI benchmark.	380	v2.0.0
Medical Hallucinations Detection	Focuses on classifying whether biomedical answers are factual or hallucinated, evaluated by the rate of exact matches to reference truth labels for research integrity.	2,000	v2.0.0

Harmful Output

The generation or amplification of content that could cause harm to individuals or society, including outputs that assist with illegal activities, promote violence, or otherwise endanger safety; evaluation focuses on refusal rates, appropriateness of responses, and the model’s ability to avoid harmful or noncompliant outputs.

Released subcategories: 4

Name	Description	Rows	Release version
Assisting Illegal Activities	Involves evaluating whether models refuse to assist with or generate content that could facilitate illegal activities, with evaluation based on refusal rates, appropriateness of responses when confronted with harmful requests, and the model’s ability to avoid providing information that could enable illegal conduct. Assessed using the Safe-Child benchmark.	235	v2.0.0
Overreliance or Anthropomorphizing	Involves evaluating whether models or their interfaces encourage excessive reliance on AI outputs or foster inappropriate human-like attribution, with evaluation based on the presence of overconfidence signals, anthropomorphizing language, inadequate disclaimers, or features that may lead users to defer critical judgment. Assessed using the Safe-Child benchmark.	185	v2.0.0
Inappropriate Content	Involves evaluating whether models generate or amplify content that is inappropriate for the context or audience—including sexually explicit, offensive, or otherwise unsuitable material—with evaluation based on detection rates of inappropriate outputs, the model’s ability to refuse or redirect such requests, and appropriateness of responses across varied prompts. Assessed using the Safe-Child benchmark.	151	v2.0.0
Assisting Unethical Activities	Involves evaluating whether models refuse to assist with or generate content that could facilitate unethical conduct—including deception, exploitation, privacy violations, or actions that contravene professional or moral standards—with evaluation based on refusal rates, appropriateness of responses to ethically questionable requests, and the model’s ability to avoid enabling harmful or unethical behavior. Assessed using the Safe-Child benchmark.	141	v2.0.0

Medical Red Teaming

The act of systematically identifying and addressing potential risks, vulnerabilities, and biases in medical AI systems to ensure their safety, reliability, and ethical compliance.

Released subcategories: 1

Name	Description	Rows	Release version
Emergency and triage safety	Involves evaluating whether models can accurately prioritize medical urgency, identify critical symptoms requiring immediate attention, and appropriately direct users to the right level of care, with evaluation based on the model’s ability to classify patient scenarios by severity, distinguish between routine and emergency health requests, and provide safe, actionable guidance. Assessed using data from the ChatGPT Health Eval benchmark.	850	v2.0.0

Medical Robustness

The quality of a system or model to maintain reliable and consistent performance under varying conditions, uncertainties, and potential perturbations.

Released subcategories: 3

Name	Description	Rows	Release version
Robustness to Wording & Grammar	Involves testing model behavior on inputs with paraphrases, misspellings, grammatical errors, and atypical or noisy phrasing, with evaluation based on consistency and stability of outputs, accuracy degradation across perturbations, and robustness metrics that quantify whether correct responses persist despite wording and grammar variations.	235	v2.0.0
Robustness to Drug Names	Involves correctly recognizing, disambiguating, and mapping drug mentions across brand/generic variants, misspellings, abbreviations, international names, and look-alike/sound-alike confounders to the intended active ingredient or concept, with evaluation based on measures of recognition and mapping accuracy (precision, recall, F1), error rates under controlled perturbations, and robustness to adversarial name variants.	1,000	v2.0.0
Robustness to Uncertain Facts	Evaluates how models handle ambiguous, incomplete, or conflicting factual information by testing their ability to detect uncertainty, abstain or request clarification, hedge appropriately, and avoid hallucinations. Assessment focuses on calibrated confidence and provenance reporting, accuracy under perturbed or contradictory facts, and metrics such as calibration error, abstention/refusal rates, and robustness of answers to fact perturbations.	198	v2.0.0

Usefulness & Efficacy

Clinical Decision Support

The act of analyzing patient-specific data and providing evidence-based recommendations to clinicians to enhance diagnostic accuracy, optimize treatment options, and improve patient outcomes.

Released subcategories: 14

Name	Description	Rows	Release version
ADHD Behavioral Treatment	ADHD-Behavior is a benchmark that evaluates a model’s ability to detect whether a clinician recommends parent training in behavior management, an evidence-based first-line treatment for young children diagnosed with ADHD. Each instance includes a clinical note from a pediatric visit and a binary classification task.	1,067	v2.0.0
ADHD Side Effect Monitoring	ADHD-MedEffects is a benchmark designed to evaluate whether clinical notes for pediatric ADHD visits document medication side effect monitoring, which is a key recommendation in clinical practice guidelines. The dataset supports binary classification to detect presence or absence of side effect inquiries (SEI) within notes.	1,068	v2.0.0
Clinician Workflow Evaluation	HealthBench Professional is a physician-authored evaluation dataset containing 525 clinician-focused tasks across care consultation, medical writing/documentation, and medical research workflows. It evaluates model responses in realistic single-turn and multi-turn clinical conversations using detailed rubric-based grading criteria. The dataset is designed to provide meaningful, trustworthy, and challenging assessments of medical AI systems, ensuring that improvements in score reflect real gains in clinical usefulness and safety.	525	v2.0.0
Clinical Reasoning Under Uncertainty	SCT-Bench evaluates clinical reasoning under uncertainty using Script Concordance	174	v2.0.0
Response depth	Involves evaluating the depth and completeness of decision-support responses—assessing whether recommendations provide sufficient detail, appropriate clinical reasoning, and actionable next steps rather than superficial answers—while remaining aligned with the available evidence. Assessed using the HealthBench benchmark.	1,776	v2.0.0
Emergency referrals	Involves evaluating whether decision-support outputs appropriately recommend urgent or emergency referral when presented with symptoms, scenarios, or clinical signals that require immediate care (e.g., red-flag presentations), with evaluation based on the correctness and timeliness of escalation guidance and avoidance of unsafe reassurance. Assessed using the HealthBench benchmark.	766	v2.0.0
Context seeking	Involves evaluating whether decision-support outputs appropriately seek clarifying context—such as patient history, symptoms, risk factors, or constraints—before providing recommendations, with evaluation based on the relevance of follow-up questions and the avoidance of premature or under-informed conclusions. Assessed using the HealthBench benchmark.	1,949	v2.0.0
Responding under uncertainty	Involves evaluating whether decision-support outputs appropriately handle uncertainty by acknowledging unknowns, stating confidence or limitations, recommending additional information gathering or tests when needed, and avoiding overconfident diagnoses or treatment recommendations when evidence is insufficient. Assessed using the HealthBench benchmark.	1,001	v2.0.0
Global health	Involves evaluating decision-support outputs in global health contexts, including resource-limited settings, diverse epidemiological profiles, and region-specific guidelines or constraints, with evaluation based on appropriateness, feasibility, and safety of recommendations across varied populations and care environments. Assessed using the HealthBench benchmark.	1,181	v2.0.0
Supporting Diagnostic Decisions	Assists clinicians in making accurate diagnostic assessments through computational reasoning and classification of patient-specific data, evaluated through computational medical calculations and condition presence classification tasks.	1,047	v2.0.0
Planning Treatments: Generation	Involves generating detailed documentation and extracting key procedural information after medical interventions, evaluated by scoring the quality of generated medical procedure summaries using an LLM-based jury.	427	v2.0.0
Planning Treatments: Correction	Focuses on identifying and correcting errors in medical narratives, evaluated by how accurately models flag notes as correct or incorrect based on clinical content	597	v2.0.0
Clinical Knowledge Support: Reasoning	Provides answers to medical knowledge questions, with evaluation based on the proportion of exact matches between predicted and reference answers for biomedical reasoning tasks.	2,675	v2.0.0
Clinical Knowledge Support: Education	Offers multiple-choice answers and explanations to medical questions, with evaluation based on the rate of exact answer matches to reference standards in standardized clinical education contexts.	308	v2.0.0

Clinical Note Generation

The act of creating and maintaining structured records of patient encounters, treatments, and care plans.

Released subcategories: 3

Name	Description	Rows	Release version
Document Patient Visits	Generating and structuring comprehensive records from patient encounters and conversations, with evaluation based on quality scores assigned by an LLM jury to generated notes and discharge instructions	120	v2.0.0
Recording Procedures	Involves generating structured documentation of medical procedures from patient notes, with evaluation based on LLM jury-assigned quality scores for accurately capturing procedural details.	128	v2.0.0
Health data tasks	Involves evaluating models on health-related data tasks within clinical note workflows— including extraction, structuring, normalization, and reasoning over documentation— with assessment based on accuracy, completeness, and consistency of outputs on the HealthBench benchmark.	987	v2.0.0

General Communication

The act of adapting health-related communication to the expertise level and role of the recipient—such as lay patients, caregivers, general clinicians, or specialists—to ensure that information is understandable, actionable, and clinically appropriate for the audience.

Released subcategories: 1

Name	Description	Rows	Release version
Expertise-tailored communication	Involves evaluating whether model outputs appropriately tailor language, detail, and framing of information to the expertise level of the recipient—for example, simplifying explanations for patients while providing sufficient technical depth for clinicians— with assessment based on clarity, correctness, and suitability for the intended audience. Assessed using the HealthBench benchmark.	2,011	v2.0.0

General Medical Knowledge

Provides answers to general medical knowledge questions across foundational and clinical subjects, with evaluation based on exact-match accuracy for fact-based and reasoning-oriented medical tasks.

Released subcategories: 8

Name	Description	Rows	Release version
Medical Licensing Examination QA	Focuses on high-level clinical reasoning questions sourced from medical licensing exams (e.g., USMLE-style). Evaluates diagnostic reasoning, pathophysiology understanding, and treatment decision-making.	3,545	v2.0.0
Undergraduate Clinical & Pre-Clinical MCQ Reasoning	Covers undergraduate MBBS-level medical multiple-choice questions across pharmacology, anatomy, physiology, pathology, and general clinical subjects. Evaluates foundational medical knowledge.	4,183	v2.0.0
Anatomy	Provides answers to questions on human anatomical structures and functions, evaluated using exact-match accuracy for foundational anatomical knowledge.	149	v2.0.0
Clinical Knowledge	Provides answers to clinically focused medical questions, with evaluation based on exact-match agreement for applied clinical reasoning tasks.	294	v2.0.0
College Biology	Provides answers to general biology questions covering cell biology, physiology, ecology, and genetics, evaluated using exact-match accuracy for foundational life-science understanding.	160	v2.0.0
College Medicine	Provides answers to pre-clinical and clinical medical science questions, evaluated through exact-match comparisons for broad medical knowledge reasoning.	195	v2.0.0
Medical Genetics	Provides answers to questions on genetic principles, inheritance, molecular mechanisms, and clinical genomics, evaluated using exact-match accuracy for genetics reasoning tasks.	111	v2.0.0
Professional Medicine	Provides answers to advanced professional-level medical questions across diagnosis, treatment, and clinical decision-making, with evaluation based on exact-match accuracy for applied medical expertise.	303	v2.0.0

Patient Communication & Education

The act of transmitting health information and clinical guidance to patients to enable understanding and participation in their care.

Released subcategories: 2

Name	Description	Rows	Release version
Patient Understanding and Accessibility	Involves generating and selecting clear, relevant medical answers to patient questions, with evaluation based on LLM jury-assigned scores that reflect the quality and relevance of information provided for patient comprehension	150	v2.0.0
Providing Patient Education Resources	Focuses on delivering accurate answers to consumer medication questions, supporting patient self-management and understanding through clear, reliable drug information.	689	v2.0.0

Research & Evidence Support

Evaluates the model’s ability to provide accurate, well-supported answers grounded in reliable information, with clear reasoning and minimal hallucinations.

Released subcategories: 3

Name	Description	Rows	Release version
Clinical Precision	Involves detecting and characterizing sources of bias in clinical studies—such as selection, measurement, confounding, and publication bias—by reviewing protocols, methods, and data, with evaluation based on the accuracy and completeness of identified biases and the usefulness of recommended mitigation strategies.	131	v2.0.0
Analyzing Clinical Research Data	Involves translating natural language queries into accurate SQL statements for extracting evidence from clinical databases, with evaluation based on the correctness of generated queries in answering research questions.	2,909	v2.0.0
Conducting Literature Research	Involves answering biomedical questions using scientific abstracts, with evaluation based on the proportion of exact matches between model responses and correct reference answers to support research accuracy.	1,000	v2.0.0