Code Bison: Complete Benchmark Performance Guide

google code-bison 2023-05-10 Proprietary

In-depth analysis of Code Bison performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorgoogle
Versioncode-bison
Release Date2023-05-10
Context Window8192 tokens
Input Modalitiestext
Output Modalitiestext
LicenseProprietary
Documentationhttps://ai.google.dev/gemini-api/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU65%2023-05-105-shotview
HUMANEVAL76.2pass@12023-05-10—view
GSM8K55%2023-05-100-shot CoTview
MATH34.4%2023-05-100-shot CoTview
BBH71.4%2023-05-103-shot CoTview
GPQA29.1%2023-05-100-shotview
IFEVAL59.2%2023-05-10prompt_strictview
ARC86.1%2023-05-10challengeview
MUSR49.8%2023-05-100-shotview
WINOGRANDE77.9%2023-05-100-shotview

Pricing

TierPriceCurrency
Input$0.5 / MtokUSD
Output$1 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://ai.google.dev/pricing · as of 2023-05-10

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Code Bison

Tổng quan mô hình

Google Vertex AI Code Bison 代码生成模型, 基于 PaLM 2, 8K 上下文, 支持多语言代码生成与补全。

Thông số kỹ thuật cốt lõi

Nhà cung cấpPhiên bảnNgày phát hànhCửa sổ ngữ cảnhPhương thức đầu vàoPhương thức đầu raGiấy phép
Googlecode-bison2023-05-108KtexttextProprietary

Hiệu suất benchmark

BenchmarkĐiểmĐơn vịGhi chú
MMLU (Massive Multitask Language Understanding)65.0%5-shot
HumanEval76.2pass@1—
GSM8K (Grade School Math 8K)55.0%0-shot CoT
MATH34.4%0-shot CoT
BBH (BIG-Bench Hard)71.4%3-shot CoT
GPQA29.1%0-shot
IFEval59.2%prompt_strict
ARC86.1%challenge
MUSR49.8%0-shot
WinoGrande77.9%0-shot

Giá

Đầu vàoĐầu raĐọc bộ nhớ đệmGhi bộ nhớ đệm
————

mỗi triệu token

Điểm mạnh

  • 可靠的通用模型。

Điểm yếu

  • 闭源专有模型，不支持自托管。
  • 上下文窗口 8K 偏小。

Trường hợp sử dụng

  • 代码生成与调试

Tham chiếu