StarChat2 15B v0.1: Complete Benchmark Performance Guide

other starchat2-15b-v0.1 2024-07-05 BigCode Open Model License

In-depth analysis of StarChat2 15B v0.1 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionstarchat2-15b-v0.1
Release Date2024-07-05
Context Window8192 tokens
Input Modalitiestext
Output Modalitiestext
LicenseBigCode Open Model License
Documentationhttps://huggingface.co/models

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU74.9%2024-07-055-shotview
HUMANEVAL69pass@12024-07-05—view
GSM8K50.7%2024-07-050-shot CoTview
MATH45.3%2024-07-050-shot CoTview
BBH70.6%2024-07-053-shot CoTview
GPQA25.9%2024-07-050-shotview
IFEVAL63.3%2024-07-05prompt_strictview
ARC91.7%2024-07-05challengeview
MUSR44.4%2024-07-050-shotview
WINOGRANDE75%2024-07-050-shotview

Pricing

TierPriceCurrency
Input$0.3 / MtokUSD
Output$0.3 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2024-07-05

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

StarChat2 15B v0.1

Tổng quan mô hình

HuggingFace StarChat2 15B 代码对话模型, 8K 上下文, 基于 StarCoder2 微调, 支持多语言代码生成。

Thông số kỹ thuật cốt lõi

Nhà cung cấpPhiên bảnNgày phát hànhCửa sổ ngữ cảnhPhương thức đầu vàoPhương thức đầu raGiấy phép
Otherstarchat2-15b-v0.12024-07-058KtexttextBigCode Open Model License

Hiệu suất benchmark

BenchmarkĐiểmĐơn vịGhi chú
MMLU (Massive Multitask Language Understanding)74.9%5-shot
HumanEval69.0pass@1—
GSM8K (Grade School Math 8K)50.7%0-shot CoT
MATH45.3%0-shot CoT
BBH (BIG-Bench Hard)70.6%3-shot CoT
GPQA25.9%0-shot
IFEval63.3%prompt_strict
ARC91.7%challenge
MUSR44.4%0-shot
WinoGrande75.0%0-shot

Giá

Đầu vàoĐầu raĐọc bộ nhớ đệmGhi bộ nhớ đệm
————

mỗi triệu token

Điểm mạnh

  • 可靠的通用模型。

Điểm yếu

  • 闭源专有模型，不支持自托管。
  • 上下文窗口 8K 偏小。

Trường hợp sử dụng

  • 代码生成与调试

Tham chiếu