Gemini 1.0 Flash: Complete Benchmark Performance Guide

google 1.0-flash 2024-02-15 Proprietary

In-depth analysis of Gemini 1.0 Flash performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorgoogle
Version1.0-flash
Release Date2024-02-15
Context Window32768 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://ai.google.dev/gemini-api/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU78.7%2024-02-155-shotview
HUMANEVAL65.7pass@12024-02-15—view
GSM8K86.2%2024-02-150-shot CoTview
MATH39.9%2024-02-150-shot CoTview
BBH74.8%2024-02-153-shot CoTview
GPQA37.7%2024-02-150-shotview
IFEVAL71.8%2024-02-15prompt_strictview
ARC92.6%2024-02-15challengeview
MUSR55.9%2024-02-150-shotview
WINOGRANDE80.1%2024-02-150-shotview

Pricing

TierPriceCurrency
Input$0.075 / MtokUSD
Output$0.3 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://ai.google.dev/pricing · as of 2024-02-15

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Gemini 1.0 Flash

Tổng quan mô hình

Google Gemini 1.0 Flash 经济型, 32K 上下文, 速度快成本低, 适合实时多模态应用。

Thông số kỹ thuật cốt lõi

Nhà cung cấpPhiên bảnNgày phát hànhCửa sổ ngữ cảnhPhương thức đầu vàoPhương thức đầu raGiấy phép
Google1.0-flash2024-02-1532Ktext, imagetextProprietary

Hiệu suất benchmark

BenchmarkĐiểmĐơn vịGhi chú
MMLU (Massive Multitask Language Understanding)78.7%5-shot
HumanEval65.7pass@1—
GSM8K (Grade School Math 8K)86.2%0-shot CoT
MATH39.9%0-shot CoT
BBH (BIG-Bench Hard)74.8%3-shot CoT
GPQA37.7%0-shot
IFEval71.8%prompt_strict
ARC92.6%challenge
MUSR55.9%0-shot
WinoGrande80.1%0-shot

Giá

Đầu vàoĐầu raĐọc bộ nhớ đệmGhi bộ nhớ đệm
————

mỗi triệu token

Điểm mạnh

  • GSM8K 86.2, robust math reasoning.

Điểm yếu

  • 闭源专有模型，不支持自托管。

Trường hợp sử dụng

  • 通用对话与问答

Tham chiếu