Eval Harness First

What the skill covers

The sections of its instructions, in order.

1. 01 The Gate
2. 02 Building Goldens
3. 03 Graders
4. 04 Judge Calibration Is a Prerequisite
5. 05 The Baseline
6. 06 Directory Contract
7. Phase 0 Exit Checklist
8. 07 Related Skills
9. 08 References

Reference files

2 files Phở can open while it uses the skill.

• references/grader-templates.md
• references/judge-calibration.md

More in LLM apps and fine tuning

• Embedding Strategies

  Select and optimize embedding models for semantic search and RAG applications. Use when choosing embedding models, implementing chunking strategies, or optimizing embedding…

• Hybrid Search Implementation

  Combine vector and keyword search for improved retrieval. Use when implementing RAG systems, building search engines, or when neither approach alone provides sufficient recall.

• LangChain & LangGraph Architecture

  Design LLM applications using LangChain 1.x and LangGraph for agents, memory, and tool integration. Use when building LangChain applications, implementing AI agents, or creating…

• LLM Evaluation

  Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI…

• RAG Implementation

  Build Retrieval-Augmented Generation (RAG) systems for LLM applications with vector databases and semantic search. Use when implementing knowledge-grounded AI, building document…

• Similarity Search Patterns

  Implement efficient similarity search with vector databases. Use when building semantic search, implementing nearest neighbor queries, or optimizing retrieval performance.

• Vector Index Tuning

  Optimize vector index performance for latency, recall, and memory. Use when tuning HNSW parameters, selecting quantization strategies, or scaling vector search infrastructure.

• Checkpoint Promotion

  Gate fine-tuned checkpoints with drift budgets, paired comparison, and forgetting checks before promotion. Use after a training run produces a checkpoint, when deciding whether a…

• Dataset Curation

  Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring…

• Fine-Tuning Method Selection

  Decide whether to fine-tune at all, and route to the right method (SFT, DPO/ORPO/KTO, GRPO/RLVR, continued pretraining) and base model. Use when starting any fine-tuning effort…

• GRPO & RLVR Training

  Train reasoning and verifiable-task behavior with GRPO and reinforcement learning from verifiable rewards (RLVR). Use when task success is algorithmically checkable (math, code…

• LoRA & QLoRA Recipes

  Configure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing…

• Preference Optimization

  Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between…

• Quantized Export

  Export a promoted fine-tuned model in the right deployment format — merged safetensors, LoRA-only, GGUF with imatrix, or FP8. Use after a checkpoint passes promotion, when…

• Trace To Training Data

  Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when…

• Vision-Language SFT

  Fine-tune vision-language models (VLMs) with supervised learning on image+text data. Use when adapting a VLM to a visual domain or task, configuring frozen-vision-tower LoRA, or…

• ML Pipeline Workflow

  Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps…

• Recsys Pipeline Architect

  Design composable recommendation, ranking, and feed pipelines using the six-stage Source→Hydrator→Filter→Scorer→Selector→SideEffect framework popularized by xAI's open-sourced X…

Similar ai & machine learning skills

• Hf cli
• SageMaker Deployment Planner
• Hf mem
• HuggingFace Best Model Finder
• Hugging Face Dataset Viewer
• Hugging Face Local Models