Measuring
5
Evaluation
Teaching Gemma to Code
Why this handbook
How agents work
1
The agent loop
2
Tools
3
Context and tokens
4
Prompts and skills
Measuring
5
Evaluation
6
Error analysis
Making it better
7
First failing test
8
Running Gemma locally
9
Fine-tuning with LoRA
Looking back
10
Results
Appendices
Experiment log
Glossary
Table of contents
5.1
What I want to understand
5.2
What I tried
5.3
What I learned
5.4
Sources
Edit this page
Report an issue
Measuring
5
Evaluation
5
Evaluation
Note
Status: not started
5.1
What I want to understand
How does SWE-Bench-style scoring work?
Can my local score predict the Kaggle score?
How many runs before a difference is real?
5.2
What I tried
5.3
What I learned
5.4
Sources
4
Prompts and skills
6
Error analysis