Feature: Scripts for Maze benchmark

[tikikun]

[bachvudinh]

The code that automatically eval on Maze-Bench is done. Supported Transformers and VLLM. The bench is uploaded here: https://huggingface.co/datasets/homebrewltd/Maze-Bench-v0.1:

• 40 easy maze( 3-5 step to reach target)
• 30 medium(6-10 steps)
• 30 hard maze(11-15 steps)
• 10 very hard maze(16-20 steps)

[bachvudinh]

Leaderboard (Updated)

Model Name	Overall Accuracy (%)
jan-hq/AlphaMaze-v0.1-1.5B-SFT	63
jan-hq/AlphaMaze-v0.1-1.5B-GRPO-cp-200	69
jan-hq/AlphaMaze-v0.1-1.5B-GRPO-cp-600	70

[bachvudinh]

Recreated the benchmark to reduce difficulty. The updated benchmark now includes 100 test mazes:

• 50 easy mazes (3-5 steps to target)
• 40 medium mazes (6-10 steps)
• 10 hard mazes (11-15 steps)

[bachvudinh]

Model Name	Overall Accuracy (%)	Easy (%)	Medium (%)	Hard (%)
deepseekai/DeepSeek-R1-Distill-Qwen-1.5B	0	0	0	0
deepseekai/DeepSeek-R1-Distill-Qwen-7B	0	0	0	0
jan-hq/Qwen-1.5B-distil-base	0	0	0	0
jan-hq/AlphaMaze-v0.1-1.5B-SFT	75	88	72.5	20
jan-hq/AlphaMaze-v0.1-1.5B-GRPO-cp-200	77	88	77.5	20
jan-hq/AlphaMaze-v0.1-1.5B-GRPO-cp-600	73	80	77.5	20
jan-hq/AlphaMaze-v0.2-1.5B-SFT	87	92	85	70
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-200	89	95	87.5	70
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-400	86	94	80	70
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-600	87	92	82.5	80

[bachvudinh]

• Due to identified data leakage in the training set, I've uploaded an updated benchmark (maze-bench-v0.2) and re-evaluated all recent checkpoints. Updated results are presented in the table below:

Model Name	Overall Accuracy (%)	Easy (%)	Medium (%)	Hard (%)
jan-hq/AlphaMaze-v0.2-1.5B-SFT	86	90	85	70
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-200	86	90	80	90
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-400	91	96	85	90
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-600	91	94	90	80
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-800	88	88	90	80
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-1000	92	94	92.5	80
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-1200	89	94	85	80
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-1400	91	94	87.5	80
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-1600	93	96	95	70
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-1800	92	94	92.5	80
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-2000	90	92	92.5	70

[bachvudinh]

Updated Detailed result and more experiments for submission:

Model Name	Overall Accuracy (%)	Easy	Medium	Hard	Errors: Exceed max token	Errors: Hit Wall (Blocked)	Errors: Did Not Reach Target
deepseekai/DeepSeek-R1-Distill-Qwen-1.5B	1	1	0	0	43	36	20
deepseekai/DeepSeek-R1-Distill-Qwen-7B	5	4	1	0	16	37	42
deepseek-r1 reasoner model	74	43	28	3	0	22	4
jan-hq/AlphaMaze-v0.2-1.5B-SFT	87	46	34	7	0	11	2
jan-hq/AlphaMaze-v0.2-1.5B-GRPO-cp-1600	95	49	38	8	0	5	0