swe-bench-verified · run 2481
candidate 2.4.1-rc3 vs baseline 2.3.0
26 cases · 4 improved · 4 regressed · 18 unchanged
| Case | Base | Cand | Δ score | Δ |
|---|---|---|---|---|
| astropy-12907 | 0.41 | 0.60 | +0.190 | |
| sympy-20590 | 0.34 | 0.47 | +0.130 | |
| django-11039 | 0.55 | 0.63 | +0.080 | |
| pytest-7373 | 0.72 | 0.75 | +0.030 | |
| sympy-16988 | 0.44 | 0.44 | +0.003 | |
| requests-2317 | 0.81 | 0.81 | +0.002 | |
| seaborn-3010 | 0.63 | 0.63 | +0.002 | |
| django-15738 | 0.74 | 0.74 | +0.002 | |
| matplotlib-25311 | 0.57 | 0.57 | +0.002 | |
| matplotlib-23913 | 0.68 | 0.68 | 0.000 | |
| sphinx-10325 | 0.52 | 0.52 | 0.000 | |
| flask-5063 | 0.90 | 0.90 | 0.000 | |
| django-13401 | 0.58 | 0.58 | 0.000 | |
| astropy-14365 | 0.71 | 0.71 | 0.000 | |
| psf-black-2313 | 0.66 | 0.66 | 0.000 | |
| scikit-25747 | 0.50 | 0.50 | 0.000 | |
| pytest-11148 | 0.43 | 0.43 | 0.000 | |
| requests-1963 | 0.86 | 0.86 | 0.000 | |
| pylint-6506 | 0.47 | 0.47 | −0.001 | |
| mwaskom-3187 | 0.36 | 0.36 | −0.001 | |
| sympy-18189 | 0.61 | 0.61 | −0.002 | |
| xarray-3364 | 0.39 | 0.39 | −0.002 | |
| xarray-4094 | 0.49 | 0.45 | −0.040 | |
| sphinx-8721 | 0.38 | 0.29 | −0.090 | |
| django-16819 | 0.60 | 0.47 | −0.130 | |
| scikit-13497 | 0.44 | 0.29 | −0.150 | |
| Δ score · candidate − baseline | −0.200+0.20 | |||
Bars right of zero are gains, bars left are regressions.