I tweeted about how the soccer analytics community doesn’t seem to spend enough time on diagnostics, so I’m doing an ongoing series of posts comparing my model to one of the more common (and more elegant) models out there: Club ELO. For more info on the measure and what I’m working on, check my opening post on the ELO challenge: http://soccer.chadmurphy.org/?p=173
First, some overall diagnostics: Week 1, the actual outcome was the most likely outcome 6.5/10 times (one had a tie between two choices so I give the SVM half credit there), week 2 I had 4/10 matches. Overall, the model’s better than 50% which is pretty strong in a complex model like a soccer game.
Time for this week’s update, and ELO won week 2 by about the same margin my SVM won week 1. It also won 6/10, while the SVM won 4/10. Last week, this week, and overall are in the graph below.
As you can see, we’re basically tied. The SVM won week 1, ELO won week 2, and overall we’re about even (a 0.26 point game means ELO is, on average, 1.3% better than the SVM). It’s really interesting how two very different measures come to such similar conclusions, both on many of the individual games and in the aggregate.
The SVM had a couple of REALLY bad misses this week: Everton and Leicester City both won in what my model considered significant upsets. I tweeted about my qualitative asssessment that Everton should have been higher probability this week, but Leicester could go either way. The SVM really likes their overall odds for the season (picking them to compete for a Europa League place), but it didn’t like the odds against West Ham for some reason. I’ll do some diagnostics at some point, but this week’s going to be about putting together a new interactive tool and getting Serie A data up (as well as class prep because school starts next week for me), but I definitely want to see why the model missed so badly on Everton and Leicester. My best guess is that the game stats diverged from the actual stats, but I have no way of knowing until I check. So far I’m happy that I’m doing about as well as the dominant model.
ELO ratings have become one of the more popular ways of predicting soccer results – basically people took the chess ELO ratings and transferred the same logic over to soccer. The work at Club ELO is one of the most ambitious, amazing soccer projects out there, and they deserve all the credit they’ve been getting recently. It’s an effective, simple to calculate, straightforward way of comparing team.
There are two ways to judge the success of a model: its simplicity and its effectiveness. The original IRT model underpinning my predictions is fairly straightforward, although the new SVM is much more complex. The second (more important) comparison is how accurately the models predict results. So I’m going to benchmark my model against the Club ELO model throughout this season – initially I’m going to start with EPL, but I’m hoping to run the model on the other major soccer leagues as well. Week 1’s results are in the table below.
|Game||Murphy's SVM Predictions||Club ELO Predictions
|Manchester United v. Tottenham||0.55||0.54
|Bournemouth v. Aston Villa||0.42|
|Norwich City v. Crystal Palace||0.42|
|Everton v. Watford||0.34||0.24
|Leicester City v. Sunderland||0.54||0.45
|Chelsea v. Swansea||0.19||0.20
|Arsenal v. West Ham United||0.03||0.09
|Newcastle United v. Southampton||0.38||0.29
|Stoke City v. Liverpool||0.24||0.34
|West Bromwich Albion v. Manchester City||0.47||0.59
Each model assigns a predicted probability of the three outcomes (Win/Draw/Loss), and I gave points for how likely each model thought the actual outcome was. For example, if the model predicted an outcome was 30% likely, it earned 0.30 points.
This week was a rough one for both models – there were a lot of upsets (Chelsea’s draw and Arsenal’s loss stand out), so neither score was particularly high.
ELO performed a little better – earning 3.30 points compared to my SVM model’s 3.06. ELO was better on 6/10 games, while I was better on 4/10 games, but we were effectively even on many of the games. The ELO seems to assign a higher percentage to “draw” for most games, as the Chelsea and Everton draws cost me quite a few points, and my model was surprisingly low for Man City’s win over West Brom.
*EDIT: I think I transposed some of the columns in my initial predictions, and I got the numbers completely wrong. My model performed slightly better (3.59 points to 3.30 for ELO), and was better on 6/10 games compared to 4/10 for ELO. I’ve updated the table and future posts.
On the other hand, the SVM made up a lot of ground picking Leicester City over Sunderland and liked West Ham’s chances to win a lot more than the ELO. I think overall I’ll pick up ground on upsets and lose ground on ties, which I’m happy with.