Max Malkin
Applied Machine Learning, Fall 2026
Project 1
Beat the Bookmakers: Can We Predict the Outcome of a Soccer Match Better Than Bet365?¶
Purpose¶
I'm a big soccer fan, and use an app called FotMob to look at upcoming fixtures and match predictions. Their app shows predictions for a match winner. I wanted to see if I can recreate this, and compare it with bookmaker odds.
Data: Premier League, La Liga and Bundesliga, 2006-2026¶
The dataset I'm using is Club Football Match Data[1]. It is one CSV with ~240,000 matches across 38 leagues, with data inside coming partially from Football Data[2] and ClubElo[3].
My goal is to predict the probability of each result at full time, in one of three classes: Home win, Away win, or Draw.
Validation¶
I benchmark our model accuracy with both a train/test data split, and against a bookmaker. A bookmaker, in this case, Bet365, which is who we will compare with, loses money when its odds are wrong. Thus, the odds are a good reference.
# import everything we need in advance
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder, PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
# set a consistent color scheme for all plots
RESULT_COLORS = {'H': '#2a78d6', 'D': '#eb6834', 'A': '#1baf7a'}
RESULT_NAMES = {'H': 'Home win', 'D': 'Draw', 'A': 'Away win'}
CLUSTER_COLORS = ['#4a3aa7', '#eda100', '#e87ba4', '#008300'] * 2
# single colors for plots that aren't split by result or cluster
MAIN_COLOR = RESULT_COLORS['H']
# reference lines, shaded areas, and the simple rules we compare against
NEUTRAL_COLOR = '#c3c2b7'
# gaps between bars and heatmap cells, and text written on top of a bar
BG_COLOR = '#ffffff'
# outline around highlighted points
OUTLINE_COLOR = '#0b0b0b'
BLUES = sns.blend_palette(['#cde2fb', '#104281'], as_cmap=True)
BLUE_RED = sns.blend_palette([MAIN_COLOR, '#f0efec', '#e34948'], as_cmap=True)
sns.set_theme(style='whitegrid', rc={'axes.spines.top': False, 'axes.spines.right': False})
# we will print a lot of confusion matrices so we make a helper fn for later
def plot_confusion(y_true, y_pred, title):
cm = confusion_matrix(y_true, y_pred, labels=[2, 1, 0])
names = ['Home win', 'Draw', 'Away win']
plt.figure(figsize=(4.8, 3.6))
sns.heatmap(cm, annot=True, fmt='d', cmap=BLUES, cbar=False, linewidths=2, linecolor=BG_COLOR,
xticklabels=names, yticklabels=names)
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title(f'{title}: {accuracy_score(y_true, y_pred):.1%} accuracy')
plt.show()
url = 'https://raw.githubusercontent.com/xgabora/Club-Football-Match-Data/main/data/Matches.csv'
# MatchDate has inconsistent typing so we use low_memory and parse_dates args to make it consistent
raw = pd.read_csv(url, parse_dates=['MatchDate'], low_memory=False)
raw.shape, raw.head()
((238858, 48),
Division MatchDate MatchTime HomeTeam AwayTeam HomeElo AwayElo \
0 F1 2000-07-28 NaN Marseille Troyes 1686.34 1586.57
1 F1 2000-07-28 NaN Paris SG Strasbourg 1714.89 1642.51
2 F2 2000-07-28 NaN Wasquehal Nancy 1465.08 1633.80
3 F1 2000-07-29 NaN Auxerre Sedan 1635.58 1624.22
4 F1 2000-07-29 NaN Bordeaux Metz 1734.34 1673.11
Form3Home Form5Home Form3Away ... MaxUnder25 HandiSize HandiHome \
0 0.0 0.0 0.0 ... NaN NaN NaN
1 0.0 0.0 0.0 ... NaN NaN NaN
2 0.0 0.0 0.0 ... NaN NaN NaN
3 0.0 0.0 0.0 ... NaN NaN NaN
4 0.0 0.0 0.0 ... NaN NaN NaN
HandiAway C_LTH C_LTA C_VHD C_VAD C_HTB C_PHB
0 NaN NaN NaN NaN NaN NaN NaN
1 NaN NaN NaN NaN NaN NaN NaN
2 NaN NaN NaN NaN NaN NaN NaN
3 NaN NaN NaN NaN NaN NaN NaN
4 NaN NaN NaN NaN NaN NaN NaN
[5 rows x 48 columns])
# keep only the 3 leagues we're interested in
LEAGUES = {'E0': 'Premier League', 'SP1': 'La Liga', 'D1': 'Bundesliga'}
df = raw[raw['Division'].isin(LEAGUES)].sort_values('MatchDate').reset_index(drop=True)
# add the season
# since seasons run Aug -> May, we log the start year
# e.g. the 2025/26 season would be 2025.
df['Season'] = np.where(df['MatchDate'].dt.month >= 7, df['MatchDate'].dt.year, df['MatchDate'].dt.year - 1)
df.groupby('Division')['Season'].agg(['min', 'max', 'count'])
| min | max | count | |
|---|---|---|---|
| Division | |||
| D1 | 2000 | 2026 | 7837 |
| E0 | 2000 | 2026 | 9810 |
| SP1 | 2000 | 2026 | 9419 |
# some teams in the DF have 2 different spellings of name
# namely Nottm vs Nott'm in the PL and MGladbach vs M'Gladbach in La Liga
teams = pd.concat([df['HomeTeam'], df['AwayTeam']]).value_counts()
print(teams[teams.index.str.contains('Nott|gladbach', case=False)])
# standardize spelling
df[['HomeTeam', 'AwayTeam']] = df[['HomeTeam', 'AwayTeam']].replace(
{"Nott'm Forest": 'Nottm Forest', "M'gladbach": 'MGladbach'})
MGladbach 749 Nottm Forest 93 Nott'm Forest 61 M'gladbach 54 Name: count, dtype: int64
Data from before kickoff:¶
A feature has to be known before the match starts in order for a prediction to be accurate. Otherwise, the model would fit based on data it has no way of knowing for a future match.
| Columns | Known |
|---|---|
HomeElo, AwayElo |
Yes |
Form3, Form5 (points from the last 3 or 5 matches) |
Yes |
OddHome, OddDraw, OddAway (Bet365's odds) |
Yes |
| Shots, shots on target, corners, fouls, cards | No |
FTHome, FTAway, halftime score |
No |
C_* (clusters existing in the dataset already) |
No |
We keep the odds since they are known, but only as to benchmark against.
# keep only our known columns from above
keep = ['Division', 'Season', 'MatchDate', 'HomeTeam', 'AwayTeam', 'HomeElo', 'AwayElo',
'Form3Home', 'Form5Home', 'Form3Away', 'Form5Away', 'HomeTarget', 'AwayTarget',
'OddHome', 'OddDraw', 'OddAway', 'FTHome', 'FTAway', 'FTResult']
df = df[keep].copy()
# check proportion of missing vals
df.isna().mean().sort_values(ascending=False).head(6)
| 0 | |
|---|---|
| AwayTarget | 0.106554 |
| HomeTarget | 0.106554 |
| OddDraw | 0.009163 |
| OddAway | 0.009163 |
| OddHome | 0.009163 |
| HomeElo | 0.000037 |
Class Balance:¶
If one class is overly prevalent (e.g. a Home win happens 99% of the time) then the model will overly base outcome on this stat, instead of actual data.
Thus, we check if one is overly common or not.
order = ['H', 'D', 'A']
counts = df['FTResult'].value_counts()[order]
by_league = pd.crosstab(df['Division'].map(LEAGUES), df['FTResult'], normalize='index')[order]
fig, ax = plt.subplots(1, 2, figsize=(12, 3.4), gridspec_kw={'width_ratios': [1, 1.6]})
bars = ax[0].bar([RESULT_NAMES[r] for r in order], counts, color=[RESULT_COLORS[r] for r in order], width=0.6)
ax[0].bar_label(bars, labels=[f'{c / counts.sum():.0%}' for c in counts])
ax[0].set_title('All matches, 2000 to 2026')
by_league.rename(columns=RESULT_NAMES).plot.barh(stacked=True, ax=ax[1], color=list(RESULT_COLORS.values()),
edgecolor=BG_COLOR, linewidth=2, width=0.7)
for bars in ax[1].containers:
ax[1].bar_label(bars, fmt='{:.0%}', label_type='center', color=BG_COLOR)
ax[1].set(title='Results by league', ylabel='')
ax[1].set_xticks([])
ax[1].legend(ncol=3, loc='upper center', bbox_to_anchor=(0.5, -0.02), frameon=False)
plt.tight_layout()
plt.show()
# interpreting the results plotted below:
# one is not overly prevalent and they are nearly the same across leagues
# thus, we should be good to go here
# get home win rate for each season
seasons = df[df['Season'] < df['Season'].max()]
home_rate = (seasons['FTResult'] == 'H').groupby(seasons['Season']).mean()
ax = home_rate.plot(marker='o', color=RESULT_COLORS['H'], figsize=(11, 3.2), lw=2)
ax.axvspan(2019.5, 2020.5, color=NEUTRAL_COLOR, alpha=0.15)
ax.yaxis.set_major_formatter('{x:.0%}')
ax.set(title='Home win rate by season', xlabel='Season', ylabel='')
plt.show()
# in the 2020 season the home win rate drops significantly due to COVID
# empty stadiums are an outlier, thus we drop this from the data later
Prematch Data¶
Expected Goals¶
The dataset has no xG (expected goals), so we use shots on target as the nearest available measure.
For each team, calculate the mean shots on target made and allowed in the last 5 matches:
$$ \bar{S}_{\text{team}} = \frac{1}{5}\sum_{i=1}^{5} S_{\text{team},\,i} $$
where $S_{\text{team},\,i}$ is the team's shots on target in its $i$-th most recent match.
Elo¶
$$ \text{EloDiff} = \text{HomeElo} - \text{AwayElo} $$
The difference is more important than one individually because it gives either a positive or negative value, which gives the direction the odds should lean.[4]
Bookmakers¶
We can take the inverse of the odds to get the original probabilities, using the formula from Smarkets[5]:
$$ p_H = \frac{1/o_H}{1/o_H + 1/o_D + 1/o_A} $$
where $o_H$, $o_D$ and $o_A$ are the odds of a Home win, a Draw and an Away win.
We divide by the sum of the odds to remove the bookmaker surplus.[6] Again, we do not use the odds to train the model, just to score it.
# one row per team per match
home = pd.DataFrame({'match': df.index, 'team': df['HomeTeam'], 'side': 'Home',
'ShotOnTargetFor': df['HomeTarget'], 'ShotOnTargetAgainst': df['AwayTarget']})
away = pd.DataFrame({'match': df.index, 'team': df['AwayTeam'], 'side': 'Away',
'ShotOnTargetFor': df['AwayTarget'], 'ShotOnTargetAgainst': df['HomeTarget']})
long = pd.concat([home, away], ignore_index=True).sort_values('match', kind='stable')
# shots for last 5 matches.
# shift() removes the current match from the mean
for col in ['ShotOnTargetFor', 'ShotOnTargetAgainst']:
long[col] = long.groupby('team')[col].transform(lambda s: s.shift(1).rolling(5, min_periods=3).mean())
# group back to one match per row
wide = long.pivot(index='match', columns='side', values=['ShotOnTargetFor', 'ShotOnTargetAgainst'])
wide.columns = [stat + side for stat, side in wide.columns]
df = df.join(wide)
df.loc[df['HomeTeam'] == 'Liverpool', ['MatchDate', 'HomeTeam', 'AwayTeam', 'HomeTarget', 'ShotOnTargetForHome']].tail(4)
| MatchDate | HomeTeam | AwayTeam | HomeTarget | ShotOnTargetForHome | |
|---|---|---|---|---|---|
| 26879 | 2026-04-25 | Liverpool | Crystal Palace | 3.0 | 5.4 |
| 26930 | 2026-05-09 | Liverpool | Chelsea | 3.0 | 4.8 |
| 27005 | 2026-05-24 | Liverpool | Brentford | 8.0 | 4.4 |
| 27051 | 2026-08-29 | Liverpool | Nottm Forest | 4.0 | 5.6 |
df['EloDiff'] = df['HomeElo'] - df['AwayElo']
# we don't use the bookmaker odds to train, they are just a benchmark to test against
implied = 1 / df[['OddHome', 'OddDraw', 'OddAway']]
# plugs in NaN if an odd is missing
# we aren't too concerned about this since it is just a benchmark
# and the proportion of missing data is very low
total = implied.sum(axis=1, min_count=3)
df['Margin'] = total - 1
df[['BookH', 'BookD', 'BookA']] = implied.div(total, axis=0).to_numpy()
df[['EloDiff', 'Margin', 'BookH', 'BookD', 'BookA']].describe().round(3)
| EloDiff | Margin | BookH | BookD | BookA | |
|---|---|---|---|---|---|
| count | 27064.000 | 26818.000 | 26818.000 | 26818.000 | 26818.000 |
| mean | 0.017 | 0.064 | 0.451 | 0.253 | 0.296 |
| std | 149.407 | 0.028 | 0.173 | 0.046 | 0.156 |
| min | -495.090 | 0.017 | 0.036 | 0.045 | 0.019 |
| 25% | -90.110 | 0.045 | 0.339 | 0.236 | 0.185 |
| 50% | -0.160 | 0.056 | 0.443 | 0.270 | 0.275 |
| 75% | 91.050 | 0.071 | 0.561 | 0.285 | 0.375 |
| max | 534.180 | 0.205 | 0.926 | 0.375 | 0.877 |
Inspecting the Data Visually¶
We plot the data to look for outliers like the COVID season, values that shouldn't exist, etc.
cols = ['HomeElo', 'AwayElo', 'EloDiff', 'Form5Home', 'Form5Away', 'ShotOnTargetForHome',
'ShotOnTargetAgainstHome', 'OddHome', 'OddAway', 'Margin', 'FTHome', 'FTAway']
df[cols].hist(bins=40, figsize=(14, 7.5), layout=(3, 4), color=MAIN_COLOR)
plt.tight_layout()
plt.show()
# from the plots we can say there are no impossible values taken, no extreme outliers, and no super clear issues with the data
print(df.loc[df['OddAway'].idxmax(), ['MatchDate', 'HomeTeam', 'AwayTeam', 'OddAway', 'FTHome', 'FTAway']], '\n')
# there are several outliers based on context:
# 1. seasons before 2006 have no shot data, and we cannot impute accurate data for a whole season
# 2. matches with no fans due to COVID in 2020 drops the home win chances greatly
# thus, we drop seasons from before 2006 and the matches between May 2020 and May 2021
no_sot = df['HomeTarget'].isna().groupby([df['Season'], df['Division']]).mean().unstack()
print(no_sot[no_sot.max(axis=1) > 0.5].round(2), '\n')
ghost = df['MatchDate'].between('2020-05-16', '2021-05-31')
before = len(df)
df = df[(df['Season'] >= 2006) & ~ghost].reset_index(drop=True)
print(f'kept {len(df):,} of {before:,} matches')
MatchDate 2013-11-30 00:00:00 HomeTeam Bayern Munich AwayTeam Braunschweig OddAway 51.0 FTHome 2.0 FTAway 0.0 Name: 13529, dtype: object Division D1 E0 SP1 Season 2000 0.0 0.0 1.0 2001 0.0 0.0 1.0 2002 1.0 0.0 1.0 2003 1.0 0.0 1.0 2004 1.0 0.0 1.0 2005 1.0 0.0 0.0 kept 19,650 of 27,066 matches
Missing Values¶
2 Nottingham Forest games have no Elo. However, Elo changes very slowly over time, so we can safely use Nottingham's last known Elo in the missing places.
Shot form is missing for about 100 matches. A majority of them are clubs that were promoted to the league from the second division. We cannot use the median because the promoted clubs are usually weak and relegated back the next season. Instead we can take the median from teams in a similar Elo bin.
Seasons before 2006 were missing odds. We dropped these seasons above, thus we can do nothing here. Imputing makes no sense since we just benchmark against them, which means we would just be making up data that we trust as correct to score the model against.
print('missing per col:', df[['HomeElo', 'AwayElo', 'ShotOnTargetForHome', 'ShotOnTargetForAway', 'OddHome', 'OddDraw', 'OddAway']].isna().sum().to_dict())
# group elo by 10%
print(df.groupby(pd.qcut(df['HomeElo'], 10), observed=True)['ShotOnTargetForHome'].median().round(1).to_string())
for side in ['Home', 'Away']:
# last known elo
df[side + 'Elo'] = df.groupby(side + 'Team')[side + 'Elo'].ffill()
# fill shots on target data using median of teams within the 10th percentile
elo_band = pd.qcut(df[side + 'Elo'], 10)
for col in ['ShotOnTargetFor' + side, 'ShotOnTargetAgainst' + side]:
df[col] = df[col].fillna(df.groupby(elo_band, observed=True)[col].transform('median'))
df['EloDiff'] = df['HomeElo'] - df['AwayElo']
missing per col: {'HomeElo': 1, 'AwayElo': 1, 'ShotOnTargetForHome': 103, 'ShotOnTargetForAway': 114, 'OddHome': 0, 'OddDraw': 0, 'OddAway': 0}
HomeElo
(1467.009, 1599.678] 3.8
(1599.678, 1630.376] 4.0
(1630.376, 1653.85] 4.0
(1653.85, 1676.77] 4.0
(1676.77, 1700.77] 4.2
(1700.77, 1727.56] 4.4
(1727.56, 1759.84] 4.6
(1759.84, 1809.028] 4.8
(1809.028, 1890.06] 5.4
(1890.06, 2107.48] 6.6
Encoding¶
Divisionhas no order, so we can use OneHot encoding.FTResulthas an order, which we can base on the home team in order of how favorable the result is for the home team: loss, draw, win. Thus, we use ordinal encoding, where the order becomes $A = 0, D = 1, H = 2$.- Team names are not encoded. The players of a club change, especially over the 20 seasons the data covers. Elo is the strength metric, rather than individual players.
league_enc = OneHotEncoder(sparse_output=False)
leagues = pd.DataFrame(league_enc.fit_transform(df[['Division']]),
columns=league_enc.get_feature_names_out(), index=df.index)
df = df.join(leagues)
result_enc = OrdinalEncoder(categories=[['A', 'D', 'H']])
df['Result'] = result_enc.fit_transform(df[['FTResult']]).ravel().astype(int)
print('clubs:', df['HomeTeam'].nunique())
df[['Division', *leagues.columns, 'FTResult', 'Result']].sample(5, random_state=1)
clubs: 123
| Division | Division_D1 | Division_E0 | Division_SP1 | FTResult | Result | |
|---|---|---|---|---|---|---|
| 12636 | E0 | 0.0 | 1.0 | 0.0 | D | 1 |
| 15371 | E0 | 0.0 | 1.0 | 0.0 | D | 1 |
| 5645 | SP1 | 0.0 | 0.0 | 1.0 | D | 1 |
| 15209 | E0 | 0.0 | 1.0 | 0.0 | A | 0 |
| 15087 | E0 | 0.0 | 1.0 | 0.0 | A | 0 |
Correlated Columns¶
We want to remove the correlated columns, since an output may be split between many input columns, while having more than one of those columns has no extra effect on the outcome.
We keep the BookH column in the frame just for comparison, but we do not care about its correlation.
candidates = ['HomeElo', 'AwayElo', 'EloDiff', 'Form5Home', 'Form5Away',
'ShotOnTargetForHome', 'ShotOnTargetAgainstHome', 'ShotOnTargetForAway', 'ShotOnTargetAgainstAway', 'BookH']
corr = df[candidates].corr()
plt.figure(figsize=(10, 7.5))
sns.heatmap(corr, annot=True, fmt='.2f', cmap=BLUE_RED, vmin=-1, vmax=1, annot_kws={'size': 8},
mask=np.triu(np.ones_like(corr, dtype=bool), k=1), linewidths=1, linecolor=BG_COLOR)
plt.title('Correlation')
plt.show()
# 1. Form3 and Form5 have a correlation of 0.83 which makes sense, since one is a subset of the other
# A longer running window of team performance is more valuable since it can highlight an upward or downward trend more clearly
# Thus, we keep Form5 and drop Form3
# 2. BookH and EloDiff have a correlation of 0.94, but we don't train on BookH so we don't care about dropping it
# Bookmakers use Elo to help generate odds so this correlation is logical anyway
MATCH_FEATURES = ['EloDiff', 'Form5Home', 'Form5Away', 'ShotOnTargetForHome', 'ShotOnTargetAgainstHome', 'ShotOnTargetForAway', 'ShotOnTargetAgainstAway']
FEATURES = MATCH_FEATURES + list(leagues.columns)
FEATURES
['EloDiff', 'Form5Home', 'Form5Away', 'ShotOnTargetForHome', 'ShotOnTargetAgainstHome', 'ShotOnTargetForAway', 'ShotOnTargetAgainstAway', 'Division_D1', 'Division_E0', 'Division_SP1']
# scale the data
scaler = StandardScaler().fit_transform(df[MATCH_FEATURES])
k_range = range(2, 9)
sse, sil = [], []
# find best cluster count
for k in k_range:
km = KMeans(n_clusters=k, random_state=0, n_init='auto').fit(scaler)
sse.append(km.inertia_)
# sample_size is fine at 5000, there is no notable difference upping to 10k and 15k except runtime
sil.append(silhouette_score(scaler, km.labels_, sample_size=5000, random_state=0))
fig, ax = plt.subplots(1, 2, figsize=(12, 3.2))
ax[0].plot(k_range, sse, marker='o', color=MAIN_COLOR, lw=2)
ax[0].set(title='Elbow', xlabel='k')
ax[1].plot(k_range, sil, marker='o', color=MAIN_COLOR, lw=2)
ax[1].set(title='Silhouette', xlabel='k')
plt.show()
# optimal number of clusters
best_k = k_range[int(np.argmax(sil))]
print('best k:', best_k)
best k: 4
# try to cluster points
km = KMeans(n_clusters=best_k, random_state=0, n_init='auto').fit(scaler)
df['Cluster'] = km.labels_
df['ShotOnTargetGap'] = df['ShotOnTargetForHome'] - df['ShotOnTargetForAway']
profile = df.groupby('Cluster')[MATCH_FEATURES].mean().round(1)
profile['matches'] = df['Cluster'].value_counts()
display(profile)
fig, ax = plt.subplots(figsize=(9, 5))
for c in range(best_k):
pts = df[df['Cluster'] == c]
ax.scatter(pts['EloDiff'], pts['ShotOnTargetGap'], s=10, alpha=0.35, color=CLUSTER_COLORS[c],
marker='x', label=f'Cluster {c}')
ax.scatter(pts['EloDiff'].mean(), pts['ShotOnTargetGap'].mean(), s=180, color=CLUSTER_COLORS[c],
marker='s', edgecolor=OUTLINE_COLOR, linewidth=1.5)
ax.axvline(0, color=NEUTRAL_COLOR, lw=1)
ax.axhline(0, color=NEUTRAL_COLOR, lw=1)
# home stronger to the right
ax.set(xlabel='EloDiff', ylabel='Shot form difference (H - A)',
title='Clusters')
ax.legend(markerscale=2, frameon=False)
plt.show()
# the clusters heavily overlap and the silhouette scores are low
# EloDiff separates the clusters the most since there is more variance in the vertical axis
| EloDiff | Form5Home | Form5Away | ShotOnTargetForHome | ShotOnTargetAgainstHome | ShotOnTargetForAway | ShotOnTargetAgainstAway | matches | |
|---|---|---|---|---|---|---|---|---|
| Cluster | ||||||||
| 0 | 161.7 | 10.6 | 6.4 | 6.0 | 3.6 | 4.3 | 4.7 | 4649 |
| 1 | 1.1 | 5.1 | 5.3 | 3.9 | 5.0 | 3.9 | 4.9 | 7883 |
| 2 | -155.4 | 6.2 | 10.8 | 4.2 | 4.8 | 6.1 | 3.6 | 4845 |
| 3 | -2.7 | 6.4 | 6.7 | 6.7 | 7.3 | 6.9 | 7.1 | 2273 |
# show the results in each cluster
share = pd.crosstab(df['Cluster'], df['FTResult'], normalize='index')[order]
ax = share.rename(columns=RESULT_NAMES).plot.barh(stacked=True, color=list(RESULT_COLORS.values()), edgecolor=BG_COLOR,
linewidth=2, width=0.7, figsize=(10, 0.6 + 0.55 * best_k))
for bars in ax.containers:
ax.bar_label(bars, fmt='{:.0%}', label_type='center', color=BG_COLOR)
ax.invert_yaxis()
ax.set_xticks([])
ax.set(title='Result by Cluster', ylabel='Cluster Number')
ax.legend(ncol=3, loc='upper center', bbox_to_anchor=(0.5, -0.02), frameon=False)
plt.show()
# if a home team is the favorite, it wins about 2/3 of its fixtures
# draws are most common in clusters with equally weighted teams
# however, draw is never the most common result
Train/Test Splitting¶
We want to keep data ordered by date, otherwise the model will train on a later season and use that to predict a past one. Thus, we take the last 20% of the ordered DF with shuffle = False to be the test data.
X, y = df[FEATURES], df['Result']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False)
X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.2, shuffle=False)
test = df.loc[X_test.index]
print(f"train: {len(X_train):,} matches, {df['MatchDate'].min():%b %Y} to {df.loc[X_train.index[-1], 'MatchDate']:%b %Y}")
print(f"test: {len(X_test):,} matches, {test['MatchDate'].min():%b %Y} to {test['MatchDate'].max():%b %Y}")
train: 15,720 matches, Aug 2006 to Nov 2022 test: 3,930 matches, Nov 2022 to Sep 2026
Models¶
We compare 4 approaches in an attempt to find the best strategy. 2 methods are simple rules, while 2 methods are models.
We want to ensure the model actually is doing something useful which is more complex than following a simple rule, like, as we will see, always picking the home team.
Rule 1: Always pick a Home win since it is the most common result
Rule 2: Always pick the bookmaker's predicted win
Model 1: Logistic regression
Model 2: Logistic regression with class weights, to see if we can predict draws at all. A draw is almost never the single most likely result of a match, so plain logistic regression never picks one. Class weights make a wrong draw cost more during training, which pushes the model to predict some.[7][8][9][10] (We have not covered class weighted LR in class, but it was interesting and I wanted to try it)
Why Logistic Regression¶
- It gives a probability for each result instead of just a pick. This is what FotMob shows, and it lets us compare directly against the bookmaker's probabilities.
- Elo is built on a logistic curve. Thus, it already follows the same S-shape that logistic regression fits. EloDiff is also the feature closest to the bookmaker odds (0.94 correlation with
BookHabove). - The weights are easy to read, so we can see which features push the prediction toward a Home win.
- Soccer results have a lot of luck in them. A simple model with a penalty on its weights is less likely to overfit.
# baselines:
# 1. always pick a home win
# 2. always pick bookmaker's predicted win
# accuracy of always picking a home win is just the share of home wins
home_acc = (y_test == 2).mean()
# encoding order: 0 = A, 1 = D, 2 = H
book_probs = test[['BookA', 'BookD', 'BookH']].to_numpy()
# highest col for match -> encoded result
book_pick = book_probs.argmax(axis=1)
book_acc = accuracy_score(y_test, book_pick)
# proportion of matches where the bookmaker favorite is a draw
draw_picks = (book_pick == 1).mean()
print(f'always home win: {home_acc:.1%}')
print(f'bookmaker favorite: {book_acc:.1%}')
print(f'bookmaker draw picks: {draw_picks:.1%}')
# the bookmaker is ~10% better than always picking a home win
# the bookmaker also almost never has a draw as the favorite
always home win: 44.8% bookmaker favorite: 54.5% bookmaker draw picks: 0.2%
# we create a pipeline to run 3 steps:
# 1. PolynomialFeatures to allow a higher degree polynomial
# 2. StandardScaler to scale data to a uniform scale
# we opt for StandardScaler instead of MinMax, because MinMax would set an extreme data point to be 1, skewing scale
# 3. LogisticRegression predicts outcome probability
def lr_pipeline(degree, C, class_weight=None):
poly = PolynomialFeatures(degree, include_bias=False)
scale = StandardScaler()
# high iterations to not stop early, we have a lot of data
logreg = LogisticRegression(C=C, class_weight=class_weight, max_iter=2000)
return make_pipeline(poly, scale, logreg)
# tune degree and C on the validation set
# C is the inverse of the penalty, so a smaller C gives a simpler model with smaller weights
# we fit on X_fit and score on X_val, the test set is not touched here
# we created a validation split from 20% of the training data
# we use it to tune the degree and the penalty of weights (C)
lr_scores = {}
for degree in [1, 2]:
for C in [0.001, 0.01, 0.1, 1, 10]:
model = lr_pipeline(degree, C)
model.fit(X_fit, y_fit)
val_pred = model.predict(X_val)
lr_scores[(degree, C)] = accuracy_score(y_val, val_pred)
# pick degree and C with the highest accuracy
best_degree, best_C = max(lr_scores, key=lr_scores.get)
# table of validation accuracy
# dict -> Series with (degree, C)
lr_table = pd.Series(lr_scores)
lr_table = lr_table.rename_axis(['degree', 'C'])
# move C into the columns, so degree is the rows
lr_table = lr_table.unstack()
# convert to %
lr_table = (lr_table * 100).round(2)
print('Best degree: ', best_degree)
print('Best C: ', best_C)
# every combination is between 52% and 53%
Best degree: 2 Best C: 0.001
# logistic regression gives a probability for each result
# refit on the full training set with the best params from above, then score on test set
lr = lr_pipeline(best_degree, best_C)
lr.fit(X_train, y_train)
lr_pred = lr.predict(X_test)
plot_confusion(y_test, lr_pred, 'Logistic regression')
# 52.6% accuracy, ~2% less than the bookmaker
# the model never predicts a draw
# this aligns with the clusters
# lr[0] is PolynomialFeatures
# lr[-1] is LogisticRegression
poly_step = lr[0]
logreg_step = lr[-1]
# name of every feature after PolynomialFeatures
feature_names = poly_step.get_feature_names_out()
# coef_[2] is home wins
home_win_weights = logreg_step.coef_[2]
weights = pd.Series(home_win_weights, index=feature_names)
# sort by |size|, keep 10 largest
sorted_weights = weights.sort_values(key=abs)
top_weights = sorted_weights.tail(10).round(2)
top_weights
# every one of the top 10 weights includes EloDiff, thus Elo is most important
# Form doesn't make it in since Elo partially accounts for Form
# the weights are all small since C = 0.001 is a strong penalty
| 0 | |
|---|---|
| EloDiff^2 | 0.04 |
| EloDiff ShotOnTargetForAway | 0.04 |
| EloDiff ShotOnTargetAgainstAway | 0.05 |
| EloDiff Form5Away | 0.05 |
| EloDiff ShotOnTargetAgainstHome | 0.05 |
| EloDiff Division_E0 | 0.05 |
| EloDiff ShotOnTargetForHome | 0.06 |
| EloDiff Division_SP1 | 0.07 |
| EloDiff Form5Home | 0.07 |
| EloDiff | 0.08 |
# logistic regression never predicts a draw, and the bookmaker almost never does either
# class_weight='balanced' weighs each result by the inverse of frequency
# thus, a wrong draw costs more than a wrong home win, which pushes the model to predict draws
lr_bal = lr_pipeline(best_degree, best_C, class_weight='balanced')
lr_bal.fit(X_train, y_train)
bal_pred = lr_bal.predict(X_test)
plot_confusion(y_test, bal_pred, 'Logistic regression, balanced')
# some draws are now predicted correctly but the accuracy drops to about 50%
# most of the draw predictions are wrong
# thus, a draw is close to unpredictable before kickoff
Example Predictions¶
We compare the model and bookmaker probabilities for the most recent test set match between 3 pairs of rivals:
- Arsenal vs Man City (Premier League)
- Real Madrid vs Barcelona (La Liga)
- Bayern Munich vs Dortmund (Bundesliga)
# (home team, away team)
games = [('Arsenal', 'Man City'), ('Real Madrid', 'Barcelona'), ('Bayern Munich', 'Dortmund')]
# most recent match in the test set for each pair, with the same team at home
last_matches = []
for h, a in games:
pair = test[(test['HomeTeam'] == h) & (test['AwayTeam'] == a)]
last_matches.append(pair.tail(1))
rows = pd.concat(last_matches)
# predict_proba gives the columns in our encoding order (A, D, H)
model_probs = lr.predict_proba(rows[FEATURES])
# flip the columns to H, D, A to match the bookmaker columns
model_probs = model_probs[:, ::-1]
# one subplot per match, with a stacked bar for the bookmaker and one for the model
fig, axes = plt.subplots(len(rows), 1, figsize=(10, 1.5 * len(rows)))
# loop over each subplot, match, and set of model probabilities together
# iterrows gives (index, row), we only need the row
for ax, (_, m), p in zip(axes, rows.iterrows(), model_probs):
book_row = m[['BookH', 'BookD', 'BookA']].to_numpy(float)
probs = pd.DataFrame([book_row, p], index=['Bookmaker', 'Model'], columns=order)
probs.plot.barh(stacked=True, ax=ax, legend=False, color=list(RESULT_COLORS.values()),
edgecolor=BG_COLOR, linewidth=2, width=0.8)
# print the probability inside each part of the bar
for bars in ax.containers:
ax.bar_label(bars, fmt='{:.0%}', label_type='center', color=BG_COLOR, fontweight='bold')
ax.set_xticks([])
ax.grid(False)
title = (f"{m['HomeTeam']} vs {m['AwayTeam']}, {m['MatchDate']:%d %b %Y}, final score "
f"{m['FTHome']:.0f}:{m['FTAway']:.0f}")
ax.set_title(title, loc='left', fontsize=11)
# one shared legend for all the subplots
# each legend entry is a plain square in the result's color
handles = [plt.Rectangle((0, 0), 1, 1, color=c) for c in RESULT_COLORS.values()]
fig.legend(handles, RESULT_NAMES.values(), ncol=3, loc='lower center', frameon=False, bbox_to_anchor=(0.5, -0.04))
plt.tight_layout()
plt.show()
# the model and the bookmaker usually only differ by a few percentage points
# however, the bookmaker also uses injuries, lineups, and betting volume, which we don't have
# test accuracy for each model
lr_acc = accuracy_score(y_test, lr_pred)
bal_acc = accuracy_score(y_test, bal_pred)
# home_acc and book_acc are the baselines from above
scores = pd.Series({
'Always home win': home_acc,
'Logistic regression (balanced)': bal_acc,
'Logistic regression': lr_acc,
'Bookmaker favorite': book_acc,
})
# create a plot
fig, ax = plt.subplots(figsize=(8, 4.5))
# the 2 simple rules are grey and our 2 models are blue
colors = [NEUTRAL_COLOR, MAIN_COLOR, MAIN_COLOR, NEUTRAL_COLOR]
bars = ax.barh(scores.index, scores.values, color=colors, edgecolor=BG_COLOR, height=0.6)
ax.xaxis.set_major_formatter('{x:.0%}')
ax.set_xlim(0, 0.7)
ax.set_title('Test set accuracy by model / benchmark')
labels = [f'{v:.1%}' for v in scores]
ax.bar_label(bars, labels=labels, padding=4, fontweight='bold')
plt.tight_layout()
plt.show()
Conclusion¶
My Goal¶
I set out to try and predict the outcomes of a soccer fixture, using only data known from before kickoff. I compare this data with bookmaker odds, specifically Bet365, to see how close I could get to their predictions.
What I Learned About the Dataset¶
The raw dataset had nearly 240,000 matches from 38 different leagues. That is a lot of data, so I decided to focus on the 3 that are the most interesting: the Premier League in England, La Liga in Spain, and Bundesliga in Germany, all the top flights in their respective countries. After removing all the other leagues, I was left with about 20,000 fixtures and 123 clubs.
Regarding the quality of the data, it was pretty good out of the box. There were only a couple small fixes needed: fixing the 2 different club name spellings, and 2 Elo values were missing. There were also about 100 shot form values missing, mainly for clubs that were newly promoted, hence that data was probably not tracked in a lower division.
The most interesting part of the data is the home advantage that comes from having the club's fans at the stadium. The home team win rate is 44.8%, significantly higher than the away side's 30.3%. I see the home win rate drop significantly in the 2020 season, when fans were not allowed in the stadiums because of COVID. A global pandemic is definitely an outlier, thus I dropped that season in all the leagues.
The 3 classes were well balanced across the leagues, so I did not use a separate model for each league (or pick a single league to study).
Elo supplies the most information to my model. EloDiff had a 0.94 correlation with the bookmaker's home side win prediction, meaning the bookmakers lean heavily on that value.
The clusters that the KMeans clusterer found were very overlapped and their silhouette score was low. A draw is never the most common result in any of the clusters, which gives me an indication that they were the hardest to predict.
Model Effectiveness¶
My final numbers on accuracy across the 2 models and 2 baseline rules were as such:
| Type | Method | Accuracy |
|---|---|---|
| Rule | Bookmaker prediction | 54.5% |
| Model | Logistic Regression | 52.6% |
| Model | Logistic Regression (balanced) | 48.7% |
| Rule | Always picking a home win | 44.8% |
Plain logistic regression is the most effective model. It's nearly 8% more effective than just always picking a home side win, which means there is at least some pattern that the model was able to learn from the prematch data.
Unfortunately, my model's accuracy is 1.9% behind the bookmaker's accuracy. However, considering that the bookmakers aren't just using one dataset and 10 features, but full team lineups, news, injuries, and a ton of other data, I would say that it is not a bad result at all for my model.
The model also is very reluctant to predict a draw to be most likely. In almost every fixture, one team has at least a slight advantage, so a win is always more likely. The bookmaker also only predicts a draw 0.2% of the time, which means the intuition to not pick draws is correct.
What the Model Says¶
A team's Elo decided most matches. The top 10 weights always included EloDiff; form and shots on target only appear in combination with EloDiff, which makes sense considering that Elo partially encapsulates the recent form of a club.
I can also say that the model's predicted probabilities are realistic. The difference between its output and the bookmaker's are only different by a few percentage points.
Learnings about the Model¶
I learned that the settings barely matter on the model. Trying multiple weights and degrees only moves the accuracy of the model in a range between 52 and 53 percent, so a borderline insignificant effect. The simplest settings worked the best, with degree 2 and a C of 0.001.
Logistic regression was a good choice of model, since Elo is built on a logistic curve. The model also outputs probabilities, just like FotMob and Bet365, so it is very easy to compare results.
I compare with these sources because accuracy is a weird metric for soccer, even the bookmaker's predicted winner still loses 45% of the time. It just means that there is a clear limit to what I can predict with only prematch data.
I am not more accurate than the bookmaker, and I cannot be, because the bookmaker simply uses more data. They use injuries, news, squad lineups and the betting volume, which I can't include since my constraints are to only use one dataset. Soccer also just has a lot of luck involved, like any sport. Any cards, saved goals, fouls, etc., can change the course of a game very quickly.
There are, however, some limitations. Accuracy only checks the highest predicted probability against the outcome of the match, not how good the probabilities actually are, so something like a 51% home win chance prediction compared to a 99% one is graded the same. Also, since I did not have expected goals in the dataset, I decided to take the average of the last 5 games' goals for that team and use it as an indicator.
Overall¶
Overall, I am happy with the accuracy of my model and the outcome of this experiment. Getting within 2% of a professional bookmaker using only one dataset and using only prematch data shows that it is not only hard to predict the outcome at the final whistle, but that very simple features carry a lot of weight in trying to predict it.
References¶
Data¶
[1] Club Football Match Data (GitHub)
[3] ClubElo
Elo Ratings¶
[4] Elo rating system (Wikipedia)
Bookmaker Odds¶
[5] How to calculate implied probability in betting (Smarkets)
[6] Mathematics of bookmaking (Wikipedia)
Class-Weighted Logistic Regression¶
[7] How to Improve Logistic Regression in Imbalanced Data with Class Weights (Medium)
[8] Weighted Logistic Regression for Imbalanced Dataset (GeeksforGeeks)