Max Malkin
Applied Machine Learning, Fall 2026
Project 1

Beat the Bookmakers: Can We Predict the Outcome of a Soccer Match Better Than Bet365?¶

Purpose¶

I'm a big soccer fan, and use an app called FotMob to look at upcoming fixtures and match predictions. Their app shows predictions for a match winner. I wanted to see if I can recreate this, and compare it with bookmaker odds.

Data: Premier League, La Liga and Bundesliga, 2006-2026¶

The dataset I'm using is Club Football Match Data[1]. It is one CSV with ~240,000 matches across 38 leagues, with data inside coming partially from Football Data[2] and ClubElo[3].

My goal is to predict the probability of each result at full time, in one of three classes: Home win, Away win, or Draw.

Validation¶

I benchmark our model accuracy with both a train/test data split, and against a bookmaker. A bookmaker, in this case, Bet365, which is who we will compare with, loses money when its odds are wrong. Thus, the odds are a good reference.

In [1]:
# import everything we need in advance
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder, PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
In [2]:
# set a consistent color scheme for all plots
RESULT_COLORS = {'H': '#2a78d6', 'D': '#eb6834', 'A': '#1baf7a'}
RESULT_NAMES = {'H': 'Home win', 'D': 'Draw', 'A': 'Away win'}
CLUSTER_COLORS = ['#4a3aa7', '#eda100', '#e87ba4', '#008300'] * 2
# single colors for plots that aren't split by result or cluster
MAIN_COLOR = RESULT_COLORS['H']
# reference lines, shaded areas, and the simple rules we compare against
NEUTRAL_COLOR = '#c3c2b7'
# gaps between bars and heatmap cells, and text written on top of a bar
BG_COLOR = '#ffffff'
# outline around highlighted points
OUTLINE_COLOR = '#0b0b0b'
BLUES = sns.blend_palette(['#cde2fb', '#104281'], as_cmap=True)
BLUE_RED = sns.blend_palette([MAIN_COLOR, '#f0efec', '#e34948'], as_cmap=True)
sns.set_theme(style='whitegrid', rc={'axes.spines.top': False, 'axes.spines.right': False})
In [3]:
# we will print a lot of confusion matrices so we make a helper fn for later
def plot_confusion(y_true, y_pred, title):
    cm = confusion_matrix(y_true, y_pred, labels=[2, 1, 0])
    names = ['Home win', 'Draw', 'Away win']
    plt.figure(figsize=(4.8, 3.6))
    sns.heatmap(cm, annot=True, fmt='d', cmap=BLUES, cbar=False, linewidths=2, linecolor=BG_COLOR,
                xticklabels=names, yticklabels=names)
    plt.xlabel('Predicted')
    plt.ylabel('Actual')
    plt.title(f'{title}: {accuracy_score(y_true, y_pred):.1%} accuracy')
    plt.show()
In [4]:
url = 'https://raw.githubusercontent.com/xgabora/Club-Football-Match-Data/main/data/Matches.csv'

# MatchDate has inconsistent typing so we use low_memory and parse_dates args to make it consistent
raw = pd.read_csv(url, parse_dates=['MatchDate'], low_memory=False)

raw.shape, raw.head()
Out[4]:
((238858, 48),
   Division  MatchDate MatchTime   HomeTeam    AwayTeam  HomeElo  AwayElo  \
 0       F1 2000-07-28       NaN  Marseille      Troyes  1686.34  1586.57   
 1       F1 2000-07-28       NaN   Paris SG  Strasbourg  1714.89  1642.51   
 2       F2 2000-07-28       NaN  Wasquehal       Nancy  1465.08  1633.80   
 3       F1 2000-07-29       NaN    Auxerre       Sedan  1635.58  1624.22   
 4       F1 2000-07-29       NaN   Bordeaux        Metz  1734.34  1673.11   
 
    Form3Home  Form5Home  Form3Away  ...  MaxUnder25  HandiSize  HandiHome  \
 0        0.0        0.0        0.0  ...         NaN        NaN        NaN   
 1        0.0        0.0        0.0  ...         NaN        NaN        NaN   
 2        0.0        0.0        0.0  ...         NaN        NaN        NaN   
 3        0.0        0.0        0.0  ...         NaN        NaN        NaN   
 4        0.0        0.0        0.0  ...         NaN        NaN        NaN   
 
   HandiAway  C_LTH  C_LTA C_VHD  C_VAD  C_HTB  C_PHB  
 0       NaN    NaN    NaN   NaN    NaN    NaN    NaN  
 1       NaN    NaN    NaN   NaN    NaN    NaN    NaN  
 2       NaN    NaN    NaN   NaN    NaN    NaN    NaN  
 3       NaN    NaN    NaN   NaN    NaN    NaN    NaN  
 4       NaN    NaN    NaN   NaN    NaN    NaN    NaN  
 
 [5 rows x 48 columns])
In [5]:
# keep only the 3 leagues we're interested in
LEAGUES = {'E0': 'Premier League', 'SP1': 'La Liga', 'D1': 'Bundesliga'}
df = raw[raw['Division'].isin(LEAGUES)].sort_values('MatchDate').reset_index(drop=True)
# add the season
# since seasons run Aug -> May, we log the start year
# e.g. the 2025/26 season would be 2025.
df['Season'] = np.where(df['MatchDate'].dt.month >= 7, df['MatchDate'].dt.year, df['MatchDate'].dt.year - 1)
df.groupby('Division')['Season'].agg(['min', 'max', 'count'])
Out[5]:
min max count
Division
D1 2000 2026 7837
E0 2000 2026 9810
SP1 2000 2026 9419
In [6]:
# some teams in the DF have 2 different spellings of name
# namely Nottm vs Nott'm in the PL and MGladbach vs M'Gladbach in La Liga
teams = pd.concat([df['HomeTeam'], df['AwayTeam']]).value_counts()
print(teams[teams.index.str.contains('Nott|gladbach', case=False)])

# standardize spelling
df[['HomeTeam', 'AwayTeam']] = df[['HomeTeam', 'AwayTeam']].replace(
    {"Nott'm Forest": 'Nottm Forest', "M'gladbach": 'MGladbach'})
MGladbach        749
Nottm Forest      93
Nott'm Forest     61
M'gladbach        54
Name: count, dtype: int64

Data from before kickoff:¶

A feature has to be known before the match starts in order for a prediction to be accurate. Otherwise, the model would fit based on data it has no way of knowing for a future match.

Columns Known
HomeElo, AwayElo Yes
Form3, Form5 (points from the last 3 or 5 matches) Yes
OddHome, OddDraw, OddAway (Bet365's odds) Yes
Shots, shots on target, corners, fouls, cards No
FTHome, FTAway, halftime score No
C_* (clusters existing in the dataset already) No

We keep the odds since they are known, but only as to benchmark against.

In [7]:
# keep only our known columns from above
keep = ['Division', 'Season', 'MatchDate', 'HomeTeam', 'AwayTeam', 'HomeElo', 'AwayElo',
        'Form3Home', 'Form5Home', 'Form3Away', 'Form5Away', 'HomeTarget', 'AwayTarget',
        'OddHome', 'OddDraw', 'OddAway', 'FTHome', 'FTAway', 'FTResult']
df = df[keep].copy()

# check proportion of missing vals
df.isna().mean().sort_values(ascending=False).head(6)
Out[7]:
0
AwayTarget 0.106554
HomeTarget 0.106554
OddDraw 0.009163
OddAway 0.009163
OddHome 0.009163
HomeElo 0.000037

Class Balance:¶

If one class is overly prevalent (e.g. a Home win happens 99% of the time) then the model will overly base outcome on this stat, instead of actual data.

Thus, we check if one is overly common or not.

In [8]:
order = ['H', 'D', 'A']
counts = df['FTResult'].value_counts()[order]
by_league = pd.crosstab(df['Division'].map(LEAGUES), df['FTResult'], normalize='index')[order]

fig, ax = plt.subplots(1, 2, figsize=(12, 3.4), gridspec_kw={'width_ratios': [1, 1.6]})
bars = ax[0].bar([RESULT_NAMES[r] for r in order], counts, color=[RESULT_COLORS[r] for r in order], width=0.6)
ax[0].bar_label(bars, labels=[f'{c / counts.sum():.0%}' for c in counts])
ax[0].set_title('All matches, 2000 to 2026')

by_league.rename(columns=RESULT_NAMES).plot.barh(stacked=True, ax=ax[1], color=list(RESULT_COLORS.values()),
                                                 edgecolor=BG_COLOR, linewidth=2, width=0.7)
for bars in ax[1].containers:
    ax[1].bar_label(bars, fmt='{:.0%}', label_type='center', color=BG_COLOR)
ax[1].set(title='Results by league', ylabel='')
ax[1].set_xticks([])
ax[1].legend(ncol=3, loc='upper center', bbox_to_anchor=(0.5, -0.02), frameon=False)
plt.tight_layout()
plt.show()

# interpreting the results plotted below:
# one is not overly prevalent and they are nearly the same across leagues
# thus, we should be good to go here
No description has been provided for this image
In [9]:
# get home win rate for each season
seasons = df[df['Season'] < df['Season'].max()]
home_rate = (seasons['FTResult'] == 'H').groupby(seasons['Season']).mean()

ax = home_rate.plot(marker='o', color=RESULT_COLORS['H'], figsize=(11, 3.2), lw=2)
ax.axvspan(2019.5, 2020.5, color=NEUTRAL_COLOR, alpha=0.15)
ax.yaxis.set_major_formatter('{x:.0%}')
ax.set(title='Home win rate by season', xlabel='Season', ylabel='')
plt.show()

# in the 2020 season the home win rate drops significantly due to COVID
# empty stadiums are an outlier, thus we drop this from the data later
No description has been provided for this image

Prematch Data¶

Expected Goals¶

The dataset has no xG (expected goals), so we use shots on target as the nearest available measure. For each team, calculate the mean shots on target made and allowed in the last 5 matches:

$$ \bar{S}_{\text{team}} = \frac{1}{5}\sum_{i=1}^{5} S_{\text{team},\,i} $$

where $S_{\text{team},\,i}$ is the team's shots on target in its $i$-th most recent match.

Elo¶

$$ \text{EloDiff} = \text{HomeElo} - \text{AwayElo} $$

The difference is more important than one individually because it gives either a positive or negative value, which gives the direction the odds should lean.[4]

Bookmakers¶

We can take the inverse of the odds to get the original probabilities, using the formula from Smarkets[5]:

$$ p_H = \frac{1/o_H}{1/o_H + 1/o_D + 1/o_A} $$

where $o_H$, $o_D$ and $o_A$ are the odds of a Home win, a Draw and an Away win.

We divide by the sum of the odds to remove the bookmaker surplus.[6] Again, we do not use the odds to train the model, just to score it.

In [10]:
# one row per team per match
home = pd.DataFrame({'match': df.index, 'team': df['HomeTeam'], 'side': 'Home',
                     'ShotOnTargetFor': df['HomeTarget'], 'ShotOnTargetAgainst': df['AwayTarget']})
away = pd.DataFrame({'match': df.index, 'team': df['AwayTeam'], 'side': 'Away',
                     'ShotOnTargetFor': df['AwayTarget'], 'ShotOnTargetAgainst': df['HomeTarget']})
long = pd.concat([home, away], ignore_index=True).sort_values('match', kind='stable')

# shots for last 5 matches.
# shift() removes the current match from the mean
for col in ['ShotOnTargetFor', 'ShotOnTargetAgainst']:
    long[col] = long.groupby('team')[col].transform(lambda s: s.shift(1).rolling(5, min_periods=3).mean())

# group back to one match per row
wide = long.pivot(index='match', columns='side', values=['ShotOnTargetFor', 'ShotOnTargetAgainst'])
wide.columns = [stat + side for stat, side in wide.columns]
df = df.join(wide)
df.loc[df['HomeTeam'] == 'Liverpool', ['MatchDate', 'HomeTeam', 'AwayTeam', 'HomeTarget', 'ShotOnTargetForHome']].tail(4)
Out[10]:
MatchDate HomeTeam AwayTeam HomeTarget ShotOnTargetForHome
26879 2026-04-25 Liverpool Crystal Palace 3.0 5.4
26930 2026-05-09 Liverpool Chelsea 3.0 4.8
27005 2026-05-24 Liverpool Brentford 8.0 4.4
27051 2026-08-29 Liverpool Nottm Forest 4.0 5.6
In [11]:
df['EloDiff'] = df['HomeElo'] - df['AwayElo']

# we don't use the bookmaker odds to train, they are just a benchmark to test against
implied = 1 / df[['OddHome', 'OddDraw', 'OddAway']]

# plugs in NaN if an odd is missing
# we aren't too concerned about this since it is just a benchmark
# and the proportion of missing data is very low
total = implied.sum(axis=1, min_count=3)
df['Margin'] = total - 1
df[['BookH', 'BookD', 'BookA']] = implied.div(total, axis=0).to_numpy()
df[['EloDiff', 'Margin', 'BookH', 'BookD', 'BookA']].describe().round(3)
Out[11]:
EloDiff Margin BookH BookD BookA
count 27064.000 26818.000 26818.000 26818.000 26818.000
mean 0.017 0.064 0.451 0.253 0.296
std 149.407 0.028 0.173 0.046 0.156
min -495.090 0.017 0.036 0.045 0.019
25% -90.110 0.045 0.339 0.236 0.185
50% -0.160 0.056 0.443 0.270 0.275
75% 91.050 0.071 0.561 0.285 0.375
max 534.180 0.205 0.926 0.375 0.877

Inspecting the Data Visually¶

We plot the data to look for outliers like the COVID season, values that shouldn't exist, etc.

In [12]:
cols = ['HomeElo', 'AwayElo', 'EloDiff', 'Form5Home', 'Form5Away', 'ShotOnTargetForHome',
        'ShotOnTargetAgainstHome', 'OddHome', 'OddAway', 'Margin', 'FTHome', 'FTAway']
df[cols].hist(bins=40, figsize=(14, 7.5), layout=(3, 4), color=MAIN_COLOR)
plt.tight_layout()
plt.show()

# from the plots we can say there are no impossible values taken, no extreme outliers, and no super clear issues with the data
No description has been provided for this image
In [13]:
print(df.loc[df['OddAway'].idxmax(), ['MatchDate', 'HomeTeam', 'AwayTeam', 'OddAway', 'FTHome', 'FTAway']], '\n')

# there are several outliers based on context:
#   1. seasons before 2006 have no shot data, and we cannot impute accurate data for a whole season
#   2. matches with no fans due to COVID in 2020 drops the home win chances greatly
# thus, we drop seasons from before 2006 and the matches between May 2020 and May 2021

no_sot = df['HomeTarget'].isna().groupby([df['Season'], df['Division']]).mean().unstack()
print(no_sot[no_sot.max(axis=1) > 0.5].round(2), '\n')

ghost = df['MatchDate'].between('2020-05-16', '2021-05-31')
before = len(df)
df = df[(df['Season'] >= 2006) & ~ghost].reset_index(drop=True)
print(f'kept {len(df):,} of {before:,} matches')
MatchDate    2013-11-30 00:00:00
HomeTeam           Bayern Munich
AwayTeam            Braunschweig
OddAway                     51.0
FTHome                       2.0
FTAway                       0.0
Name: 13529, dtype: object 

Division   D1   E0  SP1
Season                 
2000      0.0  0.0  1.0
2001      0.0  0.0  1.0
2002      1.0  0.0  1.0
2003      1.0  0.0  1.0
2004      1.0  0.0  1.0
2005      1.0  0.0  0.0 

kept 19,650 of 27,066 matches

Missing Values¶

  1. 2 Nottingham Forest games have no Elo. However, Elo changes very slowly over time, so we can safely use Nottingham's last known Elo in the missing places.

  2. Shot form is missing for about 100 matches. A majority of them are clubs that were promoted to the league from the second division. We cannot use the median because the promoted clubs are usually weak and relegated back the next season. Instead we can take the median from teams in a similar Elo bin.

  3. Seasons before 2006 were missing odds. We dropped these seasons above, thus we can do nothing here. Imputing makes no sense since we just benchmark against them, which means we would just be making up data that we trust as correct to score the model against.

In [14]:
print('missing per col:', df[['HomeElo', 'AwayElo', 'ShotOnTargetForHome', 'ShotOnTargetForAway', 'OddHome', 'OddDraw', 'OddAway']].isna().sum().to_dict())

# group elo by 10%
print(df.groupby(pd.qcut(df['HomeElo'], 10), observed=True)['ShotOnTargetForHome'].median().round(1).to_string())

for side in ['Home', 'Away']:
    # last known elo
    df[side + 'Elo'] = df.groupby(side + 'Team')[side + 'Elo'].ffill()
    # fill shots on target data using median of teams within the 10th percentile
    elo_band = pd.qcut(df[side + 'Elo'], 10)
    for col in ['ShotOnTargetFor' + side, 'ShotOnTargetAgainst' + side]:
        df[col] = df[col].fillna(df.groupby(elo_band, observed=True)[col].transform('median'))

df['EloDiff'] = df['HomeElo'] - df['AwayElo']
missing per col: {'HomeElo': 1, 'AwayElo': 1, 'ShotOnTargetForHome': 103, 'ShotOnTargetForAway': 114, 'OddHome': 0, 'OddDraw': 0, 'OddAway': 0}
HomeElo
(1467.009, 1599.678]    3.8
(1599.678, 1630.376]    4.0
(1630.376, 1653.85]     4.0
(1653.85, 1676.77]      4.0
(1676.77, 1700.77]      4.2
(1700.77, 1727.56]      4.4
(1727.56, 1759.84]      4.6
(1759.84, 1809.028]     4.8
(1809.028, 1890.06]     5.4
(1890.06, 2107.48]      6.6

Encoding¶

  1. Division has no order, so we can use OneHot encoding.
  2. FTResult has an order, which we can base on the home team in order of how favorable the result is for the home team: loss, draw, win. Thus, we use ordinal encoding, where the order becomes $A = 0, D = 1, H = 2$.
  3. Team names are not encoded. The players of a club change, especially over the 20 seasons the data covers. Elo is the strength metric, rather than individual players.
In [15]:
league_enc = OneHotEncoder(sparse_output=False)
leagues = pd.DataFrame(league_enc.fit_transform(df[['Division']]),
                       columns=league_enc.get_feature_names_out(), index=df.index)
df = df.join(leagues)

result_enc = OrdinalEncoder(categories=[['A', 'D', 'H']])
df['Result'] = result_enc.fit_transform(df[['FTResult']]).ravel().astype(int)

print('clubs:', df['HomeTeam'].nunique())
df[['Division', *leagues.columns, 'FTResult', 'Result']].sample(5, random_state=1)
clubs: 123
Out[15]:
Division Division_D1 Division_E0 Division_SP1 FTResult Result
12636 E0 0.0 1.0 0.0 D 1
15371 E0 0.0 1.0 0.0 D 1
5645 SP1 0.0 0.0 1.0 D 1
15209 E0 0.0 1.0 0.0 A 0
15087 E0 0.0 1.0 0.0 A 0

Correlated Columns¶

We want to remove the correlated columns, since an output may be split between many input columns, while having more than one of those columns has no extra effect on the outcome.

We keep the BookH column in the frame just for comparison, but we do not care about its correlation.

In [16]:
candidates = ['HomeElo', 'AwayElo', 'EloDiff', 'Form5Home', 'Form5Away',
              'ShotOnTargetForHome', 'ShotOnTargetAgainstHome', 'ShotOnTargetForAway', 'ShotOnTargetAgainstAway', 'BookH']
corr = df[candidates].corr()
plt.figure(figsize=(10, 7.5))
sns.heatmap(corr, annot=True, fmt='.2f', cmap=BLUE_RED, vmin=-1, vmax=1, annot_kws={'size': 8},
            mask=np.triu(np.ones_like(corr, dtype=bool), k=1), linewidths=1, linecolor=BG_COLOR)
plt.title('Correlation')
plt.show()
No description has been provided for this image
In [17]:
# 1. Form3 and Form5 have a correlation of 0.83 which makes sense, since one is a subset of the other
#    A longer running window of team performance is more valuable since it can highlight an upward or downward trend more clearly
#    Thus, we keep Form5 and drop Form3
# 2. BookH and EloDiff have a correlation of 0.94, but we don't train on BookH so we don't care about dropping it
#    Bookmakers use Elo to help generate odds so this correlation is logical anyway
MATCH_FEATURES = ['EloDiff', 'Form5Home', 'Form5Away', 'ShotOnTargetForHome', 'ShotOnTargetAgainstHome', 'ShotOnTargetForAway', 'ShotOnTargetAgainstAway']
FEATURES = MATCH_FEATURES + list(leagues.columns)
FEATURES
Out[17]:
['EloDiff',
 'Form5Home',
 'Form5Away',
 'ShotOnTargetForHome',
 'ShotOnTargetAgainstHome',
 'ShotOnTargetForAway',
 'ShotOnTargetAgainstAway',
 'Division_D1',
 'Division_E0',
 'Division_SP1']
In [18]:
# scale the data
scaler = StandardScaler().fit_transform(df[MATCH_FEATURES])
k_range = range(2, 9)
sse, sil = [], []

# find best cluster count
for k in k_range:
    km = KMeans(n_clusters=k, random_state=0, n_init='auto').fit(scaler)
    sse.append(km.inertia_)
    # sample_size is fine at 5000, there is no notable difference upping to 10k and 15k except runtime
    sil.append(silhouette_score(scaler, km.labels_, sample_size=5000, random_state=0))

fig, ax = plt.subplots(1, 2, figsize=(12, 3.2))
ax[0].plot(k_range, sse, marker='o', color=MAIN_COLOR, lw=2)
ax[0].set(title='Elbow', xlabel='k')
ax[1].plot(k_range, sil, marker='o', color=MAIN_COLOR, lw=2)
ax[1].set(title='Silhouette', xlabel='k')
plt.show()

# optimal number of clusters
best_k = k_range[int(np.argmax(sil))]
print('best k:', best_k)
No description has been provided for this image
best k: 4
In [19]:
# try to cluster points
km = KMeans(n_clusters=best_k, random_state=0, n_init='auto').fit(scaler)
df['Cluster'] = km.labels_
df['ShotOnTargetGap'] = df['ShotOnTargetForHome'] - df['ShotOnTargetForAway']

profile = df.groupby('Cluster')[MATCH_FEATURES].mean().round(1)
profile['matches'] = df['Cluster'].value_counts()
display(profile)

fig, ax = plt.subplots(figsize=(9, 5))
for c in range(best_k):
    pts = df[df['Cluster'] == c]
    ax.scatter(pts['EloDiff'], pts['ShotOnTargetGap'], s=10, alpha=0.35, color=CLUSTER_COLORS[c],
               marker='x', label=f'Cluster {c}')
    ax.scatter(pts['EloDiff'].mean(), pts['ShotOnTargetGap'].mean(), s=180, color=CLUSTER_COLORS[c],
               marker='s', edgecolor=OUTLINE_COLOR, linewidth=1.5)
ax.axvline(0, color=NEUTRAL_COLOR, lw=1)
ax.axhline(0, color=NEUTRAL_COLOR, lw=1)
# home stronger to the right
ax.set(xlabel='EloDiff', ylabel='Shot form difference (H - A)',
       title='Clusters')
ax.legend(markerscale=2, frameon=False)
plt.show()

# the clusters heavily overlap and the silhouette scores are low
# EloDiff separates the clusters the most since there is more variance in the vertical axis
EloDiff Form5Home Form5Away ShotOnTargetForHome ShotOnTargetAgainstHome ShotOnTargetForAway ShotOnTargetAgainstAway matches
Cluster
0 161.7 10.6 6.4 6.0 3.6 4.3 4.7 4649
1 1.1 5.1 5.3 3.9 5.0 3.9 4.9 7883
2 -155.4 6.2 10.8 4.2 4.8 6.1 3.6 4845
3 -2.7 6.4 6.7 6.7 7.3 6.9 7.1 2273
No description has been provided for this image
In [20]:
# show the results in each cluster
share = pd.crosstab(df['Cluster'], df['FTResult'], normalize='index')[order]

ax = share.rename(columns=RESULT_NAMES).plot.barh(stacked=True, color=list(RESULT_COLORS.values()), edgecolor=BG_COLOR,
                                                  linewidth=2, width=0.7, figsize=(10, 0.6 + 0.55 * best_k))
for bars in ax.containers:
    ax.bar_label(bars, fmt='{:.0%}', label_type='center', color=BG_COLOR)
ax.invert_yaxis()
ax.set_xticks([])
ax.set(title='Result by Cluster', ylabel='Cluster Number')
ax.legend(ncol=3, loc='upper center', bbox_to_anchor=(0.5, -0.02), frameon=False)
plt.show()

# if a home team is the favorite, it wins about 2/3 of its fixtures
# draws are most common in clusters with equally weighted teams
# however, draw is never the most common result
No description has been provided for this image

Train/Test Splitting¶

We want to keep data ordered by date, otherwise the model will train on a later season and use that to predict a past one. Thus, we take the last 20% of the ordered DF with shuffle = False to be the test data.

In [21]:
X, y = df[FEATURES], df['Result']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False)
X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.2, shuffle=False)
test = df.loc[X_test.index]

print(f"train: {len(X_train):,} matches, {df['MatchDate'].min():%b %Y} to {df.loc[X_train.index[-1], 'MatchDate']:%b %Y}")
print(f"test:  {len(X_test):,} matches, {test['MatchDate'].min():%b %Y} to {test['MatchDate'].max():%b %Y}")
train: 15,720 matches, Aug 2006 to Nov 2022
test:  3,930 matches, Nov 2022 to Sep 2026

Models¶

We compare 4 approaches in an attempt to find the best strategy. 2 methods are simple rules, while 2 methods are models.

We want to ensure the model actually is doing something useful which is more complex than following a simple rule, like, as we will see, always picking the home team.

  • Rule 1: Always pick a Home win since it is the most common result

  • Rule 2: Always pick the bookmaker's predicted win

  • Model 1: Logistic regression

  • Model 2: Logistic regression with class weights, to see if we can predict draws at all. A draw is almost never the single most likely result of a match, so plain logistic regression never picks one. Class weights make a wrong draw cost more during training, which pushes the model to predict some.[7][8][9][10] (We have not covered class weighted LR in class, but it was interesting and I wanted to try it)

Why Logistic Regression¶

  1. It gives a probability for each result instead of just a pick. This is what FotMob shows, and it lets us compare directly against the bookmaker's probabilities.
  2. Elo is built on a logistic curve. Thus, it already follows the same S-shape that logistic regression fits. EloDiff is also the feature closest to the bookmaker odds (0.94 correlation with BookH above).
  3. The weights are easy to read, so we can see which features push the prediction toward a Home win.
  4. Soccer results have a lot of luck in them. A simple model with a penalty on its weights is less likely to overfit.
In [22]:
# baselines:
# 1. always pick a home win
# 2. always pick bookmaker's predicted win

# accuracy of always picking a home win is just the share of home wins
home_acc = (y_test == 2).mean()

# encoding order: 0 = A, 1 = D, 2 = H
book_probs = test[['BookA', 'BookD', 'BookH']].to_numpy()
# highest col for match -> encoded result
book_pick = book_probs.argmax(axis=1)
book_acc = accuracy_score(y_test, book_pick)

# proportion of matches where the bookmaker favorite is a draw
draw_picks = (book_pick == 1).mean()

print(f'always home win:      {home_acc:.1%}')
print(f'bookmaker favorite:   {book_acc:.1%}')
print(f'bookmaker draw picks: {draw_picks:.1%}')

# the bookmaker is ~10% better than always picking a home win
# the bookmaker also almost never has a draw as the favorite
always home win:      44.8%
bookmaker favorite:   54.5%
bookmaker draw picks: 0.2%
In [23]:
# we create a pipeline to run 3 steps:
# 1. PolynomialFeatures to allow a higher degree polynomial
# 2. StandardScaler to scale data to a uniform scale
#    we opt for StandardScaler instead of MinMax, because MinMax would set an extreme data point to be 1, skewing scale
# 3. LogisticRegression predicts outcome probability
def lr_pipeline(degree, C, class_weight=None):
    poly = PolynomialFeatures(degree, include_bias=False)
    scale = StandardScaler()
    # high iterations to not stop early, we have a lot of data
    logreg = LogisticRegression(C=C, class_weight=class_weight, max_iter=2000)
    return make_pipeline(poly, scale, logreg)

# tune degree and C on the validation set
# C is the inverse of the penalty, so a smaller C gives a simpler model with smaller weights
# we fit on X_fit and score on X_val, the test set is not touched here

# we created a validation split from 20% of the training data
# we use it to tune the degree and the penalty of weights (C)
lr_scores = {}
for degree in [1, 2]:
    for C in [0.001, 0.01, 0.1, 1, 10]:
        model = lr_pipeline(degree, C)
        model.fit(X_fit, y_fit)
        val_pred = model.predict(X_val)
        lr_scores[(degree, C)] = accuracy_score(y_val, val_pred)

# pick degree and C with the highest accuracy
best_degree, best_C = max(lr_scores, key=lr_scores.get)

# table of validation accuracy
# dict -> Series with (degree, C)
lr_table = pd.Series(lr_scores)
lr_table = lr_table.rename_axis(['degree', 'C'])
# move C into the columns, so degree is the rows
lr_table = lr_table.unstack()
# convert to %
lr_table = (lr_table * 100).round(2)

print('Best degree: ', best_degree)
print('Best C: ', best_C)

# every combination is between 52% and 53%
Best degree:  2
Best C:  0.001
In [24]:
# logistic regression gives a probability for each result
# refit on the full training set with the best params from above, then score on test set
lr = lr_pipeline(best_degree, best_C)
lr.fit(X_train, y_train)
lr_pred = lr.predict(X_test)
plot_confusion(y_test, lr_pred, 'Logistic regression')

# 52.6% accuracy, ~2% less than the bookmaker
# the model never predicts a draw
# this aligns with the clusters

# lr[0] is PolynomialFeatures
# lr[-1] is LogisticRegression
poly_step = lr[0]
logreg_step = lr[-1]
# name of every feature after PolynomialFeatures
feature_names = poly_step.get_feature_names_out()

# coef_[2] is home wins
home_win_weights = logreg_step.coef_[2]
weights = pd.Series(home_win_weights, index=feature_names)

# sort by |size|, keep 10 largest
sorted_weights = weights.sort_values(key=abs)
top_weights = sorted_weights.tail(10).round(2)
top_weights

# every one of the top 10 weights includes EloDiff, thus Elo is most important
# Form doesn't make it in since Elo partially accounts for Form
# the weights are all small since C = 0.001 is a strong penalty
No description has been provided for this image
Out[24]:
0
EloDiff^2 0.04
EloDiff ShotOnTargetForAway 0.04
EloDiff ShotOnTargetAgainstAway 0.05
EloDiff Form5Away 0.05
EloDiff ShotOnTargetAgainstHome 0.05
EloDiff Division_E0 0.05
EloDiff ShotOnTargetForHome 0.06
EloDiff Division_SP1 0.07
EloDiff Form5Home 0.07
EloDiff 0.08

In [25]:
# logistic regression never predicts a draw, and the bookmaker almost never does either
# class_weight='balanced' weighs each result by the inverse of frequency
# thus, a wrong draw costs more than a wrong home win, which pushes the model to predict draws
lr_bal = lr_pipeline(best_degree, best_C, class_weight='balanced')
lr_bal.fit(X_train, y_train)
bal_pred = lr_bal.predict(X_test)
plot_confusion(y_test, bal_pred, 'Logistic regression, balanced')

# some draws are now predicted correctly but the accuracy drops to about 50%
# most of the draw predictions are wrong
# thus, a draw is close to unpredictable before kickoff
No description has been provided for this image

Example Predictions¶

We compare the model and bookmaker probabilities for the most recent test set match between 3 pairs of rivals:

  1. Arsenal vs Man City (Premier League)
  2. Real Madrid vs Barcelona (La Liga)
  3. Bayern Munich vs Dortmund (Bundesliga)
In [26]:
# (home team, away team)
games = [('Arsenal', 'Man City'), ('Real Madrid', 'Barcelona'), ('Bayern Munich', 'Dortmund')]

# most recent match in the test set for each pair, with the same team at home
last_matches = []
for h, a in games:
    pair = test[(test['HomeTeam'] == h) & (test['AwayTeam'] == a)]
    last_matches.append(pair.tail(1))
rows = pd.concat(last_matches)

# predict_proba gives the columns in our encoding order (A, D, H)
model_probs = lr.predict_proba(rows[FEATURES])
# flip the columns to H, D, A to match the bookmaker columns
model_probs = model_probs[:, ::-1]

# one subplot per match, with a stacked bar for the bookmaker and one for the model
fig, axes = plt.subplots(len(rows), 1, figsize=(10, 1.5 * len(rows)))
# loop over each subplot, match, and set of model probabilities together
# iterrows gives (index, row), we only need the row
for ax, (_, m), p in zip(axes, rows.iterrows(), model_probs):
    book_row = m[['BookH', 'BookD', 'BookA']].to_numpy(float)
    probs = pd.DataFrame([book_row, p], index=['Bookmaker', 'Model'], columns=order)
    probs.plot.barh(stacked=True, ax=ax, legend=False, color=list(RESULT_COLORS.values()),
                    edgecolor=BG_COLOR, linewidth=2, width=0.8)
    # print the probability inside each part of the bar
    for bars in ax.containers:
        ax.bar_label(bars, fmt='{:.0%}', label_type='center', color=BG_COLOR, fontweight='bold')
    ax.set_xticks([])
    ax.grid(False)
    title = (f"{m['HomeTeam']} vs {m['AwayTeam']}, {m['MatchDate']:%d %b %Y}, final score "
             f"{m['FTHome']:.0f}:{m['FTAway']:.0f}")
    ax.set_title(title, loc='left', fontsize=11)

# one shared legend for all the subplots
# each legend entry is a plain square in the result's color
handles = [plt.Rectangle((0, 0), 1, 1, color=c) for c in RESULT_COLORS.values()]
fig.legend(handles, RESULT_NAMES.values(), ncol=3, loc='lower center', frameon=False, bbox_to_anchor=(0.5, -0.04))
plt.tight_layout()
plt.show()

# the model and the bookmaker usually only differ by a few percentage points
# however, the bookmaker also uses injuries, lineups, and betting volume, which we don't have
No description has been provided for this image
In [27]:
# test accuracy for each model
lr_acc = accuracy_score(y_test, lr_pred)
bal_acc = accuracy_score(y_test, bal_pred)

# home_acc and book_acc are the baselines from above
scores = pd.Series({
    'Always home win': home_acc,
    'Logistic regression (balanced)': bal_acc,
    'Logistic regression': lr_acc,
    'Bookmaker favorite': book_acc,
})

# create a plot
fig, ax = plt.subplots(figsize=(8, 4.5))
# the 2 simple rules are grey and our 2 models are blue
colors = [NEUTRAL_COLOR, MAIN_COLOR, MAIN_COLOR, NEUTRAL_COLOR]

bars = ax.barh(scores.index, scores.values, color=colors, edgecolor=BG_COLOR, height=0.6)

ax.xaxis.set_major_formatter('{x:.0%}')
ax.set_xlim(0, 0.7)
ax.set_title('Test set accuracy by model / benchmark')

labels = [f'{v:.1%}' for v in scores]
ax.bar_label(bars, labels=labels, padding=4, fontweight='bold')

plt.tight_layout()
plt.show()
No description has been provided for this image

Conclusion¶

My Goal¶

I set out to try and predict the outcomes of a soccer fixture, using only data known from before kickoff. I compare this data with bookmaker odds, specifically Bet365, to see how close I could get to their predictions.

What I Learned About the Dataset¶

The raw dataset had nearly 240,000 matches from 38 different leagues. That is a lot of data, so I decided to focus on the 3 that are the most interesting: the Premier League in England, La Liga in Spain, and Bundesliga in Germany, all the top flights in their respective countries. After removing all the other leagues, I was left with about 20,000 fixtures and 123 clubs.

Regarding the quality of the data, it was pretty good out of the box. There were only a couple small fixes needed: fixing the 2 different club name spellings, and 2 Elo values were missing. There were also about 100 shot form values missing, mainly for clubs that were newly promoted, hence that data was probably not tracked in a lower division.

The most interesting part of the data is the home advantage that comes from having the club's fans at the stadium. The home team win rate is 44.8%, significantly higher than the away side's 30.3%. I see the home win rate drop significantly in the 2020 season, when fans were not allowed in the stadiums because of COVID. A global pandemic is definitely an outlier, thus I dropped that season in all the leagues.

The 3 classes were well balanced across the leagues, so I did not use a separate model for each league (or pick a single league to study).

Elo supplies the most information to my model. EloDiff had a 0.94 correlation with the bookmaker's home side win prediction, meaning the bookmakers lean heavily on that value.

The clusters that the KMeans clusterer found were very overlapped and their silhouette score was low. A draw is never the most common result in any of the clusters, which gives me an indication that they were the hardest to predict.

Model Effectiveness¶

My final numbers on accuracy across the 2 models and 2 baseline rules were as such:

Type Method Accuracy
Rule Bookmaker prediction 54.5%
Model Logistic Regression 52.6%
Model Logistic Regression (balanced) 48.7%
Rule Always picking a home win 44.8%

Plain logistic regression is the most effective model. It's nearly 8% more effective than just always picking a home side win, which means there is at least some pattern that the model was able to learn from the prematch data.

Unfortunately, my model's accuracy is 1.9% behind the bookmaker's accuracy. However, considering that the bookmakers aren't just using one dataset and 10 features, but full team lineups, news, injuries, and a ton of other data, I would say that it is not a bad result at all for my model.

The model also is very reluctant to predict a draw to be most likely. In almost every fixture, one team has at least a slight advantage, so a win is always more likely. The bookmaker also only predicts a draw 0.2% of the time, which means the intuition to not pick draws is correct.

What the Model Says¶

A team's Elo decided most matches. The top 10 weights always included EloDiff; form and shots on target only appear in combination with EloDiff, which makes sense considering that Elo partially encapsulates the recent form of a club.

I can also say that the model's predicted probabilities are realistic. The difference between its output and the bookmaker's are only different by a few percentage points.

Learnings about the Model¶

I learned that the settings barely matter on the model. Trying multiple weights and degrees only moves the accuracy of the model in a range between 52 and 53 percent, so a borderline insignificant effect. The simplest settings worked the best, with degree 2 and a C of 0.001.

Logistic regression was a good choice of model, since Elo is built on a logistic curve. The model also outputs probabilities, just like FotMob and Bet365, so it is very easy to compare results.

I compare with these sources because accuracy is a weird metric for soccer, even the bookmaker's predicted winner still loses 45% of the time. It just means that there is a clear limit to what I can predict with only prematch data.

I am not more accurate than the bookmaker, and I cannot be, because the bookmaker simply uses more data. They use injuries, news, squad lineups and the betting volume, which I can't include since my constraints are to only use one dataset. Soccer also just has a lot of luck involved, like any sport. Any cards, saved goals, fouls, etc., can change the course of a game very quickly.

There are, however, some limitations. Accuracy only checks the highest predicted probability against the outcome of the match, not how good the probabilities actually are, so something like a 51% home win chance prediction compared to a 99% one is graded the same. Also, since I did not have expected goals in the dataset, I decided to take the average of the last 5 games' goals for that team and use it as an indicator.

Overall¶

Overall, I am happy with the accuracy of my model and the outcome of this experiment. Getting within 2% of a professional bookmaker using only one dataset and using only prematch data shows that it is not only hard to predict the outcome at the final whistle, but that very simple features carry a lot of weight in trying to predict it.

References¶

Data¶

[1] Club Football Match Data (GitHub)

[2] Football-Data.co.uk

[3] ClubElo

Elo Ratings¶

[4] Elo rating system (Wikipedia)

Bookmaker Odds¶

[5] How to calculate implied probability in betting (Smarkets)

[6] Mathematics of bookmaking (Wikipedia)

Class-Weighted Logistic Regression¶

[7] How to Improve Logistic Regression in Imbalanced Data with Class Weights (Medium)

[8] Weighted Logistic Regression for Imbalanced Dataset (GeeksforGeeks)

[9] LogisticRegression (scikit-learn)

[10] compute_class_weight (scikit-learn)