What could be sillier than publicly recording pre-season predictions about player performances? Publicly reviewing the predictions post-season, that’s what.
Before the 2023-24 season began, I developed a process to predict the PR-Score (see Introduction to Productivity Ratings for details) of players and then shared a list of players who had the highest predicted scores, those who would get a PR-Elite or PR-Star rating.
To properly consider the accuracy of those predictions, it would be good to have an idea of how the very best players in the league normally progress. What happened in Year 2 when a player was a PR-Elite or PR-Star in Year 1?
Season-to-Season Progression
The best way to show season-to-season progression is with a Sankey diagram, and by no small coincidence, I like Sankey diagrams! Many thanks go to SankeyMATIC.com, which did the hard work of making these charts beautiful.
The first Sankey diagram shows how players who were rated PR-Elite in a preceding season (Y1) were rated in the subsequent season (Y2), based on data from 2007-08 to 2022-23.
PR-Elite seasons are hard to achieve and hard to maintain even the guys that do get to PR-Elite level don’t do it every year. On the other hand, when they slip from PR-Elite they usually “slip” down to PR-Star, which would be a pretty darn good season.
Now we can look at the players who had a PR-Star rating in a preceding season. Unlike the PR-Elite players, they have the opportunity to go up a category, and some of them did.
You will notice that the Year-2 red and orange boxes are smaller than they were in the PR-Elite Sankey diagram, showing that PR-Elite players were more likely to have a PR-Star or better subsequent season than would PR-Star players. While 82% of PR-Elite players had a PR-Star or better season the next year, only 51% of PR-Star players did that.
There is one factor that influences the performance a player has in a subsequent season which is not shown in these diagrams: age. Older players tend to decline while younger players tend to improve.
How the Predictions Were Made (In a Few Words)
I used Productivity Rating data from 2021-22 and 2022-23 to determine a “base prediction” for players, then made an adjustment to it based on the player’s age. Younger players, who tend to improve, had their base prediction increased. Older players, who tend to leave their turn indicators on, had their base prediction decreased.
Predicted PR-Elites
The Predicted PR-Elites table shows the players predicted to be PR-Elite in 2023-24, along with their 2023-24 PR-Category, PR-Score, and the number of games they played. They are sorted by position and then by 2023-24 PR-Score.
Of the 19 predicted PR-Elites, eight were PR-Elites in 2023-24 and eight were PR-Stars.
The Games Played column was included in this table as it helps explain why some players failed to earn high PR-Scores.
Sergachev and Hughes missed a significant number of games due to injuries.
Stutzle missed 10% of the scheduled games and played worse than expected when he did play, also because of an injury.
Predicted PR-Stars – Defensemen
It was impossible to nicely present the 57 predicted PR-Stars in one table (especially in the printable version of this article), so I used two tables: one for defensemen, one for forwards.
As with the PR-Elites who fell short of their prediction, the predicted PR-Star defensemen who fell short either had seasons that were a little below average (Darnell Nurse and Adam Larsson, for example) or they missed a number of games (Alex Pietrangelo and Brandon Montour, e.g.).
Predicted PR-Stars – Forwards
Six of the forwards who were predicted to play at the PR-Star level played at the PR-Elite level.
Sam Reinhart and Robert Thomas were true surprises at the PR-Elite level. If, before the season began, you truly thought that Reinhart was going to score more than 55 goals, I hope you put a few dollars on that.
The other four who hit PR-Elite this season were all predicted as PR-Stars because of the prediction process. Matthews and Kucherov each had one weak season in their previous two, Miller was a little old and Crosby was a lot old.
Frankly, predicting Matthews and Kucherov as PR-Stars should have raised some concerns about the process. I now think that it would be better to base the prediction on three seasons of data instead of only two.
Again, those who fell short of PR-Star either missed games or were having off seasons.
2023-24 Unicorns
Seven players achieved PR-Elite status in 2023-24 whose prediction was below the PR-Star level. Who are these unicorns? Can I learn anything from these unicorns that would improve the prediction process?
Unicorns 1, 2 and 3
Three unicorns can be dealt with as a group: Artemi Panarin (F, NYR, predicted PR-Score 7.97), Vincent Trochek (F, NYR, predicted PR-Score 7.97), and Noah Dobson (D, NYI, predicted PR-Score 7.92). The PR-Star category is for PR-Scores between 8.00 and 9.99, so these players just missed being predicted to play at the PR-Star level.
If I increased the predicted score by 1.5%, these three players would have been predicted as PR-Stars. But by increasing the prediction by 1.5%, a total of fourteen extra players would have been predicted as PR-Stars. Of those fourteen, only these three and Adrian Kempe had PR-Star or better seasons in 2023-24. The rest of them (Dylan Cozens, Jake Guentzel, Carter Verhaeghe, Aaron Ekblad, Brock Nelson, John Tavares, Jeff Skinner, Rasmus Andersson, Kyle Connor, J.T. Compher, and MacKenzie Weegar) were either PR-First5 or PR-Regular.
Modifying the prediction process so that these three unicorns would have been predicted as PR-Stars would have made the overall predictions less accurate.
Unicorns 4 and 5
MacKenzie Weegar and Victor Hedman were both predicted to be in the upper part of PR-First5, and both ended up in the very bottom part of PR-Elite (each with PR-Scores of 10.10).
Weegar played a full season and played a little better than expected. Hedman‘s 2023-24 season was one of resurgence; it was the third-best season of his career.
Unicorn 6
Evan Bouchard is a young lad whose statistics improved enormously in 2023-24. His Career Productivity Rating chart shows the improvement.
Obviously, it helps a player tremendously to share the ice with Connor McDavid and Leon Draisaitl. 2023-24 was a bit of a right-place-right-time situation for Bouchard.
It will be interesting to see how he does in the rest of his career.
Unicorn 7
The only things that Carlson and Bouchard have in common are that they are both defensemen and that neither was predicted to play at the PR-Star level.
Bouchard is at the start of his career; Carlson is long in the tooth. Bouchard’s recent seasons have been good; Carlson’s most recent season was disappointing. Bouchard is playing with two young and formidable forwards; Carlson plays with an aging, but still capable, Ovechkin.
Without the advantages that Bouchard had, Carlson put together one heck of a season. He was sixteenth in scoring among defensemen, eighth in blocked shots by defensemen, sixteenth in the league for penalty-kill time-on-ice and he had the highest time-on-ice-per-game in his career.
Maybe Carlson had a slight advantage over Bouchard given their teams and their teammates. Edmonton had a playoff position locked and had several useful defensemen, so they didn’t have to ride Bouchard too hard.
Washington was in a fight to make the playoffs and did not have some useful defensemen. Codi Ceci, Edmonton’s fourth-best defenseman, was better than any defenseman on Washington not named Carlson. The Capitals had no option but to play Carlson every second they could.
Changes Due to the Unicorns
I see nothing in what these seven players did in 2023-24 that would lead me to change how the predictions were made.
The first three just missed the cut-off for a PR-Star prediction. Changing the process so that they would have qualified would also have added eleven other players, ten of whom were below the PR-Star level in 2023-24. Overall, the predictions would have been less accurate.
Unicorns four and five played better than predicted, and that’s going to happen in any reasonable prediction model.
Unicorns six and seven played much better than predicted, one because of his age and teammates and the other despite his age and teammates. With about 1,000 players in the league, you can expect a couple of them will have huge improvements: how do you identify which players will have those huge improvements based on their data?
Predictions – By the Numbers
Season-to-season progression showed that 82% of players that had a PR-Elite season followed it up with a PR-Star or better season. 84% of the predicted PR-Elite players had a PR-Star of a better season.
Season-to-season progression showed that 51% of players who had a PR-Star season followed it up with a PR-Star or better season. 69% of the predicted PR-Star players had a PR-Star of better season.
These results lead me to believe that the predictions were quite accurate, in that the players identified by the process largely went on to have really good seasons.
Were the results perfect? No, but only a moron would expect that a set of predictions like this could be done with perfection.
Perfection cannot be achieved because: players get injured and don’t play, making it impossible for them to hit their prediction; players get injured and continue to play but play less effectively, making it difficult for them to hit their prediction; players have below average seasons; players get paired with lesser players than they had been paired with in the past; players have above average seasons; players get paired with better players than they had been paired with in the past; coaches change.
Summary
This review of last season’s player predictions leads me to conclude that the prediction process is reasonable. The main reasons a player falls short of a prediction are injuries and below-average play, while the main reasons a player exceeds a prediction are good teammates and better-than-average play.
If I were to change anything, it might be to use Value Ratings when determining the “base predicted PR-Score” for a player. VR-Score is based on three seasons of data.
Related Articles
Introduction to Productivity Ratings