Originally published in http://www.martinsbaseballblog.wordpress.com


Baseball’s back! Hooray! And with the return of our favorite pasttime, it’s time to start checking our off-season predictions to see how close we came to the actual result. Will Harper repeat as MVP? Will PECOTA actually predict the Royals true outcome? How many people will not win BP’s Beat PECOTA game? Forecasting and discussing these is a fun activity that helps us bear the absence of baseball. But, the problem with forecasting is that it not always predicts the correct outcome. There are many factors that can make us throw our projections out the window: injuries, trades, suspensions. But all this unpredictability is what keeps us updating our models and always trying to get our most accurate read of the upcoming season.

But, with all our work going into predicting the future, there are also caveats. Sample size, probability, regression, and fitting are also points that must be taken into consideration. Take Trevor Story’s debut last night: 2 for 4 with 2 HRs versus Zach Greinke. Though Rockies fans may be bullish on Story, PECOTA predicts 8 HR for the season with a .242 TAv [1]. Just because Story had a strong debut does not mean he beat PECOTA. He may yet regress; the sample is too small; and PECOTA works on probabilities of most likely outcomes. Knowing all this, we must then ask: at what point should we completely disregard our projections.

Personally, forecasts work as guidelines. They’re not a set of rules but a more akin to suggestions. Forecasts suggest what may happen – they give a slight idea of what could happen but not what is going to happen. Because of this, I like to keep projections close to me, so I can review them and check where my projections failed and where I succeeded. But if I failed my projection – either by having a prediction margin too big or too small – or I succeeded – which most of the time is just random luck – projections can always be discarded.

Now why am I ready to discard my projections even if I succeed? To answer this, I refer to you to the German philosopher Wilhelm Dilthey. Dilthey worked in a field known as hermeneutics: stating that truth or knowledge is completely related to our own personal subjective point-of-view. If knowledge is based on our subjectivism, then we have varying degrees of knowledge which we are either ready to fully disregard or we hold very dearly as one of the axiomatic truths of our lives. Though the theory of gravity is one of our axiomatic truths; forecasting isn’t. We are ready to disregard forecasts because we know that they are imperfect and we know that they can always be made better (or worse). Therefore, even if my projection succeeded in predicting the complete 2016 (or any future) season; they are not axiomatic truths. That’s why we must treat forecasting very delicatly and should not hold them close to our hearts.

I asked Ben Lindbergh of fivethirtyeight.com and Sam Miller of baseballprospectus.com [2] for their opinions on the subject. Sam warns of a projection trap since projections either conform or don’t – which anyway causes the reader to avoid giving credit to the forecaster. At the same time, Sam holds the same position I do: projections should be treated with skepticism. The caveat, he adds, is that though we must be skeptic, they must also be treated as fact because if we either favour or reject projections; then we are ignoring our models and stating that our models are flawed, therefore rejecting the starting premise [3].

I agree with Sam. Projections should be treated with skepticism. And we must also treat them as fact. I know it’s contradictory but if we do not treat them as fact, then we must completely disregard them from the start. And if we do that: then what’s the point of forecasting? We know they are not real; we know, at the same time, that if our model is correct, they may be close to real; and we know that because our model may be flawed, we must quickly disregard them.

Though projecting – either baseball, economics, politics, etc… – is a fun activity to be engaged in, we must remember that we are not owners of a crystal ball. Projections can’t tell us what will happen. If correctly calibrated [4], they may tell us – with varying degrees of probability – what might happen. This is what causes projections to be anything but axiomatic. This is what causes Sam and I to state that we must be skeptic towards them. This is what leads me to think that projections should be quickly discarded despite agreeing with Sam that they should be treated as fact.

So when should we discard projections? That depends on how bullish you feel with your model. Though I trust my models and the models we find on Fangraphs or Baseball Prospectus, I feel that I must be ready to disregard them as soon as I feel they have failed to live up what we experience in the real world: that  is, once we can empirically disprove them.

————————————————————————–

[1]. Dubuque, Patrick, Sam Miller and Jason Wjciechwoski – Baseball Prospectus 2016, p. 160

[2]. Ben Lindbergh and Sam Miller are the hosts of the Effectively Wild Podcast on Baseball Prospectus and I am very grateful towards them for answering my questions.

[3]. He then, tongue-in-cheek finishes off that he only listens to Wade Davis’ projections.

[4]. Something that must be constantly done and reviewed so that our models take into consideration the latest data.

Advertisements