Using Deserved Run Average

Baseball Prospectus officially released the new Deserved Run Average (DRA) this week, fresh with a new set of improvements, as always. The main site will have more information coming soon to highlight some of the specific methodological tweaks that were made for the latest DRA. In the meantime, the data are here to play with and analyze, and (arguably) the most exciting update made to the statistic is the inclusion of error bars for both DRA and (by extension) Wins Above Replacement Player (WARP). This is an exciting update because the work of Jonathan Judge and the Baseball Prospectus stats team are arguably opening the newest door of the so-called “analytics movement” to the public, and embracing a general statistical concept that ought to be discussed throughout the public: uncertainty.

On Method:
When I run Twitter chats from BPMilwaukee, one of the most curious things to my mind is that followers of BPMilwaukee will not necessarily support general BP stats work. No concerns there, really; it’s not necessary to “brand” MLB stats analysis, and indeed when one begins supporting stats-as-brands, that’s just as problematic as how so-called Old School stats like Runs Batted In or Earned Run Average are used in orthodox baseball discussions. No, what I find curious is the general idea that a stat like WARP or DRA is faulty because it is “made up,” which is presumably a concern because the BP stats team are extremely transparent about how the stats are constructed and also how (and why) they are changing. So folks actually know that DRA changes…which is different than how the vast majority of websites present baseball stats. What is problematic about this attitude about DRA is that it ignores how other statistics are merely “constructs” in the very same way that DRA is merely a construct, and it also trades in the murky waters of false certainty.

For the past two years, I have worked in Community Development and Economic Development positions while completing a professional urban planning and policy degree. I used to believe that I was a “stats” guy or an “analytics” guy, but I never quite understood the importance of what actual statistical analysis means until I was forced to reckon with my biases while training for economic analysis. Before I learned and studied stats, and was required to use them on the job, I thought the “numbers” were most important. While fields aligned with statistics are concerned in some sense with “numbers” and thus with producing “numbers-oriented results” (i.e., sometimes your boss really wants the results of your analysis), by far the most important elements of statistical analysis are “concept validity,” methodology, and uncertainty. What is most important about statistical analysis is process, it turns, out: how an analyst reaches a conclusion is much more important than the concluding numbers on their own, for it is only in light of outlining methodology, and explaining what is at stake with a certain measurement, that anyone (including a consumer of those numbers) could understand the numerical results of statistical analysis.

It’s ironic that many victories of the so-called “analytics movements” are now enshrined in their own dangerous orthodoxy, for what everyone seems to have forgotten is that even if the debate was about numbers, the original controversy was to convince the “Guards of Baseball Knowledge and Value” that there were legitimately different ways of thinking about the game and that that meant there were legitimately different measurements that could be presented. Somewhere along the line, we became obsessed with those measurements, rather than the process-oriented creed of focusing on how to think about baseball. This extends to statistical analysis, then, too: it is as though when many fans were convinced of the merits of WARP and other stats, they simply turned over the box containing ERA, RBI, etc., dumped out those contents, and stuck the new measurements into the box. That was never the point, and to the extent that many of us did not communicate the significance of process-oriented thinking about baseball stats, that was our problem (and I place myself in this camp, having only realized the significance of this issue over the last few years).

Anyway, “concept validity” is the most important thing that I have learned about statistical analysis, aside from clearly stating your uncertainty in proper terms. “Concept validity” is basically the extent to which the phenomena you’re trying to measure match the methods that you’re using to measure the phenomena. What should be inherent in this process is an understanding that as an analyst’s approach to measuring phenomena changes, so too should their results change; one need not hold the numerical results of analysis sacred, for if new empirical evidence emerges, methodological research unearths a better way to measure something, or a literature review reveals a better way to define a concept, there is nothing wrong with the analytical results changing.

So, keep this in mind when you’re thinking about why DRA has “changed.” DRA doesn’t “hate” anyone on your team, or love them. It is not a mark against DRA, or WARP, that the stat is consistently updated and changed, because that is a sign that its authors are attempting to reach that mark of “concept validity.” If it is the goal of DRA “to tease out the most likely contributions of pitchers to the run-scoring that occurs around them” and updated methodological approaches, or an updated understanding of pitching-related data, helps to accomplish that goal, revising the stat is a methodological strength. That said, I can understand that within a statistics field, one may have disagreements with some of the particular methodological approaches; but I don’t take any substance of that type of disagreement to dismiss the value of the overall methodological process of DRA.

This is why the new DRA is so important: it continues Baseball Prospectus’s commitment to presenting uncertainty (as has been done on Brooks Baseball, as one example) in publishing baseball statistics. Embrace this approach: so far as DRA is “made up,” it is made according to a methodologically sound process that upholds honest and transparent thinking about uncertainty.

DRA Values
One of the approaches to constructing DRA is to valuate the Run-value of pitching outcomes, and those outcomes are published by Baseball Prospectus. These elements are arguably more important than the DRA output itself, for these outcomes show the balance of a pitcher’s performance: is a pitcher saving runs during hits, balls not in play (e.g., Home Runs, strike outs, walks, etc.), or outs on balls in play?

My favorite Brewers pitcher, Zach Davies, is a “casualty” of the new DRA (h/t to Kyle Lesniewski for beating me to this realization). But we’re not going to say, “DRA hates Zach Davies.” On the contrary, it is possible to see that from Davies’s Out Runs (-1.4), Not In Play (NIP) Runs (1.9), Hit Runs (1.4), and Framing Runs (-0.1) that Davies is not getting the job done in terms of limiting runs when the ball isn’t in play, and he’s not limiting runs that occur on hits, either. Here’s how the 2018 Brewers look, sorted by NIP Runs (Josh Hader is real!):

These run elements help to define DRA. At this point in the season, however, it’s important to note just how large the Standard Deviation appears for DRA. For example, Davies’s DRA is currently published at 6.02, but with a standard deviation of 1.00, approximately 70 percent of the time, Davies could be expected to land between 5.02 DRA and 7.02 DRA. Tracking DRA with RA9 (Runs Allowed per 9 IP), something like a 5.02 RA9 gets Davies into respectable rotation territory, and there’s no telling that the righty could also prevent runs to a greater extent (i.e., serve as an even greater outlier).

Here are Brewers starters by variation, sorted by lowest Standard Deviation.

Now, let’s repeat this measurement with WARP, which should help to underscore the extent to which fans should quote Replacement Level stats with certainty. Doesn’t this make you wonder what the error bars might be on Baseball Reference or FanGraphs WAR? Hopefully those websites follow suit and publish WAR error bars where possible.

What I find extremely interesting about this exercise is the extent to which the Brewers starting pitchers exhibit variation in their potential WARP production. Almost to a man, the Brewers remaining rotation (after Brent Suter was moved to the bullpen to make room for Wade Miley) could range anywhere from replacement level to solid rotation piece (for reference, among 149 pitchers with 17.0 IP or higher, 0.34 WARP is a median 2018 performance thus far). This will be a stat worth watching for the remainder of 2018.

Finally, the last stat worth watching is whether the Brewers can continue to out perform their DRA. For my last publication on Runs Prevented, the Brewers as a pitching staff were approximately 18 runs better than their DRA suggested. My hypothesis here is that the Brewers groundball efficiency machine is leading this charge, but that could be one of many explanations including random luck.

These statistics provide a wide range of tools for Brewers fans and analysts. Ranges of DRA and WARP can be compared in order to assess both uncertainty and potential overlapping fields of value. To my mind, the best aspect of this new presentation is that fans and analysts no longer need to feign false certainty over WARP, and this is great; one shouldn’t need to say “Zach Davies has a 6.02 DRA” right now, when one can say “Davies’s DRA ranges from 5.02 to 7.02.” This exercise can be repeated throughout the season, and perhaps through embracing uncertainty we can find better hypothesis about how and why a team is under-performing (or over-performing) their peripheral stats or DRA estimates.

Photo Credit: Patrick Gorski, USA Today Sports Images