What’s Runs Prevented? Flexible Elite Roles ||  Exploring Runs Prevented || Aces Do Not Exist || Rotation Spots || Third-Time Charmers

Here’s how the Brewers pitching staff looks when ranked by Average Runs Prevented. For additional context, each pitcher’s Games, Games Started, and Innings Pitched statistics are included.

Stats: 

Average Runs Prevented is the average of park-adjusted, league-adjusted estimates of a pitcher’s actual runs allowed compared to their expected runs allowed.

DRA Runs Prevented is the difference between a pitcher’s expected runs allowed and their DRA performance. 

Direction is the “Direction of Change” between a pitcher’s 2018 Average Runs Prevented and 2018 DRA Runs Prevented.

G is “Games” (total appearances); GS is “Games Started” (total starts); IP is “Innings Pitched.”

Why is this important? DRA is a pitching statistic that estimates each pitcher’s performance based on numerous contextual factors. DRA is a statistic that can describe a player’s performance on the field by correlating Runs Allowed per 9 IP (RA9) to DRA; it is modeled to consistently assess a player’s performance year-to-year; and it is modeled to predict next year’s RA9. Runs Prevented, on the other hand, is a purely descriptive statistic, simply aiming to measure the extent to which a pitcher compares to their park and league environments.

Before we get into the extended analysis, if you’d like to know why this topic is important, consider the following questions; for fun, the exercise could also end here, as there’s a lot to think about with this staff.

….which of these pitchers would you expect to improve in 2019?

….which of these pitchers would you expect to improve in 2019?

….which of these pitchers would you expect to improve in 2019?

By describing Runs Prevented and DRA Runs Prevented statistics year-over-year, it is possible to understand the absolute volatility of pitching performance. DRA is also potentially a tool that can be used to set someone in the right direction for analyzing statistical profiles in order to project improvement or decline.

Let’s take a look at MLB pitchers that worked in 2017 and 2018:

The value of using a statistic such as DRA is that the year-to-year Runs Prevented performance by MLB pitchers is absurdly volatile. The table above demonstrates the absolute value of change in several key statistics for pitchers that worked in both 2017 and 2018. 639 MLB pitchers worked in both 2017 and 2018 seasons. On the whole, this group was quite volatile, with the average change in runs prevented moving by nine runs prevented (positive or negative); a pitcher that worked in both 2017 and 2018 also saw their innings pitched total fluctuate between 33 and 34 innings, and their average games started fluctuate by four. Focusing specifically on starters (i.e., pitchers who started a game in 2017 and pitchers who started a game in 2018), the fluctuations are even wider.

These fluctuations would be the equivalent of Jhoulys Chacin becoming a slightly below average, slightly smaller workload pitcher in 2018, or improving steadily into “ace” territory; Freddy Peralta expanding into a more regular rotation role, or stepping back into a smaller replacement role; Corey Knebel fluctuating to a below average reliever or recovering his excellent high leverage form; or Josh Hader becoming “just” an average reliever or taking the next step in his high leverage ace development. These are just a few examples of the real impact that typical run prevention fluctuations can cause to a team. Each of these pitchers are likely to remain under Brewers contractual control in 2019, so it matters how their performances change.

Let’s dig deeper into that group of 639 pitchers that worked in both 2017 and 2018 to assess the descriptive value of DRA and Runs Prevented. A couple of caveats are in order. First, this is a biased analysis, insofar as I am expressly limiting my search to players that worked in both 2017 and 2018, which excludes a “true talent assessment” of players that missed either of those seasons for a multitude of reasons (from player development, such as Freddy Peralta, to injury, such as Jimmy Nelson). Second, since I will be describing the general direction of DRA, I am not using statistical methods to assess the significance of DRA’s predictions. With these caveats in mind, I think it remains useful to see how DRA assesses players within a single season, and across two seasons.

Now that we’ve discussed the average absolute value of Runs Prevented change between 2017 and 2018, let’s take an overview of this group of pitchers in terms of improvement or decline. Excluding pitchers with Runs Prevented totals between -2 and 2 in 2017 and 2018, which represents a relatively minimal range of fluctuation that could simply be explained by park factors or league environment, more pitchers declined than improved between 2017 and 2018. In many cases, these changes were quite major, as 132 pitchers declined by 10 or more Runs Prevented, while 53 pitchers improved by 10 or more Runs Prevented. The overall magnitude of major declining performances ensured that this group of 639 pitchers was -439 Runs Prevented (!!!) between 2017 or 2018; this means that if each 2017 team retained these pitchers, on average they would have been expected to lose approximately 44 more games (as a group) in 2018, all else held equal.

Based on 2017 performance, could anyone have predicted these directions of change among these pitchers? Once I assembled an Average Runs Prevented analysis of the 2017 MLB season, and isolated pitchers that worked in both 2017 and 2018, I analyzed several aspects of each player’s performance:

• I analyzed the 2017 Direction of Change, which is the change between 2017 Runs Prevented and 2017 DRA Runs Prevented, in order to assess whether a player overperformed or underperformed their DRA.

• I analyzed the 2017 Direction of Change and the difference between 2018 Runs Prevented and 2017 Runs Prevented, in order to assess whether a player’s between-seasons change (2017 to 2018) matched their 2017 underperformance or overperformance. Focusing on 2017 Direction of Change and between-seasons change is one way to describe the types of projections made by DRA.

• I assessed 2017 DRA Runs Prevented and 2018 DRA Runs Prevented in order to determine whether the statistic consistently estimated a pitcher’s contextual performance.

First and foremost, in terms of 2017 and 2018 consistency, DRA consistently assessed 389 pitchers as either Above Average or Below Average in both 2017 and 2018. Since the “Other Pitchers” group is quite a set of outliers, I provided a couple of key statistics about their DRA Runs Prevented.

In this case, the “Other” group is comprised of outliers, including pitchers like Wade Miley and Chase Anderson, as well as Lucas Giolito, Kyle Freeland, and Derek Holland, among others. This is an area where the biased selection of this group of pitchers could impact analysis, as developments such as a new pitch (by Miley) or backed-up stuff and command (by Giolito) create role discrepancies that would be difficult to predict without granular scouting information. Of course, these are precisely the types of uneven player development facts that teams attempt to exploit. Wade Miley was not a particularly good pitcher in 2018, indeed he could have reasonably been replaced (which is partially why he was available for a minor league contract entering 2018); his development to an average pitcher was worth 44 DRA Runs Prevented between 2017 and 2018, a massive improvement that is going to skew nearly any sample of players.

On the whole, it is worth noting that DRA Runs Prevented tracked better than Average Runs Prevented, in terms of absolute value of change, between 2017 and 2018. Among pitchers that worked in both seasons, DRA Runs Prevented fluctuated by approximately 8 runs, compared to approximately 9 runs by Average Runs Prevented. Not bad!

How does DRA work with this group of pitchers in terms of predicting the general direction of change between 2017 and 2018? Based on a pitcher’s internal 2017 difference between DRA and Runs Prevented, that pitcher’s typical improvement or decline between 2017 or 2018 matched the overperformance or underperformance (in terms of 2017 DRA versus 2017 Runs Prevented). DRA correctly assessed a pitcher’s expected performance change in 79 percent of cases:

Within this group of pitchers, DRA performs quite well in terms of assessing the actual size of the Runs Prevented change, as well as the direction. Once I categorized pitchers into groups of players that had Predicted Improvement, Predicted Decline, or some Other Prediction, I compared the change between 2017 and 2018 DRA Runs Prevented to 2017 and 2018 Actual Runs Prevented:

It should be underscored that this is a descriptive account of DRA’s predictions, rather that a statistical test of the significance of DRA’s predictions. Still, what is incredibly impressive about DRA is just how strong the statistic is in anticipating the shape of the run environment, and understanding the wide variance that can occur year over year.

What is interesting is that, according to DRA, the Brewers pitching staff was indeed better than average in 2018. However, there are 22 pitchers from that staff that might reasonably be expected to post a notable improvement or decline in 2019, if one assesses the extent to which they outperformed or underperformed their 2018 DRA. Thus, it is worth repeating the questions about who might be expected to improve in 2019, for even if the overall direction of the club’s pitchers may be expected to stay the course, their shape and distribution of Runs Prevented can all but expected to look quite different.

Baseball Prospectus’s DRA is the best and most recent attempt to evaluate pitchers.  As its name (“Deserved Run Average”) suggests, however, it has a backward-looking component.  It attempts to apportion credit and blame for what happened while a pitcher was on the mound.  Context-based Fielding Independent Pitching (cFIP), which is another recent BP statistic, attempts to measure only true talent.  (A summary of DRA and cFIP are available here.)  Both are valuable resources, but they measure pitcher effectiveness in different ways.

Chase Anderson was good last year by any publicly available measurement, though.  His ERA was 2.74, his FIP was 3.59, his cFIP was 93 (where 95-to-105 is average, and a lower number is better), and his DRA- was 81.9 (here 100 is average, with a lower number better).  These were all career-best marks, so there were two possible interpretations: either Anderson had developed and taken a step forward, or 2017 was one of those unrepeatable career years that players occasionally have.

In 2018, Anderson has not been as good.  All of his cumulative numbers have regressed, and both his DRA- and cFIP are worse than his career average.  All of the public pitching stats agree that he has not been good, but they all agreed that he was good last season.  Both the forward-looking stats and backward-looking stats agree on this.

It is this conundrum I find the most interesting.  In broad strokes, we can break pitching metrics down into two categories: forward-looking and backward-looking.  FIP and cFIP are prospective, while ERA is retrospective.  DRA is somewhere between the two, but seeks to explain past performance.  And both sets believed Anderson was good last year in terms of underlying performance and run prevention.

But that performance has not carried over into this year.  The traditional regression examples are pitchers who have good ERAs but bad cFIPs or DRAs, which indicates that they just got lucky and there was no uptick in performance.  Those pitchers are expected to not be as good the next year.  By contrast, good peripherals (as taken into account by cFIP) that match good run prevention numbers are supposed to indicate that someone is able to sustain that performance going forward.  Anderson breaks that model, though.

There are possible explanations for this that don’t require metrics to have missed.  Anderson could be pitching while hurt this year, or he could have made a mechanical adjustment that has not worked.  But we don’t know whether that has occurred, so I am assuming he isn’t dealing with any physical issues beyond the general fatigue we expect big league players to battle through.  He could also just be an outlier in these numbers; a sample size of one is insufficient to draw overarching conclusions about the validity of the stats, and that is not what I am attempting to do here.

Anderson provides the fulcrum for this discussion, but he is just a part of a larger question about how to measure pitcher performance.  There has been increased focus on how accurate our defensive metrics are because they have not kept up with shift tendencies, but pitching, although not seen as reliable as offensive numbers, has not received similar scrutiny.  I don’t believe anyone is suggesting that the current pitching metrics are perfect, but I wonder what else is missing that can be incorporated to help fix this type of blind spot.

The most basic analysis of Anderson’s season last year would have been that it was just a career year and he was likely to return to being the type of fourth starter he had been previously.  DRA, cFIP, and similar metrics provided possible justifications for Anderson having made substantive improvements that would carry forward.  To this point, though, those possibilities have not panned out.

I do not intend to just point to pitchers who public metrics cannot comprehend.  Instead, Anderson demonstrates a specific phenomenon: he had a career year last season, but it was backed up by improved peripherals in such a way that we don’t normally see in fluky performances.  The statistics that try to remove luck from the equation thought that Anderson had improved, but, at least to this point, it appears he had not.

FIP and DRA both seek to improve on Earned Runs Average (ERA) and Runs Average (RA9) as a way to evaluate pitchers.  The components of the two are different. For example, FIP uses the three true outcomes (strike outs, walks, and home runs), while DRA’s formula is more complex, and FIP is scaled to ERA while DRA is scaled to RA9. Both stats attempt to produce an ERA-scale number that provides a “better” approximation of a pitcher’s performance, and research over the last year addresses the descriptive and predictive outlooks of both stats. Additionally, recent Baseball Prospectus research into MLB Advanced Media demonstrates predictive characteristics of FIP and DRA compared to popular Statcast “x” metrics.  FIP and DRA can be compared with their “traditional” counterparts to determine whether a pitcher has gotten lucky, or has some other ability to control the type of contact hitters are generating.

In the last twenty years, 241 starters have made at least 100 appearances.  Of those 241, only 14 have posted an ERA at least ten percent lower than their FIP. As you may have guessed, Anderson is one of those fourteen pitchers.  There is no particular pattern to the caliber of names on the list; they range from as successful as Kyle Hendricks (2.96 ERA) to as unsuccessful as Armando Galarraga (4.78 ERA).  Anderson is right in the middle of this list, as his career 3.92 ERA ranks eighth.

There is, however, a pattern to the type of skillset that appears on this list; the highest BABIP allowed on this list is .288, which is lower than the league average has been at any point during these twenty years.  I don’t think this is a huge surprise, but it does confirm an intuitive understanding of the relationship between peripherals and run prevention.  Pitchers who give up fewer runs than their three true outcomes would suggest are better at minimizing hard contact.

This relates to Anderson because it tells us about the indicators of his success.  There have been a multitude of factors this year; as described above, his command has deserted him and his walk and strikeout rates have inverted.  His messy 4.42 ERA is, for better or worse, the main story.  In 2017, his ERA was 2.74, and the Brewers were counting on him to replicate at least most of that performance.  He has been unable to do that.

Weirdly, Anderson’s BABIP this season is the lowest of his career (.219).  Given the ballooning ERA and poor walk and strikeout rates, I would have expected him to be getting hit particularly hard.  That does not seem to be the case, however, and what we should take from that is uncertain.  On the one hand, his low BABIP seems to be driving his ability to outperform his FIP/DRA, which is a positive given that that ability is a calling card of his success.  On the other hand, it may be the only reason his performance has been somewhat respectable, and any type of BABIP regression could see his season balloon out of control.

Going forward, this gives us another aspect of Anderson’s performance to focus on.  His walk and strikeout rates will get the headlines because they are the most obvious and most visible, but his BABIP is interesting as well.  He is a FIP- and DRA-beater, but he does that despite not having ridiculously low BABIPs (career .285).  If the low BABIP in 2018 is covering his inability to outpitch his peripherals, then any type of regression in his batted ball luck could be disastrous.  We will just have to wait and see.

Photo Credit: Benny Sieu, USA Today Sports Images

On Method:
When I run Twitter chats from BPMilwaukee, one of the most curious things to my mind is that followers of BPMilwaukee will not necessarily support general BP stats work. No concerns there, really; it’s not necessary to “brand” MLB stats analysis, and indeed when one begins supporting stats-as-brands, that’s just as problematic as how so-called Old School stats like Runs Batted In or Earned Run Average are used in orthodox baseball discussions. No, what I find curious is the general idea that a stat like WARP or DRA is faulty because it is “made up,” which is presumably a concern because the BP stats team are extremely transparent about how the stats are constructed and also how (and why) they are changing. So folks actually know that DRA changes…which is different than how the vast majority of websites present baseball stats. What is problematic about this attitude about DRA is that it ignores how other statistics are merely “constructs” in the very same way that DRA is merely a construct, and it also trades in the murky waters of false certainty.

For the past two years, I have worked in Community Development and Economic Development positions while completing a professional urban planning and policy degree. I used to believe that I was a “stats” guy or an “analytics” guy, but I never quite understood the importance of what actual statistical analysis means until I was forced to reckon with my biases while training for economic analysis. Before I learned and studied stats, and was required to use them on the job, I thought the “numbers” were most important. While fields aligned with statistics are concerned in some sense with “numbers” and thus with producing “numbers-oriented results” (i.e., sometimes your boss really wants the results of your analysis), by far the most important elements of statistical analysis are “concept validity,” methodology, and uncertainty. What is most important about statistical analysis is process, it turns, out: how an analyst reaches a conclusion is much more important than the concluding numbers on their own, for it is only in light of outlining methodology, and explaining what is at stake with a certain measurement, that anyone (including a consumer of those numbers) could understand the numerical results of statistical analysis.

It’s ironic that many victories of the so-called “analytics movements” are now enshrined in their own dangerous orthodoxy, for what everyone seems to have forgotten is that even if the debate was about numbers, the original controversy was to convince the “Guards of Baseball Knowledge and Value” that there were legitimately different ways of thinking about the game and that that meant there were legitimately different measurements that could be presented. Somewhere along the line, we became obsessed with those measurements, rather than the process-oriented creed of focusing on how to think about baseball. This extends to statistical analysis, then, too: it is as though when many fans were convinced of the merits of WARP and other stats, they simply turned over the box containing ERA, RBI, etc., dumped out those contents, and stuck the new measurements into the box. That was never the point, and to the extent that many of us did not communicate the significance of process-oriented thinking about baseball stats, that was our problem (and I place myself in this camp, having only realized the significance of this issue over the last few years).

Anyway, “concept validity” is the most important thing that I have learned about statistical analysis, aside from clearly stating your uncertainty in proper terms. “Concept validity” is basically the extent to which the phenomena you’re trying to measure match the methods that you’re using to measure the phenomena. What should be inherent in this process is an understanding that as an analyst’s approach to measuring phenomena changes, so too should their results change; one need not hold the numerical results of analysis sacred, for if new empirical evidence emerges, methodological research unearths a better way to measure something, or a literature review reveals a better way to define a concept, there is nothing wrong with the analytical results changing.

So, keep this in mind when you’re thinking about why DRA has “changed.” DRA doesn’t “hate” anyone on your team, or love them. It is not a mark against DRA, or WARP, that the stat is consistently updated and changed, because that is a sign that its authors are attempting to reach that mark of “concept validity.” If it is the goal of DRA “to tease out the most likely contributions of pitchers to the run-scoring that occurs around them” and updated methodological approaches, or an updated understanding of pitching-related data, helps to accomplish that goal, revising the stat is a methodological strength. That said, I can understand that within a statistics field, one may have disagreements with some of the particular methodological approaches; but I don’t take any substance of that type of disagreement to dismiss the value of the overall methodological process of DRA.

This is why the new DRA is so important: it continues Baseball Prospectus’s commitment to presenting uncertainty (as has been done on Brooks Baseball, as one example) in publishing baseball statistics. Embrace this approach: so far as DRA is “made up,” it is made according to a methodologically sound process that upholds honest and transparent thinking about uncertainty.

DRA Values
One of the approaches to constructing DRA is to valuate the Run-value of pitching outcomes, and those outcomes are published by Baseball Prospectus. These elements are arguably more important than the DRA output itself, for these outcomes show the balance of a pitcher’s performance: is a pitcher saving runs during hits, balls not in play (e.g., Home Runs, strike outs, walks, etc.), or outs on balls in play?

My favorite Brewers pitcher, Zach Davies, is a “casualty” of the new DRA (h/t to Kyle Lesniewski for beating me to this realization). But we’re not going to say, “DRA hates Zach Davies.” On the contrary, it is possible to see that from Davies’s Out Runs (-1.4), Not In Play (NIP) Runs (1.9), Hit Runs (1.4), and Framing Runs (-0.1) that Davies is not getting the job done in terms of limiting runs when the ball isn’t in play, and he’s not limiting runs that occur on hits, either. Here’s how the 2018 Brewers look, sorted by NIP Runs (Josh Hader is real!):

These run elements help to define DRA. At this point in the season, however, it’s important to note just how large the Standard Deviation appears for DRA. For example, Davies’s DRA is currently published at 6.02, but with a standard deviation of 1.00, approximately 70 percent of the time, Davies could be expected to land between 5.02 DRA and 7.02 DRA. Tracking DRA with RA9 (Runs Allowed per 9 IP), something like a 5.02 RA9 gets Davies into respectable rotation territory, and there’s no telling that the righty could also prevent runs to a greater extent (i.e., serve as an even greater outlier).

Here are Brewers starters by variation, sorted by lowest Standard Deviation.

Now, let’s repeat this measurement with WARP, which should help to underscore the extent to which fans should quote Replacement Level stats with certainty. Doesn’t this make you wonder what the error bars might be on Baseball Reference or FanGraphs WAR? Hopefully those websites follow suit and publish WAR error bars where possible.

What I find extremely interesting about this exercise is the extent to which the Brewers starting pitchers exhibit variation in their potential WARP production. Almost to a man, the Brewers remaining rotation (after Brent Suter was moved to the bullpen to make room for Wade Miley) could range anywhere from replacement level to solid rotation piece (for reference, among 149 pitchers with 17.0 IP or higher, 0.34 WARP is a median 2018 performance thus far). This will be a stat worth watching for the remainder of 2018.

Finally, the last stat worth watching is whether the Brewers can continue to out perform their DRA. For my last publication on Runs Prevented, the Brewers as a pitching staff were approximately 18 runs better than their DRA suggested. My hypothesis here is that the Brewers groundball efficiency machine is leading this charge, but that could be one of many explanations including random luck.

These statistics provide a wide range of tools for Brewers fans and analysts. Ranges of DRA and WARP can be compared in order to assess both uncertainty and potential overlapping fields of value. To my mind, the best aspect of this new presentation is that fans and analysts no longer need to feign false certainty over WARP, and this is great; one shouldn’t need to say “Zach Davies has a 6.02 DRA” right now, when one can say “Davies’s DRA ranges from 5.02 to 7.02.” This exercise can be repeated throughout the season, and perhaps through embracing uncertainty we can find better hypothesis about how and why a team is under-performing (or over-performing) their peripheral stats or DRA estimates.

Photo Credit: Patrick Gorski, USA Today Sports Images