Baseball Prospectus is, in general, at the forefront of ball-tracking systems, and Wayne Boyle, Sean O’Rourke, Jeff Long, and Harry Pavlidis have spent countless hours researching and writing about the the technology, and flaws inherent in all current ball-tracking systems. Earlier this year they wrote an outstanding piece about the various issues that baseball would face if it implemented robot umpires, from lack of accuracy, to the time it takes to produce a reliable result, to the impact on game flow. They conclude that while the various technology is amazing, and improving all the time, that:
“With all the potential problems and complications of implementing a robo strike zone it seems that, instead of a robo zone, an unbiased and skilled umpire is still our best choice.”
This result may surprise some, who see the various network ball-and-strike trackers and assume they are gospel, but in reality, the error margins that still exist in those systems are every bit as important as the error margins suffered by human umpires.
METHOD:
In order to prove this point, Baseball Prospectus Milwaukee began a project in early February to build a state of the art robot umpire. We took several old PCs we had laying around, the RAM and SSD from a gaming PC used primarily for indie adventure games, Sega Genesis emulation, and the Jackbox Party Pack, and the best animatronics available from an old abandoned mouse-themed pizza restaurant. Combining these into a plausible human robot umpire was difficult, and the we probably should have considered the challenge in creating a robot capable of passing the Turing Test before we attempted any such thing, as it is actually far far more difficult than creating a system to call balls and strikes. Undeterred, we emerged from out lab with what we call the:
Axis
Neuronet
Gantry
Evaluation
Locomaton
ANGEL went through rigorous training in the ways and customs of humans, using a combination of regressive simulation and machine learning to understand the general tenor and concepts of human interaction. Moreover, the systems that allowed him to process human language in real time also allow him to calibrate his pitch-reading algorithms on the fly. In theory, ANGEL should have given us the best of both worlds, providing PitchFX accuracy quickly and accurately, backed by the mannerisms and judgement of something close to a human umpire. Instead, things went wrong almost immediately.
TEST RESULTS
The First
We got off to a rousing start as ANGEL behaved exactly as expected, getting into a proper crouch when cued by the motion of the pitcher, and making a slightly robotic, but plausible strike call. He stays within his own vertical plane most of the time, which can be slightly off-putting, but it works. Adding flair to his arm motion was critical to establishing his humanity, however we would see later that this unnecessary motion may have caused some problems with the bot’s ability to re-calibrate.
The Second
We started seeing some issues in the second and became worried. Once ANGEL is deployed we cannot do any patching or fixes until the game is over as the unit would just be staring off into space for a bit while the fixes were implemented. ANGEL has specific protocols every inning, including use of the ball/strike “clicker.” Here, that programming failed and the unit was forced to make adjustments. His pocket seems to interrupt his actions and instead of drawing the clicker up to his field of vision, the unit glances down. You can see the bot rocking back and forth attempting to re-establish his baseline readings.
Any kind of out-of-protocol machine-learning can have knock-on effects and we hypothesize that this started an inescapable cascade of bugs. We were terrified of what might happen going forward.
The Third
Until the third ANGEL’s actual calls weren’t too bad. There wasn’t much outside of the norm and the game was pretty clean, but two plays had a clear effect on the bot’s physical makeup, which accelerated the code breakdown we witnessed in the 2nd. This pop-up…
…did a number on ANGEL’s head flexibility, balance, and servo motors. While ANGEL is programmed to umpire his position, including plays at home and in the field, vertical popups can still cause issues as the bot’s head fires backwards. The results were immediately obvious. Take a look at ANGEL on this pitch low and off the plate.
He doesn’t stop tracking the ball when most humans would, instead following it into the mitt, through the recovery, and all the way back to the pitcher. ANGEL is supposed to disengage at some point to “count” the ball or strike, re-calibrate, and reset. Instead his tracking stays locked on, and a proper calibration never happened. We didn’t really expect this experiment to work, but we didn’t foresee this many issues this early on, and we were now officially worried if the experiment would be discovered before the game finished.
The Fourth and Fifth
Things stabilized in the 4th, and for the first time all day our optimism returned. If ANGEL gets too far out of whack he’s supposed to restart and bring his systems back into equilibrium. In the 3rd we detected a slight drift of the strike zone to ANGEL’s right, but he nailed this called ball.
And this high, borderline pitch was also correctly called a ball as well.
The opposing pitches seemed to confirm that ANGEL was back in action and we were all very excited, but then, disaster struck.
The Sixth
We spoke earlier of ANGEL’s strike call motion, and the challenge of providing him with a motion that doesn’t let on his mechanical existence. After quite a lot of deliberation on the subject we decided to keep it simple, and mechanically balanced. It’s a fluid motion, and intended to be easy on joints and motion sensors. On this day it was not simple enough. Here we see ANGEL pop up after giving the pitcher a low, outside strike call.
It was the last such call that would occur as the quick move of ANGEL’s head jarred his balance and pitch sensors, and, from that point out, shrunk the zone. There would be several pitches in this exact location, or better throughout the rest of the game, but ANGEL simply could not read them right. It was perhaps his most robotic move of the night as his head remains exactly forward-facing, and his body remains square to home plate. Our theory is that the move locked certain joints in his body and neck while throwing off his internal gyroscopic detection and balance control (IGDABC).
As ANGEL’s perceptions are reduced he still calls things as he sees them, he just can’t perceive information about the ball’s location on the outside while maintaining a fixed head position. If he moves his upper body to track the ball, it skews his vertical and horizontal readings. At this point, ANGEL was shot.
The Eighth
The 7th saw quite a few swings, which protected ANGEL from immediate exposure, but with the entry of Josh Hader into the game, everything became pretty clear pretty quickly. Hader excels at dotting the corners, and with the corners now gone from ANGEL’s field of vision, Hader was going to struggle as much as Hader can ever struggle.
He attempted to work both sides of the plate and threw several pitches exactly on the edges.
While a normal human umpire may have called some of these strikes and some of these balls, ANGEL called all of them balls as his zone continued to shrink. Several of these pitches had been strikes earlier in the day and both Hader and Manny Pina knew this, and became increasingly perturbed. Hader escaped both of his innings due to stellar play by the defense, including a diving catch by Lorenzo Cain and an outstanding throw from Pina to cut down the speedy Jarrod Dyson, but the Brewer bench and Closer Corey Knebel had seen what was happening, and the situation we feared finally arose.
The Ninth
At this point we were just hoping that the Diamondbacks would take a few bad swings and let us escape from this potentially embarrassing situation, but it was not to be. Knebel’s first pitch, a high outside curveball, put on display just how poorly ANGEL was following the ball. Where he tracked the ball far too much a few innings ago, his head and tracking were now completely off, as he never even glanced over, and stood right up.
On the pitch before it all went wrong, ANGEL actually appeared to make the correct call, but it was very close, and had been called a strike routinely earlier.
Manny Pina, I think at this point suspected it was a robot, and decided to engage in conversation. ANGEL is programmed to shut down conversation as he would be quickly discovered with any prolonged interaction. When ANGEL bent over to confront Pina, the last of his outside detection blew, which set up the next pitch.
The Incident
There are a few things going on here. It looks possible that ANGEL is fixed on Pina’s head instead of the ball, throwing off any perception he may have had left. Pina did himself no favors with a poor framing job, but one of the purposes of robots is to eliminate this subjective nonsense. ANGEL sees Pina and Knebel reach for their heads which triggers his mimicry circuits. Humans become more relaxed if you mirror them slightly.
Now we are in full breakdown mode. With Knebel charging forward to confront the umpire, and the damage done to ANGEL throughout the game they have almost certainly triggered his self-defense mode and Knebel is in real danger. A normal umpire would have immediately ejected pitcher and catcher for arguing balls and strikes, but in his crashing mode, he clearly intends to kill the closer.
Thankfully, ANGEL is programmed with facial recognition of all 30 managers and Craig Counsell’s fast work to get between Knebel and ANGEL likely saved Knebel’s life. ANGEL instantly recognizes the manager and his programming reverts to “Banal Manager Argument Prerogative One”. He tosses Counsell from the game and once again resets. Danger has passed and with just a few outs to go, he finishes the job. Thank goodness we did not see extras. It could have been a bloodbath.
Conclusion
Using the Brooks Baseball tool we can see just how tight zones ended up on the night. ANGEL’s strike zones were ridiculously small for lefties and righties alike, and there was a noted change in the lower right hand part of the zone (from ANGEL’s perspective) as the night wore on.
The Automaton simply couldn’t handle the complexity of a Major League Baseball game. I mean, I have no idea what was going on here:
And while it’s unfortunate, we are still years away from robot umps taking over. Next time someone mentions the possibility, just remember that we tested this very idea on May 22nd 2018, and it almost resulted in an on-field massacre the likes of which we’ve never seen.
For the time being, ANGEL will have to stay on the shelf. It remains Joe West’s World.
Well done.