John Charles Bradbury, an Economics Professor at Kennesaw State University, did a fascinating study a few years back on whether changing managers really has an effect on turning a team around. What he found was that ultimately managers had little to no effect on performance output given the same personnel. In a nutshell, good players make good managers. The only real difference was one of perception, from about 2000 on, there was a slight increase in attendance (on average about 1000 fans per game) following a managerial change, likely the result of some optimism stemming from the front office doing something to change a losing trend.
The study took a considerable sampling of data and integrated a comprehensive review of previous research. It focussed on whether or not replacement managers were able to generate increased output from individual players.
I’d read some reviews of the study prior to actually reading the study itself (waiting for adobe to update), and most of the secondary commentary seemed to take the study’s abstract conclusions and run with them without bothering to look into the text, which appeared, (at least to this reader) to be speckled with disclaimers and reservations. To be fair, numerous readers took issue with the fact that by looking strictly at individual performance the study neglected the ultimate benchmark — winning — which is potentially problematic because winning and individual performance don’t always correlate. By looking strictly at performance, however, you eliminate strength of schedule as a factor, not to mention the impact of injuries, trades, changes in batting order and so on.
The author estimated the impact of managers on player performance using a sample of major-league baseball players from 1980 to 2009, available from Baseball-Databank.org. He estimated Equation 1 using the Baltagi and Wu (1999) random-effects method, which corrects for detected first-order serial correlation.
(1) Performanceiy = γ Manageriy+ β1 League Performancey + β2 Career Performancei + β3 Ageiy + β4 Ageiy2 + θ Parkiy + νi+ εiy
Performance is the individual performance of player i in year y. Manager is a vector of individual manager dummy variables. For hitters, performance is measured using on-base-plus-slugging (OPS), which is a simple metric for measuring how effective a hitter is at producing runs. For pitchers, performance is measured using earned run average (ERA). The coefficients for the dummy variables in vector γ should reflect the impact that individual managers have on player output. League Performance is the league average OPS for hitters and league average ERA for pitchers. The league average controls for fluctuations in run scoring in the leagues may cause deviations in performance across leagues and over time. Career Performance measures the quality of the observed players by averaging the performance of each player over his entire career, which normally spans several managers.
Of the 134 managers in the sample, the estimates for 25 managers are statistically significant at the ten-percent level for hitters. 21 managers are associated with improvement and four managers are associated with a decline. For pitchers, the estimates for 24 managers are statistically significant at the ten-percent level. 15 managers are associated with player improvement and nine managers are associated with a performance decline. Five managers are associated with improvement and decline for both groups; however, in all cases, the managers are associated with the opposite effect for the two groups of players. Thus, no manager is associated with improving performance for both offense and defense.
The results indicate that if managers have some influence on player performance, the impact is small and difficult to identify.
Baseball Between the Numbers, analyst James Click also tried to tease some signs of managerial impact out of statistical record but came up empty. After examining the measurable effect of in-game strategies (bunting, stolen bases, intentional walks), wins and losses relative to run differential, playing time distribution, in-game substitutions (pinch-hitters, relief pitchers, and defensive replacements), and direct impact on player performance (coaching), Click was unable to find evidence of a repeatable skill in any one of those five areas for any of the 456 managers he studied. That is to say that, much like clutch hitting, individual performances varied so much from season to season that the results appeared to be as much the result of chance as anything else.
There was also the 2006 study by James Cliff in which he stated:
“Only six times in thirty-three years has any manager used sacrifice attempts, stolen base attempts, and intentional walks to increase his team’s win expectation over an entire season. Even the best managers cost their team more than a game per season by employing these tactics. At worst they can cost a team three games per season.”
Finally, Chris Jaffe wrote a definitive and comprehensive analysis of Manager competency and effect in his book, Evaluating Baseball’s Managers. A History and Analysis of Performance in the Major Leagues, 1876-2008. (highly recommended). In it he more or less shows that good managers don’t have much of an effect, and even bad managers don’t do as much harm as you might think.
Independent of whatever considerable support the argument that managers don’t matter may have, I still have my reservations. In the first study above, OPS doesn’t correlate with winning as much as several other statistics, namely runs scored, but even more troublesome is one very problematic variable – the unearned run.
Unearned runs are not like other runs, they are the neglected stepchildren of baseball, they are the runt of the litter that nobody wants, they are the ugly babies that the Spartans would throw off a cliff. The problem with unearned runs is that outside of an error here or there, no one is truly accountable for them. Defensive metrics being what they are, a study that looks at changes in offensive output and ERA doesn’t control for runs that cross the plate that are unearned. The unearned run can be a death knell in the late innings. It can also be an indication that defensive alignment isn’t what it should be, fundamental defensive practices may not be in place, and players may simply not be well coached.
Defense appeared to be lumped together as one of many variables affecting ERA. Compounding this problem is the fact that the unearned run may be a direct result of poor defense. This omission renders an entire competitive dimension (team defense) inconsequential, when any fan can tell you it is not.
You can make a strong argument that while a player can rise to the major leagues on a given set of abilities such as hitting and throwing, defense, perhaps more so than any other aspect of the game, may be a reflection of focus, preparation, and most importantly (for purposes of this argument), effort … These traits are inexorably linked to good coaching. Bear in mind it was defense (among other things) that let the Mets down in the late innings of the series vs. the Marlins. Defense may in fact be the greatest litmus for a manager’s overall effectiveness, but in the end how do you measure a leader’s ability to get his players to “run through walls” for him?
There are countless anecdotal narratives that run contrary to the claim that managers don’t matter — there have been numerous cases where a change has resulted in a dramatic turn-around. Buck Schowalter of the Orioles as recently as 2011 changed his team’s fortunes almost upon his arrival. Could it be that given a manager’s already marginal impact on the field of play (as shown in multiple studies) it takes a really tremendous manager to actually effect a turn-around? Well, if this is true, and it may be, it would undermine the broader argument that manager’s don’t make much of a difference no matter how good (or bad) they may be because it would imply that while most managers are mediocre (which is why broad based studies show a negligible effect) the exceptional ones do make a difference.
In the end we are left with the possibility that a manager’s effect may not be measurable by any conventional analysis. How do you measure inspiration? How do you quantify cohesion? How do you control for that little bit of effort that could be the difference between a run saved and another loss?
As much as Bill James and Nate Silver and others have tried to quantify managerial performance, it is an elusive and ethereal component of the game that is far too complex to pin down with metrics.
I’ve always gotten the sense that Sandy Alderson ascribes to the premise that managers don’t really make much of a difference. Even great managers after all make about as much as a middle reliever. In a baseball landscape where value in wins has become a catchphrase, the numbers people will tell you that the difference between a truly extraordinary manager like a Herzog or a Cox or a Hodges, and an Art Howe is maybe 3 to 5 wins over the course of a season, if that …
My contention nevertheless resides primarily in all those intangible inspirational and interpersonal aspects of the game that numbers can’t speak to, warm fuzzies such as determination, camaraderie, and above all, hope. There is nothing like giving a band of dejected players immersed in patterns of learned helplessness the gift of believing in themselves and in each other. How do you measure that? And yes, that sounds like something only a select few exceptional leaders would be able to pull off.
It’s always been a glaring paradox to me how an organization that in practice may marginalize the role of the manager, will, with the same breath trumpet the importunate significance of a good development program. Your manager, and your development program, are in practice part of the same operational system. In the end, having seen exceptional coaches take rag tag assemblages of high school kids and transform them into champions, I can’t in good conscience ascribe to the notion that coaches, managers, and leaders, don’t play a role in that effort. They matter, but they matter in ways that relate more to the spirit than they do to numbers on the field.
As Vince Lombardi once said:
“The spirit, the will to win and the will to excel – these are the things that endure and these are the qualities that are so much more important than any of the events that occasion them.”