On The Mysterious Impact Index

The Impact Index is one or many or all (since this is a post about a system of cricket stats) of the following:

1. It is an "organic and holistic analytics system in cricket."
2. It is an "analytics system used in the game of cricket to determine the accurate worth of a player's performance, in a match, team and career context."
3. It measures "the impact that a player has on a particular match. It does not measure potential or talent or flair. . . just the actual deeds of the player in that match relative to the deeds of the other players. From the information only contained on the score sheet."

Recently, the authors published a ranking method. Their claims about this were similarly modest.

"Wisden Impact ranking system is the most scientific and accurate cricket ranking systems in the world today. Besides accounting for the knowledge of its experienced panel, the rankings are exclusively powered by Impact Index – the most holistic analytics system in the game today."

"Impact Index is the most innovative analytics system in the game today, which is why it was written about in Wisden Almanack (UK) in 2012 itself. Its ranking function is one of its most valuable applications. It is a huge improvement from the panel-led award selections that others indulge in."

Its quite a system. Organic, holistic, scientific, objective (at least more than anything else), measures impact not potential or talent or flair, relies only on the scoresheet but creates a context for each performance, transparent, calculations can be done on the back of the envelope. It can do the at least the following

"For batsmen, the system can evaluate how much pressure (of falling wickets) he absorbed successfully, the impact of the partnerships he built and of course, the impact of his strike rate (which is different from the usual “strike rate” numbers quoted for players). For bowlers, it can spot partnership-breaking impact, pressure-building impact (when a bowler takes quick wickets) and of course, economy impact (again, different from the usual “economy” numbers used). It also separates the lower-order wickets from the others. It also measures captaincy, fielding and wicket-keeping impact too. For all cricketers, it can also measure their failure rate.

Everything is measured from the scorecard. There are no external inputs taken from anywhere."

They tell you how the calculations are carried out. Well, sort of anyways. This system is described in a step-by-step methodology for ODI games. The method for Test matches is, we are informed, in the spirit of transparency, a "little more complex". Remember that unlike limited overs games, in which three results are possible (other than "no result"), in Tests, 4 results are possible. Before I delve into the methodology (to the extent that this is possible), here is a revealing paragraph about the system from their FAQ.

It is clearly better than batting and bowling averages as the skews that result inherently from those methods are intrinsically avoided. This system enables a more holistic measurement by taking into account the context of the performance (intangibles like state of the pitch, state of the match in which the performance happened, strength of opposition, even the state of series/tournament within which the match is occurring), which has always been considered beyond the realm of cricket statistics.

It is better than other rating systems because more than being a statistical system, Impact Index is a different lens with which to evaluate the game. It does not number crunch with external factors but organically from what exactly transpired in the specific match (which also results in far less unnecessary crunching). This leads to far fewer assumptions and a far, far more objective system, which also makes it much more accurate in its essence.

It is a very versatile system that can be extended to cover almost any aspect of cricket more accurately than before. Most importantly, it is a transparent, honest system that cannot be manipulated. Conclusions cannot be just arrived at and hidden behind the opacity that all other systems seem to eventually display.

So we are dealing with a system which makes far fewer assumptions than other stats systems. It accounts for the state of the pitch from the scorecard (this is not as controversial as it sounds, it is a good idea in principle), and it claims to be 'far more objective' (its not clear what this means). One of the 'stats systems' mentioned by the Index is the batting average. In the FAQ, we are told that the batting average is skewed because one success can submerge many failures ("the skews of conventional stats systems (like averages) where one big performance can cover for repeated failures thereafter"). In the methodology, we are told that "Impact Index, unlike every other cricket statistical system in the world, is transparent in its workings." Now, the following four things are true:

1. I can tell you exactly how the batting average is calculated (most of you already know this and don't need me to explain it) and we can therefore try to improve it

2. The Impact Index authors themselves are able to critique the batting average
3. The Impact Index authors suggest that every other cricket statistical system (of which they consider batting average to be one) in the world is not transparent. (yet, see 2)

4. I can't tell you how the Impact Index for a match is calculated based on their 'step by step' methodology

Either the Impact Index people don't understand what 'transparency' means, or they simply don't care. This "most scientific system" is an impenetrable black box. I asked them to publish a sample calculation for a single Test match or a single ODI match. They refused to publish such a thing. I wrote back to confirm that I had understood their position, which was that their methodology was proprietary and not for public consumption, correctly. They wrote back saying that this was the case. Further, they said:

"We have given as much as we can in the methodology without giving the specific numbers."

"We think the methodology is good enough to understand how the system works. Beyond that, we cannot go into the specifics."

Whatever else it may be, transparent it is not. And I challenge anybody to take any scorecard and actually calculate the Impact Index based on the instructions given here. If you can do it, I will withdraw this review and publish your calculation instead.

Its impossible to evaluate the merits of the methodology because of two basic problems beyond the fact that they won't publish it.

First, they keep alluding to other 'stats systems'. The only ones which they do refer to specifically (like batting average or strike rate) are entirely transparent (in that, you know exactly how the calculation works and what is being measured). They don't refer to any other actual 'stats system'. There is not a single reference to one on their entire website. The description of the batting average as a 'stat system' is at best only trivially accurate.

Second, they do not provide a reference to a single peer review. Even if they didn't want to make their methodology public, they could describe it to other professional statisticians who understand cricket well and get it reviewed. While I disagree with the idea that methodologies should be private, many others legitimately think that it is Ok to keep these things private. Even so, peer review should be possible.

It is not a surprise that their claims for their product are all over the place. So are some of their basic facts. For example, point 6 in their step by step methodology is as follows:

"For a bowler, Impact points are garnered for a) Wickets (the top 8 wickets and the last 3 have different values, in most cases, as per the circumstances of the match)"

There are two possibilities here. Either they think that 11 wickets are available per innings, or, they count the 8th wicket in the first group or the last, depending on the circumstances of the match. This second possibility is interesting and would require explanation. I suspect that the flunky who was told to gin up this "methodology" simply didn't care enough to ensure that it was accurate or even plausible.

I looked at the results they have published. These leave me with serious doubts not only about their understanding of what they are measuring, but about the extent to which they actually care about being accurate and reasonable. Take this 'oddity' for example. It is currently featured on their 'Impact Oddities' page.

In the famous Adelaide Test in 2003 against Australia that India won, Dravid’s second innings unbeaten 72 had a higher impact than his first innings 233.

This is why.

The first innings totals for both sides crossed 500 – Australia 556 (Ponting 242) and India 523 (Dravid 233, Laxman 148). Even though Dravid (and Laxman) stabilised the innings from 82-4, Dravid shared his impact with these two and all the others who contributed to the first innings tally of 1079 (at an average of 54 per wicket) for both sides.

But in the second innings, the character of the Test changed entirely. Australia made just 196, and India were set 233 to win in the fourth innings, which they did after losing 6 wickets (second innings average runs per wicket: 27) – with Dravid staying not out to see his team home in a tense fourth innings chase. Even without his massive 233 in the first innings, this unbeaten 72 ensured Dravid had a big impact in the game.

Now, without his first innings 233, the 2nd innings 72 would not have existed. Leave aside the fact that India's 4th wicket in the 1st innings of that game fell at 85 and not 82. Apart from that 303 run 5th wicket stand, the other 9 partnerships in the Indian 1st innings produced a grand total of 220. If we leave out the highest stand of the Australian first innings (138 for the 5th wicket between Simon Katich and Ricky Ponting), the other 9 partnerships in the Australian innings produced 417. Take away the next highest stand and the bottom 8 stands for India produced 154, for Australia 304. You begin to see very quickly that while both sides made 500+, they way they got there was very different. Any reasonably sophisticated statistical system should be able to account for this. But even without taking the specifics of the Test seriously, the last sentence in this 'oddity', is, well, odd. What does it say about the "Impact Index" that it seriously considers the 72 to be independent of the first half of the match?

Whether Tests should be divided into the 1st match innings and the 2nd match innings for the purposes of analysis, is itself an interesting question. What's more, it is a choice and constitutes an assumption in any model of analysis. Statistics are constructed. They are not objective because they come about due to well specified, replicable methods which include explicitly stated assumptions. The best things that can said about statistics are that (a) What they measure and how they measure can be described exactly, and (b) Given such a precise description, anybody else who performs the same mathematical operations on the same data, will produce the same results. This is where statistics get their basic power.

If the method of calculation is not available, statistics are meaningless. As my friend who has a Ph.D. in mathematics from Berkeley likes to say, equations in which one is not told what each of the symbols stands for, are basically just drawings.

Other essays, such as the one about Ian Bishop are interesting. Bishop was a fantastic bowler who did equally well home and away. He played only 6 Tests in Asia but did pretty well in those too. Even though it is a good story it contains the one germ which seems to animate the entire existence of the Impact Index - grievance. The Index positions itself as a device for providing alternative insights into cricketing contests. Finding forgotten or overlooked heroes. Bishop is one such for them. They write -

"Although Ambrose was given the Man of the Series award for having a better bowling average, strike rate and wickets/innings bowled ratio, Ian Bishop had the maximum impact in the series as he generated his best where it mattered the most."

This is a stretch. Ambrose played three Tests in that series, Bishop played 4. They took nearly the same number of wickets (20 for Ambrose, 21 for Bishop who bowled 30 overs more). After 3 Tests, West Indies were down 1-0 in the series. Ambrose took 16 wickets in 89 overs in the 4th and 5th Tests and West Indies won both. Bishop did brilliantly too, taking 13 wickets in 86 overs. If Bishop's was a "series defining" performance because it came in the final Test with series tied 1-1, what was Ambrose's which came with West Indies 1-0 down with 2 to play? What does it say about the merits of the "Impact Index" that it measures Bishop to be above Ambrose?

We'll never know. Unless the Impact Index people see good sense and make their calculations public. They are now affiliated with Wisden India. The Wisden Almanack (see this year's Wisden India Almanack. It is beautifully produced) made its name by providing verifiable facts, painstakingly collected and presented, year after year. It did not present opaque numbers which came out of a black box. It did not simply say "X is the best player in the world this year. Just take our word for it." That it has now adopted the 'Impact Index' without demanding clarity is a blow to its credibility. That it hosts the error riddled, gratuitously adverbed, ultimately empty explanation of the method and many findings whose provenance is a mystery is even worse.

Statistics are interesting because they (like history) offer a way to think about complex things. The Impact Index cannot be considered a statistical system until we know exactly how it works and how it considers Test matches and limited overs matches (two different games as far as calculating performance is concerned). Statistical analysis in cricket has virtue because it can be a way to lose oneself deep in the intricate arrangements of the cricketing contest, but also because learning how to use statistics to think is essential for understanding measurements in far more important areas of life like politics and economics, assessing risks and opportunities. In this sense, the Impact Index is failure on multiple levels. By refusing to explain their methodology clearly, they produce nonsense. But more importantly, they also convey a misleading sense of what statistics are. This is perhaps their biggest failing.

On Wisden India's website, they tout this system as being better than any other system in existence (of these, they can cite exactly zero except for basic things like batting averages). It is, they say "not like those random novelty academic one-off statistical exercises that the media finds fashionable to quote from, off-and-on, which vanish without a trace thereafter". At least those academic statistical exercises are actual statistical exercises which others can study.

I'd say to the people at Impact Index the same thing we tell undergraduates. Show. Don't tell.