Monday, March 12, 2012

DRS: Accuracy, Judgment And The End Of The Umpire

Jacques Kallis had this to say about ball-tracking recently:
"How accurate it is, I don't know. Have decisions improved? I think they have but we have got to accept that there are probably one or two that, as cricketers, we will think 'I'm not so sure', but maybe that's an improvement on absolute shockers which is what you wanted to take out of the game. We are getting that right to a degree but I am not convinced how accurate it really is."
The great South African all rounder's comments illustrate a crucial conceptual fault line in the current implementation of the DRS. The DRS - the Decision Review System consists of specific approved technologies which are deployed in a specific communications protocol which specifies what kind of questions can be asked, who can ask them, how they are to be answered, when can they be asked etc.

The ICC's General Manager for Cricket, Kallis's former teammate and South African wicketkeeper Dave Richardson says that Kallis is wrong when he claims that 99% of cricketers will say that ball tracking is not accurate. Of the ball tracking system, Richardson says:
"The bottom line is that they are going to be more consistent and more accurate than the human eye, that is just natural,"
There are two sets of distinction that are important here. The first is the difference between a reliable system and an accurate system. The second, is the difference between an error and a mistake.

A reliable system is one which is consistently predictable. Given the same set of events on the field, its result will be the same. And accurate system is one which will judge a set of events accurately. A reliable system can be reliably inaccurate.

How is accuracy determined? How is it determined that on average, the ball-tracking system is more accurate than the human Umpire? By what measure is it more accurate? The conventional answer to this question is - that if a ball-tracking system predicts that a ball will miss the stumps by 2 millimetres, then a human umpire is less likely to agree with it, than in the case of a ball that will miss the stumps by 20 millimetres. In an interview conducted in July 2011, Paul Hawkins of Hawkeye Innovations told me that in some small internal tests conducted by his company, the ball tracking system was 10 times more accurate than the average human umpire.

The problem with this mode of analysis is that human umpires do not judge LBWs in the same way that ball-tracking systems do. They do not compute results in centimetres - they do not ask - would that have missed by 2 cm or 5 cm?. There is no reason to assume that human judgment proceeds along the same steps as the ball-tracking system. The ball-tracking system predicts the path of the ball by considering forces that act on the ball at each time interval. The Umpire on the other hand uses a heuristic where the operative question is - would it have hit the stumps or not? The Umpire's judgment is made in real time, and depends on the fact that the Umpire is (a) an experienced expert, and (b) is present in the best possible position to judge the event in real time.

The current practice could easily be reversed. Why can't we say, for example, in a case where the Umpire thought it was Out, and a ball-tracking system thought it would have missed the stumps by a whisker, that the ball tracking system is wrong and the Umpire is right? There is no scientific or logical reason not to do this. The LBW decision given by Umpire Ian Gould against Sachin Tendulkar in the 2011 World Cup Semi Final is the perfect illustration of this point. The idea that accuracy be measured in the terms used in the ball-tracking system is a purely normative one. It is not "natural" at all despite what David Richardson says in the above story. It is a choice, and a very deliberate one at that. I will explain in a moment why.

It is absurd to say that in the case of an LBW decision which most, if not all, experienced Umpire would have thought was close, and roughly half would have ruled Out, and the other half would have ruled Not Out (the very definition of a marginal decision), that the half that disagreed with the ball-tracking solution was wrong and the other half was right. The ball-tracking solution does not permit the Umpires to be in a position in which either decision, made in the live moment, would be reasonable.

Now, there are obviously instances where Umpires make mistakes in predicting the path of the ball. But this is precisely the point - they make mistakes. Not errors. A mistake is a decision which an Umpire would reverse if he had the chance to take a second look. An error, such as it is, is a decision where the Umpire is likely to stick with his original decision upon a second (or any number of) look even if the ball-track provides a contradictory result (Gould's LBW against Tendulkar would be classed as an "error").

I suspect that the ICC has chosen use the ball-track to verify the Umpire's judgment, as opposed to using an Umpire to verify the Umpire's judgment, because they want to take subjectivity out of the equation. This is a fool's errand. All they have managed to do is to shift the subjectivity from the Umpire to an elaborate rule-making apparatus consisting of the ball-track and margins of error introduced by the ICC - margins which have nothing to do with the limits of the ball-track's capabilities, except in the case of the 40 cm rule.

In doing so, the ICC have effectively ended the Umpire's role a judge of the action. The ironically named "Umpire's Call" is, as you all know by now, the residue of a verification of an Umpire's actual call by this apparatus. The "Umpire's Call" stands when the technology cannot conclusively verify the decision.

What we are seeing right now is an unsatisfactory marriage between two distinct modes of predicting where the ball would have gone had it not hit the pad. If the ICC wants to use the ball-track, then thats fine. Umpires aren't necessary at all. But it is not fair to the Umpire to have their decisions verified by the ball-tracking system.

The current system as far as the predicted path is concerned (other points - where the ball pitched, where it hit the pad etc. are events that actually happen and can be verified by technology) is as follows:

1. Umpire makes a decision in real time.
2. Players request a review.
3. The decision is reviewed using the ball-track.

There is no distinction in the system between an error and a mistake. The "Umpire's Call", as I have shown is  too narrow and is not in fact an Umpire's call at all.

A better system would be:

1. Umpire makes a decision in real time.
2. Players request a review.
3. Third Umpires reviews the decision using two perfectly placed cameras - square of the stumps and and in the line of middle stump, and judges whether the Umpire's decision is an error or a mistake.
4. If it is not a mistake, then the decision stands.
5. If it is a mistake, then the decision is reversed after the mistake is verified by the ball-track.

Adding that extra step will change the nature of the LBW judgments. If players know that there is that extra step, they will also hesitate to take the 50-50 chance that Jacques Kallis says they take these days. Teams will hesitate to be strategically aggressive with their first available review.

Will it happen? I don't know. Currently, the ICC seems to believe that the ball tracking system is naturally more consistent and accurate than the Umpire. If this is the case, then why don't they use it instead of the Umpires? The answer to that question is complicated, and is unlikely to satisfy either the Umpires, the players or the ICC itself.

The ICC must decide - does it want to continue to use Umpires, or does it want to use ball-tracking? Right now, it has come to a weak compromise which puts Umpires in an impossible situation. It also makes players, who are used to the way Umpires make judgments, distrustful of the technology. The reasonable responsible to Jacques Kallis's question about whether or not the system is accurate would be a prescriptive Yes, not the descriptive one that Richardson has presently offered. We have reached the de facto end of the Umpire as a judge in cricket. The ICC must either restore the Umpire or retire him.

No comments:

Post a Comment