In the most recent statistics reported by the ICC, "[i]n Test matches, the increase in correct decisions was 4.27 per cent and in ODIs was 5.01 per cent producing an overall improvement of 4.49 per cent to 98.26 per cent."
Since DRS was first tested in a Test series (featured Sri Lanka and India in 2008), Umpiring Decision statistics have been reported multiple times. After that Sri Lankan series, in a report from September 19, 2008:
The headline result of the trial was incredibly positive with a correct decision percentage of 98.04% against the usual Test match average of around 94%.
Of the total 204 appeals during the series, 48 were referred by either the bastmen or the fielding captain and 25% of these referrals were successful. In simple terms, the review system ensured that in this particular series the number of correct decisions was 6% higher than it would have been without the referral system."Since then, in various reports on November 27, 2009, February 11, 2010, September 8, 2010, September 4, 2011, ICC officials like Mr. Richardson and Mr. Vincent van der Bijl have consistently reported a 4-6% improvement in decision making due to the DRS - a move from about 91-93% to 95-98%. In one of the reports, Mr. Richardson said that "The ultimate aim is to make sure we avoid clear mistakes, mistakes which you get to know sitting in your sitting room. If we can get the percentage of correct decision-making from 92-93 per cent to 97-98 per cent, it means we have achieved our objective."
Based on some initial email communications, I learnt the following:
1. The purpose of DRS is to eliminate "clear mistakes" or "obvious mistakes".
2. The use of the DRS results in an improvement of about 4-5% in Umpiring decisions. (it improves from the low 90s to the high 90s)
3. In a 12 month period from June 11 2011 - June 11, 2012, DRS was used in 30 Tests and 38 ODIs. 340 player reviews were requested in these 68 games. Of these 80.58% were for LBWs. 57 out of the 340 were determined to be "Umpires Call" situations (1 out of 6 Player Reviews. However, the ICC does not keep a record of the number of LBW decisions which resulted in Umpires Calls.)
The statistics measure appeals. As Law 27 specifies, an Umpire is cannot declare a batsman out, even if the batsman is Out as per the Laws unless a fielder makes an appeal. As the ICC's Colin Gibson conveyed in his response to my email questions, in a nutshell, "an appeal is a request for the umpire to give a ruling for a case [in which] otherwise he would not get involved even if the batsman is out under one of the laws and no appeal made."
So if the ICC says that 98% of the decisions in a Test in which DRS is used are correct, while only 93% of decision in a Test in which DRS is not used are correct, it means the following:
A. If 100 appeals were made in a Test, the Umpires would rule correctly 93 times.
B. If 100 appeals were made in a Test, and Player Reviews were available, the Umpires end up ruling correctly 98 times.
Here the issue is complicated. How is it determined that a correct decision was made in each case? Is a "correct" decision available for comparison, independent of a method (either DRS supported or otherwise) of decision making?
Basically, the ICC is comparing decisions reached through two different methods of judgment - both on the field, and later, for judging the decision itself. So comparing the two is not like comparing 2 students who took the same test at the same time in the same examination hall, it is like comparing two students at two different levels of advancement (say a 9th grade Hindi Exam in a Hindi medium school vs as 11th grade Hindi Exam in an English medium school). It is, by any reasonable standard meaningless to say that DRS is better based on these figures. This is not of course, to deny that DRS could be better. It is only to say that the statistics do not offer any evidence that might lead to such a conclusion.
Here's why.
I'll consider measurements in the case of DRS first:
In the DRS case, the measurements are relatively straightforward. Within DRS, there are mechanisms for concluding whether the on field decision was (a) correct (b) wrong (c) marginal - or "Umpire's Call". There are strict and unambiguous standards for a decision to be "Umpire's Call".
When the performance of the Umpires+DRS is measured, "Umpire's Call" decisions are (correctly in my view) counted in the correct decisions column. The 2-3% or so decisions that do go wrong are because of human error, or, because the team in question ran out of reviews, and hence the decision could not be reviewed.
In the case of the non-DRS tests:
Here, the ICC finds that about 93 out of 100 decisions are correct. It follows that of these 93 decisions, some must be marginal, while many must be obviously correct. Similarly, it follows that of the 7 decisions that are judged to be wrongly made by Umpires, some must be marginal, while others must be more obviously wrong.
The ICC told me that judgment calls are made about how to score each decision -"the referees confirmed by the ICC umpire and referee manager classifies each decision – easy, hard or very difficult. we would regard an obvious mistake as Hard. this naturally is a matter of judgment by ref and manager. Manager helps keep a global benchmark standard" Further, they said that "these incorrect decisions are often not what we would call obvious – the majority are marginal and very difficult like feather edges."
The 93 out of 100 decisions are the definitely correct ones. The 7 out of 100 wrong decisions include some very difficult decisions which the Umpire got wrong, as well easy decisions which the Umpire got obviously wrong.
The 97 or 98 out of 100 decisions which are correct under DRS, include the ones that Umpire+DRS managed to eventually get right, as well as the decisions that technology could not decisively determine, and had to be left as "Umpire's Call".
I also asked the ICC whether they keep a record as to why a particular decision was judged that way it was by the Third Umpire in DRS. My question and the ICC's answer are as follows:
Q: (In light of the LBW decision against Chris Gayle) When on-field decisions are upheld after a player review, is a record kept as why the decision was upheld? If so, what percentage of decision are upheld due to (1) and what percentage of decisions are upheld due to (2) below
The ICC did report a statistic about Umpire's Call, as stated earlier in this post - 57 out of 340 Player Reviews between June 11, 2011 and June 11, 2012 were Umpire's Call situations. However, my best understanding of the ICC's response to my question above is that they do not record which "Umpire's Call" decisions were ball-tracking related, and which were because video replays were inconclusive (as the ICC does not keep a record of which "Umpire's Call" decisions were in LBW appeals).
While the ICC's individual claims - such as that 93 out of 100 decisions were correct in non-DRS situations and 97 out of 100 decisions were correct in DRS situations (for example, the figures vary slightly depending on the time period under consideration), are probably reasonable, the conclusion that DRS therefore produces a 4% improvement is not reasonable. This is because the correctness of decisions in non-DRS and DRS situations is measured differently.
More importantly, the ICC's method of measuring non-DRS decisions uses a version of Review. There are other experts who review the decisions on the on-field expert. They make judgments as to whether the on-field decision was Easy, Hard or Very Difficult, and then reach an expert judgment as to whether or not the Umpire on the field made a wrong decision (i.e., a decision which they could determine was wrong beyond reasonable doubt, without the benefit of DRS level technology). Why such a review is not used in actual matches, with the Third Umpire reviewing the On-field Umpire's decisions without the determinative constraints of the ball-tracking system's margin of error, is not easy for me to understand.
By presenting its statistics in the way that it does (97% in DRS is a 4% improvement over 93% non-DRS) the ICC is claiming that it is possible to say whether or not an Umpire's decision was right or wrong without the benefit of DRS level technology, just as well as DRS is able to determine whether an Umpire's decision was right or wrong. If this is so, then the obvious question is - why is DRS necessary? The ICC's Umpiring Decision Statistics undermine the ICC's view that DRS is superior to multiple expert human eyes looking at an appeal.
The ICC cannot have it both ways. if their statistics are logically consistent, then DRS is unnecessary. If DRS is necessary, then their statistics are not logically consistent.
So if the ICC says that 98% of the decisions in a Test in which DRS is used are correct, while only 93% of decision in a Test in which DRS is not used are correct, it means the following:
A. If 100 appeals were made in a Test, the Umpires would rule correctly 93 times.
B. If 100 appeals were made in a Test, and Player Reviews were available, the Umpires end up ruling correctly 98 times.
Here the issue is complicated. How is it determined that a correct decision was made in each case? Is a "correct" decision available for comparison, independent of a method (either DRS supported or otherwise) of decision making?
Basically, the ICC is comparing decisions reached through two different methods of judgment - both on the field, and later, for judging the decision itself. So comparing the two is not like comparing 2 students who took the same test at the same time in the same examination hall, it is like comparing two students at two different levels of advancement (say a 9th grade Hindi Exam in a Hindi medium school vs as 11th grade Hindi Exam in an English medium school). It is, by any reasonable standard meaningless to say that DRS is better based on these figures. This is not of course, to deny that DRS could be better. It is only to say that the statistics do not offer any evidence that might lead to such a conclusion.
Here's why.
I'll consider measurements in the case of DRS first:
In the DRS case, the measurements are relatively straightforward. Within DRS, there are mechanisms for concluding whether the on field decision was (a) correct (b) wrong (c) marginal - or "Umpire's Call". There are strict and unambiguous standards for a decision to be "Umpire's Call".
When the performance of the Umpires+DRS is measured, "Umpire's Call" decisions are (correctly in my view) counted in the correct decisions column. The 2-3% or so decisions that do go wrong are because of human error, or, because the team in question ran out of reviews, and hence the decision could not be reviewed.
In the case of the non-DRS tests:
Here, the ICC finds that about 93 out of 100 decisions are correct. It follows that of these 93 decisions, some must be marginal, while many must be obviously correct. Similarly, it follows that of the 7 decisions that are judged to be wrongly made by Umpires, some must be marginal, while others must be more obviously wrong.
The ICC told me that judgment calls are made about how to score each decision -"the referees confirmed by the ICC umpire and referee manager classifies each decision – easy, hard or very difficult. we would regard an obvious mistake as Hard. this naturally is a matter of judgment by ref and manager. Manager helps keep a global benchmark standard" Further, they said that "these incorrect decisions are often not what we would call obvious – the majority are marginal and very difficult like feather edges."
The 93 out of 100 decisions are the definitely correct ones. The 7 out of 100 wrong decisions include some very difficult decisions which the Umpire got wrong, as well easy decisions which the Umpire got obviously wrong.
The 97 or 98 out of 100 decisions which are correct under DRS, include the ones that Umpire+DRS managed to eventually get right, as well as the decisions that technology could not decisively determine, and had to be left as "Umpire's Call".
I also asked the ICC whether they keep a record as to why a particular decision was judged that way it was by the Third Umpire in DRS. My question and the ICC's answer are as follows:
Q: (In light of the LBW decision against Chris Gayle) When on-field decisions are upheld after a player review, is a record kept as why the decision was upheld? If so, what percentage of decision are upheld due to (1) and what percentage of decisions are upheld due to (2) below
Reviews should report which of the following three things happened:
1. Umpire upheld decision on the field based on conclusive supporting evidence.
2. Umpire upheld decision on the field based on inconclusive contrary evidence.
3. Umpire recommended reversal of the decision on the field based on conclusive contrary evidence.
ICC: We do not keep these statistics. To reverse a decision there has to be a high degree of confidence that the decision is incorrect. If there is not a high degree of confidence then the on field decision stands. mainly clean catch and feathered caught behinds- there is a standard protocol to handle these.
1. Umpire upheld decision on the field based on conclusive supporting evidence.
2. Umpire upheld decision on the field based on inconclusive contrary evidence.
3. Umpire recommended reversal of the decision on the field based on conclusive contrary evidence.
ICC: We do not keep these statistics. To reverse a decision there has to be a high degree of confidence that the decision is incorrect. If there is not a high degree of confidence then the on field decision stands. mainly clean catch and feathered caught behinds- there is a standard protocol to handle these.
The ICC did report a statistic about Umpire's Call, as stated earlier in this post - 57 out of 340 Player Reviews between June 11, 2011 and June 11, 2012 were Umpire's Call situations. However, my best understanding of the ICC's response to my question above is that they do not record which "Umpire's Call" decisions were ball-tracking related, and which were because video replays were inconclusive (as the ICC does not keep a record of which "Umpire's Call" decisions were in LBW appeals).
While the ICC's individual claims - such as that 93 out of 100 decisions were correct in non-DRS situations and 97 out of 100 decisions were correct in DRS situations (for example, the figures vary slightly depending on the time period under consideration), are probably reasonable, the conclusion that DRS therefore produces a 4% improvement is not reasonable. This is because the correctness of decisions in non-DRS and DRS situations is measured differently.
More importantly, the ICC's method of measuring non-DRS decisions uses a version of Review. There are other experts who review the decisions on the on-field expert. They make judgments as to whether the on-field decision was Easy, Hard or Very Difficult, and then reach an expert judgment as to whether or not the Umpire on the field made a wrong decision (i.e., a decision which they could determine was wrong beyond reasonable doubt, without the benefit of DRS level technology). Why such a review is not used in actual matches, with the Third Umpire reviewing the On-field Umpire's decisions without the determinative constraints of the ball-tracking system's margin of error, is not easy for me to understand.
By presenting its statistics in the way that it does (97% in DRS is a 4% improvement over 93% non-DRS) the ICC is claiming that it is possible to say whether or not an Umpire's decision was right or wrong without the benefit of DRS level technology, just as well as DRS is able to determine whether an Umpire's decision was right or wrong. If this is so, then the obvious question is - why is DRS necessary? The ICC's Umpiring Decision Statistics undermine the ICC's view that DRS is superior to multiple expert human eyes looking at an appeal.
The ICC cannot have it both ways. if their statistics are logically consistent, then DRS is unnecessary. If DRS is necessary, then their statistics are not logically consistent.
No comments:
Post a Comment