Subsequent extensions of the approach included versions that could deal with “under-credits” and ordinal scales. [7] These extensions converge with the intra-class correlation family (ICC), which allows us to estimate reliability for each level of measurement, from the notion (kappa) to the ordinal (or ICC) at the interval (ICC or ordinal kappa) and the ratio (ICC). There are also variations that may consider the agreement by the evaluators on a number of points (for example.B. two people agree on the rates of depression for all points of the same semi-structured interview for a case?) as well as cases of raters x (for example. B how do two or more evaluators agree on whether 30 cases have a diagnosis of depression, yes/no a nominal variable). Therefore, the common probability of an agreement will remain high, even in the absence of an “intrinsic” agreement between the councillors. A useful interrater reliability coefficient (a) is expected to be close to 0 if there is no “intrinsic” agreement and (b) increased if the “intrinsic” agreement rate improves. Most probability-adjusted match coefficients achieve the first objective. However, the second objective is not achieved by many well-known measures that correct the odds.
[4] The field in which you work determines the level of compliance allowed. If it is a sporting competition, you can accept a 60% agreement to nominate a winner. However, if you look at the data from oncologists who choose to take a treatment, you need a much higher agreement – more than 90%. In general, more than 75% are considered acceptable in most areas. Bland and Altman[15] expanded this idea by graphically showing the difference in each point, the average difference and the limits of vertical match with the average of the two horizontal ratings. The resulting Bland-Altman plot shows not only the general degree of compliance, but also whether the agreement is related to the underlying value of the article. For example, two advisors could closely match the estimate of the size of small objects, but could disagree on larger objects. Kappa is similar to a correlation coefficient, as it can`t exceed 1.0 or -1.0.
Because it is used as a measure of compliance, only positive values are expected in most situations; Negative values would indicate a systematic disagreement. Kappa can only reach very high values if the two matches are good and the target condition rate is close to 50% (because it incorporates the base rate in the calculation of joint probabilities). Several authorities have proposed “thumb rules” to interpret the degree of the agreement, many of which coincide at the center, although the words are not identical. [8] [9] [10] [11] There are several operational definitions of “inter-advisory reliability” that reflect different views on what a reliable agreement between advisors is. [1] There are three operational definitions of agreements: requirement, claim, requirement, specific means to request or request something when it is due or necessary.