21. Such disagreement indicates poor inter-rater reliability. 22. APA was most concerned to demonstrate inter-rater reliability rather than necessarily validity. 23. The inter-rater agreement between evaluators was calculated using the Fleiss ? scale. 24. Another piece to keep in mind is the effects of rater motivation on judgmental evaluations. 25. Still, these organizations continue to use multi-rater feedback in their development processes. 26. Data obtained from ratings rely totally on the rater 's familiarity with the child. 27. Many times rater training is used in subjective trials, such as in CNS studies. 28. Inter-rater agreement can now be applied to measuring the computer's performance. 29. It was written by Tony Phelan and Joan Rater , and directed by Donna Deitch. 30. The bond rater also said it is concerned about possible regulatory and legal actions against ValuJet.