Interrater Reliability

Usage

agreement(
  data,
  vars,
  sft = FALSE,
  heatmap = TRUE,
  heatmapDetails = FALSE,
  wght = "unweighted",
  exct = FALSE,
  kripp = FALSE,
  krippMethod = "nominal",
  bootstrap = FALSE,
  icc = FALSE,
  iccType = "ICC2",
  pathologyContext = FALSE,
  diagnosisVar = NULL,
  confidenceLevel = 0.95,
  minAgreement = 0.6,
  showInterpretation = TRUE,
  outlierAnalysis = FALSE,
  pairwiseAnalysis = FALSE,
  categoryAnalysis = FALSE,
  diagnosticStyleAnalysis = FALSE,
  styleClusterMethod = "ward",
  styleDistanceMetric = "agreement",
  numberOfStyleGroups = 3,
  identifyDiscordantCases = FALSE,
  raterCharacteristics = FALSE,
  experienceVar = NULL,
  trainingVar = NULL,
  institutionVar = NULL,
  specialtyVar = NULL
)

Arguments

data

The data as a data frame. Each row represents a case/subject, and columns represent different raters/observers.

vars

Variables representing different raters/observers. Each variable should contain the ratings/diagnoses given by each observer for the same set of cases.

sft

Show frequency tables for each rater and cross-tabulation tables for pairwise comparisons.

heatmap

Show agreement heatmap visualization with color-coded agreement levels.

heatmapDetails

Show detailed heatmap with kappa values and confidence intervals for all rater pairs.

wght

Weighting scheme for kappa analysis. Use 'squared' or 'equal' only with ordinal variables. Weighted kappa accounts for the degree of disagreement.

exct

Use exact method for Fleiss' kappa calculation with 3 or more raters. More accurate but computationally intensive.

kripp

Calculate Krippendorff's alpha, a generalized measure of reliability for any number of observers and data types.

krippMethod

Measurement level for Krippendorff's alpha calculation. Choose based on your data type.

bootstrap

Calculate bootstrap confidence intervals for Krippendorff's alpha (1000 bootstrap samples).

icc

Calculate ICC for continuous or ordinal data. Appropriate for quantitative measurements.

iccType

Type of ICC to calculate. Choose based on your study design and measurement model.

pathologyContext

Enable pathology-specific analysis including diagnostic accuracy metrics and clinical interpretation.

diagnosisVar

Gold standard or consensus diagnosis for calculating diagnostic accuracy of individual raters.

confidenceLevel

Confidence level for confidence intervals (default 95\

minAgreementMinimum kappa value considered acceptable agreement (default 0.6).

showInterpretationDisplay interpretation guidelines for kappa values and ICC coefficients.

outlierAnalysisIdentify cases with consistently poor agreement across raters.

pairwiseAnalysisDetailed analysis of agreement between each pair of raters.

categoryAnalysisAnalysis of agreement for each diagnostic category separately.

diagnosticStyleAnalysisIdentify diagnostic "schools" or "styles" among pathologists using hierarchical clustering based on diagnostic patterns. This reveals whether pathologists cluster by experience, training, geographic region, or diagnostic philosophy.

styleClusterMethodHierarchical clustering method for identifying diagnostic styles. Ward's linkage was used in the original Usubutun et al. 2012 study.

styleDistanceMetricDistance metric for style clustering. Percentage agreement was used in original study.

numberOfStyleGroupsNumber of diagnostic style groups to identify. Original study found 3 distinct styles.

identifyDiscordantCasesIdentify specific cases that distinguish different diagnostic style groups.

raterCharacteristicsInclude analysis of rater characteristics (experience, training, institution) in relation to diagnostic styles.

experienceVarVariable indicating years of experience or experience level of each rater.

trainingVarVariable indicating training institution or background of each rater.

institutionVarVariable indicating current practice institution of each rater.

specialtyVarVariable indicating specialty (e.g., generalist vs specialist) of each rater.

A results object containing:

results$todo					a html
results$overviewTable					a table
results$kappaTable					a table
results$iccTable					a table
results$pairwiseTable					a table
results$categoryTable					a table
results$outlierTable					a table
results$diagnosticAccuracyTable					a table
results$krippTable					a table
results$interpretationTable					a table
results$heatmapPlot					an image
results$pairwisePlot					an image
results$categoryPlot					an image
results$confusionMatrixPlot					an image
results$diagnosticStyleTable					a table
results$styleSummaryTable					a table
results$discordantCasesTable					a table
results$diagnosticStyleDendrogram					an image
results$diagnosticStyleHeatmap					an image
results$frequencyTables					a html

Tables can be converted to data frames with asDF or as.data.frame. For example:results$overviewTable$asDFas.data.frame(results$overviewTable) Function for Interrater Reliability.