Abstract: Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even be-tween experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation effort using the proposed scheme shows significant improvement in IAA from the conven-tional 60’s to 80’s (Cohen’s Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.
Authors: Qiang Ning, Hao Wu, Dan Roth (University of Illinois at Urbana-Champaign, University of Pennsylvania)