The interpretation of a phrase usually depends on the previous discussion (i.e., context) of the conversation (Tepperman et al 2006). For example, the sentence “yeah, right” can have a different meaning (both sarcastically and otherwise) when following a sentence like “with java 8 we fix all problems” versus “breaking backward compatibility is risky”. For this reason, rating a comment without its context can be compared to eavesdropping on a group conversation and only catching the last phrase of the conversation. However, due to the technical and unstructured nature of software development artifacts, the impact of context might be different in technical prose than in normal language. Here, we want to analyze the impact of context on agreement between raters.