Performance vs. TFIDF

On the whole, when viewing a document in isolation, WordSieve is better able to match the document to its original context than TFIDF. Using a repeated-measures analysis of variance shows that these findings are statistically reliable, F(1, 382)=91.03, p<.05.

The overall comparison of WordSieve and TFIDF are shown below. The following table reports the average similarity of all relevant documents of all users to their original context as measured by TFIDF and WordSieve. WordSieve's mean performance at this task surpassed that of TFIDF by 54%.

As shown in figures 3 and 4, the overall patterns generally hold true when breaking down the comparisons by context and user. WordSieve outperformed TFIDF in all cases except for users 2 and 3 where it performed almost as well. Without a larger set of users, it is difficult to determine why it did not do as well in those cases. However, it may be significant that both had accessed relatively small number of relevant documents (only user 7 accessed fewer). Overall, these analyses show that the results are reproducible across diverse subsets of the data.

  

Figure 3: Comparisons by User

  

Figure 4: Comparisons by Query