Discussion

WordSieve extracts context information from documents in the form of keywords. This is standard for many intelligent information agents, and for the purposes of document retrieval, seems to be a natural approach. However, the results of these experiments suggest the benefit of taking into account extra contextual information available in user document access patterns, and the effectiveness of the WordSieve algorithm for this task. The experiments suggest that keyword occurrence patterns exist in user document accesses over time, and that those patterns can help characterize the context within which the document was accessed, compared to a method such as TFIDF that is unable to capitalize on this information. The results also suggest the viability of using a small, short term memory (the MostFrequentWords level) and probabilistic networks in learning about the user's context, though more experiments and long-term studies are needed.

This research also raises the issue of what constitutes the context of a given instance of a web search. For this experiment, we defined the context as the relevant terms from the topic for which the users were asked to search. However, a number of comments we received from users after they had performed the experiment question those assumptions. The users all seemed to have more trouble with the butterfly query than with the genetic algorithm query. Two users commented that they had found much information about a particular kind of butterfly, but not about Asian butterflies in general. This suggests that the user's perception of the context was informed both by the explicit task given the user and the kind of information the user actually found, and suggests that perhaps the context should be defined not only as what the users were asked to find, but what they were asked to find plus what they actually did find. That type of context description could be generated by asking the users to write five keywords that they thought characterized the web pages they actually found, but that were not in the original question, and using those to augment the task description. However, we expect that both TFIDF and WordSieve would benefit from these expanded queries, so that the comparative performance patterns for the two algorithms would still hold.

We believe we can improve this algorithm's performance. We have not yet sufficiently explored the values assigned to the free parameters in the system to see how we can increase the accuracy of this model and decrease the variance in its performance. Also, another level could be added which would become sensitized to only non-discriminators. These words could then be banned from the upper two levels, or perhaps from the lower levels altogether, assuring that space in the profile is reserved for useful words only.

Obviously, there are many forms of contextual information that WordSieve does not yet take into account. For example, WordSieve does not take into account the location of a term on a page, nor is it able to treat a document differently if a user keeps returning to it. For example, consider a situation where a user performs a web search, then clicks on many links from that page, returning frequently to make another choice. That page is probably useful in determining the context, but WordSieve would not treat it in any special way (other than re-reading it many times). Thus we see WordSieve as a useful tool to reflect one aspect of context, to be augmented with others for richer descriptions.

Another interesting question concerns how to account for the quality of a document in the subject domain being searched. WordSieve's analysis does not address this. In Calvin, users can set up customized filters to stop certain documents from being indexed (e.g., to filter out advertisements and error pages), but this increases the need for user configuration.