Comparing Specification Marks Method with Maximum Entropy-based Method

The main goal of this section is to evaluate the Specification Marks Method montoyo2001senseval and the Maximum Entropy-based Method (in particular, MEfix System) on a common data set, to allow for direct comparisons. The individual evaluation of each method has been carried out on the noun set (17 nouns) of the Spanish lexical-sample task rigau2001senseval from SENSEVAL-2⁶. Table 12 shows precision, recall and coverage of both methods.

Table: Comparison of ME and SM for nouns in SENSEVAL-2 Spanish lexical sample

	Precision	Recall	Coverage
SM	0.464	0.464	0.941
ME	0.646	0.646	1

In order to study a possible cooperation between both methods, we count those cases that: the two methods return the correct sense for the same occurrence, at least one of the methods provides the correct sense and finally, none of both provides the correct sense. A summary of the obtained results is shown in the Table 13. These results clearly show that there is a large room for improvement when combining both system outcomes. In fact, they provide also a possible upper bound precision for this technology, which can be set to 0.798 (more than 15 percentual points higher than the current best system). Table 14 presents a complementary view: wins, ties and loses between ME and SM when each context is examined. Although ME performs better than SM, there are 122 cases (15 %) solved only by the SM method.

Table: Correct classifications of ME and SM for nouns in SENSEVAL-2 Spanish lexical sample

	Contexts	Percentage
Both OK	240	0.300
One OK	398	0.498
Zero OK	161	0.202

Table: Wins, ties and loses of ME and SM systems for nouns in SENSEVAL-2 Spanish lexical sample

	Wins	Ties	Loses
ME - SM	267	240	122

Footnotes

...Senseval-2⁶

For both Spanish and English SENSEVAL-2 corpora, when applying the Specification Marks method we used the whole example as the context window for the target noun