Word-sense Disambiguation - Evaluation


Comparing and evaluating different WSD systems is extremely difficult, because of the different test sets, sense inventories, and knowledge resources adopted. Before the organization of specific evaluation campaigns most systems were assessed on in-house, often small-scale, data sets. In order to test one's algorithm, developers should spend their time to annotate all word occurrences. And comparing methods even on the same corpus is not eligible if there is different sense inventories.

In order to define common evaluation datasets and procedures, public evaluation campaigns have been organized. Senseval (now renamed SemEval) is an international word sense disambiguation competition, held every three years since 1998: Senseval-1 (1998), Senseval-2 (2001), Senseval-3 (2004), and its successor, SemEval (2007). The objective of the competition is to organize different lectures, preparing and hand-annotating corpus for testing systems, perform a comparative evaluation of WSD systems in several kinds of tasks, including all-words and lexical sample WSD for different languages, and, more recently, new tasks such as semantic role labeling, gloss WSD, lexical substitution, etc. The systems submitted for evaluation to these competitions usually integrate different techniques and often combine supervised and knowledge-based methods (especially for avoiding bad performance in lack of training examples).

In recent years 2007-2012, the WSD evaluation task choices had grown and the criterion for evaluating WSD has changed drastically depending on the variant of the WSD evaluation task. Below enumerates the variety of WSD tasks:

Read more about this topic:  Word-sense Disambiguation

Other articles related to "evaluation, evaluations":

Implementing NATOPS - Unit NATOPS Evaluation
... A unit NATOPS evaluation is conducted for every squadron/unit every 18 months by the appropriate NATOPS evaluator ... The unit NATOPS evaluation includes NATOPS evaluations for each crew position (ground evaluation and an evaluation flight) selected at random by the evaluator to ...
Hy's Law - Hy’s Law Cases Have The Following Three Components
... Food and Drug Administration, Center for Drug Evaluation and Research (CDER) Center for Biologics Evaluation and Research (CBER) in their final document of ...
Evaluation - Methods and Techniques
... Evaluation is methodologically diverse ... analysis Cost-benefit analysis Data mining Delphi Technique Design Focused Evaluation Discourse analysis Educational accreditation Electronic portfolio Environmental scanning Ethnography Experiment ...
Board Examination - Evaluation
... The answer sheets are sent back to the board of education overseeing the certifications ... The papers are evaluated based on examples of ideal answers ...

Famous quotes containing the word evaluation:

    Evaluation is creation: hear it, you creators! Evaluating is itself the most valuable treasure of all that we value. It is only through evaluation that value exists: and without evaluation the nut of existence would be hollow. Hear it, you creators!
    Friedrich Nietzsche (1844–1900)

    Good critical writing is measured by the perception and evaluation of the subject; bad critical writing by the necessity of maintaining the professional standing of the critic.
    Raymond Chandler (1888–1959)