The quality of a lexical resource such as FrameNet is crucial for its usefulness to both humans and computers. Data quality is evaluated in terms of its consistency and completeness. Consistency means that the body of data should obey any restrictions placed on it and that it should not be self-contradictory. Completeness means that the data set should exemplify frames completely so as to support machine learning and human comprehension.
An example of a consistency requirement is that for a pair of frames related by the Inheritance relation, each core FE of the parent frame is normally48 mapped to a FE of the child frame. An example of a completeness requirement is that each core FE needs to be exemplified in the frame description. As these examples illustrate, the quality requirements on the data flow from the principles of frame semantics and the desire to support certain kinds of applications. In particular, many requirements are motivated by the theoretical understanding and the envisioned practical applications of Frame-to-frame relations discussed in Chapter 6. Overall, more than 100 quality requirements have been defined. Achieving these requirements turns out to be quite a challenge because the FrameNet data and documentation are maintained continuously and simultaneously by the FrameNet team. In view of the large size of the FN database, strictly manual quality control measures are too costly. Therefore, semi-automatic and, to some degree, formal consistency management approaches are used which reduce the necessary effort considerably. FrameNet's quality management measures are more fully discussed in ([Scheffczyk and Ellsworth, 2006][Scheffczyk et al., 2006]).
All of these data are connected via the LU table, which associates lemmas with frames and is referred to by the annotation sets. There are many reasons to keep these databases distinct for our purposes:
Each of the three databases consists of several tables that are connected to each other. For example, there are separate tables for frames and FEs, where the FE table is linked to the frame table.
Quality management is is carried out using several different techniques:
The third measure is particularly important because, as experience has shown, violations to quality requirements are inherent to the linguistic enterprise.
Although the high-level interface takes care of many consistency problems, this interface cannot take care of all problems by forbidding inconsistencies: changing or deleting data often violates quality requirements. For example, the descriptions of frames as they appear in the frame report are stored as (XML) text fields in the Frame Database. Within these descriptions, FEs may be referenced. For the description of a specific frame, the interface allows the user to mark up only FEs that are really defined and also belong to this frame. If, however, a referenced FE is deleted or its name changes, these references become invalid. A violation of a quality requirement might not be an error but an exception to the requirement. In linguistics it is not always possible to fully specify why some data are an exception to a quality requirement. Therefore, FrameNet has implemented approaches to deal with exceptions. Common exceptions are test cases (e.g., frames having a name starting with "Test"), which are excepted from consistency checking completely.51 Providing the ability to document and tolerate errors, at least for the short and possibly for the long term, also lets the FrameNet development team decide whether and how violations are to be resolved.
Fig. 7.1 illustrates our two quality management approaches, which are motivated by the characteristics of the databases.
A number of data checking programs check annotations in the Annotation Database for correctness, completeness, and style. Each program generates a specific error report showing violations of a quality requirement. Thus, the program provides an algorithmic (imperative) definition of quality. Depending on whether or not it is clear in advance how a certain type of error can be remedied, the checking tools generate either machine-readable output in the form of native database commands that can be applied to the Annotation Database for automatic repair, or human-readable output for manual inspection.
The chief advantages of imperative quality assurance are fast performance, a very specific output, and the possibility of automatically performing repair actions. With regard to speed of execution in particular, checking programs are the only practical way to check annotations for quality.
For checking the quality of the Lexical Database, the Frame Database, and documentation, FrameNet employs the CDE toolkit ([Scheffczyk et al., 2004][Scheffczyk et al., 2003]).52 Here, declarative consistency rules define quality in formal logic. For each consistency rule, CDET generates a formal description of violations and possible repairs, which can then be transformed to other output formats.
Defining quality declaratively has a number of advantages: A general-purpose specification language improves the understanding and formalization of quality requirements. It also allows for reasoning about consistency rules. CDET's fairly simple consistency rule language supports incremental consistency checking - a key to tight process integration.
Since the Lexical Database and the Frame Database are sufficiently "relational" and hold a limited amount of data, declarative consistency management can be used. Because of the advantages it offers, we would prefer to use this approach for the Annotation Database also. However, performance issues and, more importantly, the complexity of the annotated natural-language sentences make declarative consistency checking impractical if not altogether impossible.
The application of formal and rigorous quality management has produced both a significant increase in the quality of FrameNet's data and a decrease in the effort necessary for maintenance and repair. The most important quality requirements are now satisfied by the FrameNet data. Violated quality requirements are extensively documented by the imperative scripts and by CDET. The General Release Notes, R1.3, on the Framenet website contain detailed information about quality requirements and whether they are satisfied or not. In cases of violation, we document which linguistic entities cause the violations.
Although many quality requirements are still violated in the current FrameNet data release, it represents a clear improvement over previous data releases. (1) The most important quality requirements are satisfied. (2) FrameNet and its users have precise knowledge of violations of less important quality requirements. Thus, the quality of FrameNet's data can now be evaluated much more easily by its developers and users.
