Creating Rules for PreTI

PreTI is driven by rules. The rules syntax is briefly discussed in "PreTI Plug-in User's Guide"; we included the description into the User's Guide to make it easier for the user to understand how PreTI operates. However, the rules are not supposed to be created manually; instead, we have provided a set of tools to produce them automatically.

The two sources for PreTI rules are markup schemas and sample documents. A markup schema defines syntactic properties of the markup, the way its elements are allowed to be nested and ordered. A schema language is used to describe a schema; examples of schema languages are XML DTD, RELAX NG, W3C XML Schema, and others.

What remains beyond the scope of a markup definition expressed by a schema is the appearance of text units. For example, titles and subtitles usually consist of a single sentence and do not end with a period; list items begin with a dash and often end with a semicolon; among inline elements, one can distinguish class and variable names or book titles in the text flow without the aid of a markup; abbreviated names, WWW links, e-mail addresses can also be recognized easily. Hints aiding the recognition, while related to human perception, can be reconstructed from sample documents. Sources for the patterns are already marked up, and it just remains to discover common characteristics and differences.

Document markup grammars are often being developed in an attempt to cover a wide range of documents and subject areas, and are consequently quite extensive. In fact, many institutions and individual authors are wise enough to restrict their vocabularies to smaller grammar subsets. Subsetting is a good thing:

PreTI, too, works better with reasonably restricted subsets than with complete grammars. For similar reasons.

Group rules reflect the way group and block elements are nested and ordered, and define when and which group tags to offer to the user. They are generated from the schema. We've chosen RELAX NG, http://relaxng.org/, as the schema language for PreTI; both a tool to convert to this language from most others (with notorious exception of XML Schema), Trang, exists, and parsing and manipulation of the schema can be implemented with acceptual amount of effort. Actually, a parser from Jing has been used; having an API for access to RELAX NG would be helpful though.

The makegroup utility takes the subset, rules-dita.xml and the schema, ditabase.rng, and produces a list of group rules.

java makegroup rules-dita.xml ditabase.rng > rules-dita.group

For each block- or group-level element, it generates an entry. The entry has two kinds of clauses, first-in and follows. The first-inclause specifies that an element can be offered as the opening tag just before the current element. The follows clause advises that the containing element can be closed just before the current one.

For example, the group rules for p, the tag for plain paragraphs, have the following form:

The rules state that a p begins conbody when met inside concept and body inside topic; it also closes lists, tables, and sections. Upon closer investigation, the last condition is not one that helps enter documents; each time a new paragraph is typed, PreTI will offer to close the current section. It happens because the schema allows a mix of sections and paragraphs inside a body, and paragraphs may follow sections.

While this rule can be manually removed from the final rules file, this particular feature of the markup language is questionable. Few documentation styles allow unstructured text after a section or chapter; specific efforts should be employed to visually separate the paragraph from the section which precedes it. Thus, it make sense, both to make the rules more convenient and the markup more consistent, to modify the schema by making it stricter and disallowing paragraphs after sections^[1] When this modification is made to the schema, the last rule goes away.

A similar thing happens with title. It would be natural to expect that a title inside a body starts a new section. However, due to the fact the schema allows a mix of block-level elements inside section without restricting the ordering of the elements, title cannot be used as a section start. Declaring title as the mandatory initial element of section solves the problem and makes entering text easier.

The rule for the title element is:

Additionally, makegroup generates context constraints for block-level elements. They are then merged with recognition patterns based on features of block contents; their generation is discussed in the next chapter.

Rules for block-level elements help choose appropriate markup candidates for paragraphs of text and order them so that the most likely choices appear at the top of the list. They include rules of two kinds: based on the current context (containing and preceding elements) and on the contents of the paragraph.

Context rules are generated my makegroup. Another utility, makeblock, processes sample documents and outputs contents-based rules for each block element. Additionally, it orders the elements by their frequency of appearance in the samples in the decreasing order, so that equally rated, but more frequently used, tags appear first in the list of choices.

java makeblock rules-dita.xml samle1.xml sample2.xml sample3.xml ...

With a representative set of samples, it generates reliable statistical characteristics for each of the block-level elements, based on the number of word and sequences in a paragraph, as well as on initial and final substrings. For example, for title it produces:

and for p:

Which means, that in title there can be two sentences at most, and most likely a single sentence only; never more than 7 and preferably not more than 5 words; a title never ends with a punctuation character. p is the most frequent block-level element, thus it gets one by default; an additional point is added if it ends with a period.

Certain rules cannot be inferred from sample data, but help increase efficiency of the recognition. For example, we find it useful to specify:

for listitem, and then replace the initial dash or asterisk with a corresponding value of the mark attribute during postprocessing.

Yet another utility, makeinline, generates patterns for inlines. It takes the rules file, to determine which tags are for inline markup, and one or more sample documents to derive the patterns from inlines found in them. The syntax for the patterns may look cryptic, but is actually very simple. A pattern consists of two parts (each ending with character z):

The traits are:

For example, =Uz+luz stands for 'an inline starting with an uppercase letter and consisting of on more letters'. The syntax is limited so that it allows use of simple algorithms for accumulation of instances in a pattern — the basis for the learning algorithm employed by PreTI. Full regular expressions would be more powerful, but much more difficult to implement efficiently.

However, in contrast to the other two groups of rules, inline rules can be left empty. A PreTI implementation constructs patterns for the tags used during the text input, and actually performs ad hoc what makeinline does with sample data. In our experience, the only patterns that should be included into the rules file are those for easily distinguishable objects, such as URLs and e-mail addresses. The use of most other tags depends on the style of a particular author and, unless consistence in markup is required among several writers, should only depend on the dynamic learning algorithm.

A makefile Make.rules for the UNIX make utility is included into the distribution. It illustrates the sequence of actions required to generate rules from a subset and a schema. A call

make -f Make.rules IDENT=dita RNG=ditabase.rng SAMPLES="dita-samples/*.xml" 

will generate file rules-dita.all from the subset stored in file rules-dita.xml and sample DITA documents matching dita-samples/*.xml, according to RELAX NG schema ditabase.rng.

The generated rules can then be edited and tested using PreTIplug-in for jEdit. To add a new mode for DITA markup, one can copy rules-dita.all to .jedit/preti/rules-dita.xml and add the following lines to .jedit/properties:

On subsequent runs of jEdit, the default mode for files with extension .dit will be dita. Alternatively, one can set the buffer mode in the Buffer Options dialog, or specify it inside the document in an XML comment according to jEdit's rules.