Package org.basex.util.ft

Class Tokenizer

java.lang.Object

org.basex.util.ft.FTIterator

org.basex.util.ft.Tokenizer

All Implemented Interfaces:

Comparable<org.basex.util.ft.LanguageImpl>, Iterator<FTSpan>

Direct Known Subclasses:

WesternTokenizer

public abstract class Tokenizer
extends FTIterator

Abstract tokenizer.

Author:

BaseX Team 2005-21, BSD License, Jens Erat

Constructor Summary

Constructors

Constructor	Description
Tokenizer()

Method Summary

Modifier and Type	Method	Description
final int	compareTo(org.basex.util.ft.LanguageImpl o)
final boolean	equals(Object obj)
int	hashCode()
protected abstract byte	prec()	Returns the precedence of the processor.
static boolean	supportFor(Language language)	Checks if the language is supported by the available tokenizers.
boolean	supports(Language ln)	Checks if the specified language is supported.

Methods inherited from class org.basex.util.ft.FTIterator

init, nextToken, remove

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining, hasNext, next

Constructor Details

Tokenizer

public Tokenizer()

Method Details

supportFor

public static boolean supportFor(Language language)

Checks if the language is supported by the available tokenizers.

Parameters:

language - language to be found

Returns:

result of check

prec

protected abstract byte prec()

Returns the precedence of the processor. If two language implementations support the same language, the processor with the higher precedence will be selected.

Returns:

precedence

supports

public boolean supports(Language ln)

Checks if the specified language is supported.

Parameters:

ln - language

Returns:

true if language is supported

equals

public final boolean equals(Object obj)

Overrides:

equals in class Object

compareTo

public final int compareTo(org.basex.util.ft.LanguageImpl o)

Specified by:

compareTo in interface Comparable<org.basex.util.ft.LanguageImpl>

hashCode

public int hashCode()

Overrides:

hashCode in class Object