Package org.basex.util.ft

Class WesternTokenizer

java.lang.Object
org.basex.util.ft.FTIterator
org.basex.util.ft.Tokenizer
org.basex.util.ft.WesternTokenizer
All Implemented Interfaces:
Comparable<org.basex.util.ft.LanguageImpl>, Iterator<FTSpan>
public final class WesternTokenizer extends Tokenizer
Full-text tokenizer.
Author:
BaseX Team 2005-21, BSD License, Christian Gruen

  • Constructor Details

    • WesternTokenizer

      public WesternTokenizer(FTOpt fto)
      Constructor.
      Parameters:
      fto - full-text options

  • Method Details

    • init

      public WesternTokenizer init(byte[] txt)
      Description copied from class: FTIterator
      Initializes the iterator.
      Specified by:
      init in class FTIterator
      Parameters:
      txt - text
      Returns:
      self reference

    • hasNext

      public boolean hasNext()

    • next

      public FTSpan next()

    • nextToken

      public byte[] nextToken()
      Description copied from class: FTIterator
      Returns the next token. May be called as an alternative to Iterator.next() to avoid the creation of new FTSpan instances.
      Specified by:
      nextToken in class FTIterator
      Returns:
      token

    • prec

      protected byte prec()
      Returns the precedence of the processor. If two language implementations support the same language, the processor with the higher precedence will be selected.
      Returns:
      precedence

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • supports

      public boolean supports(Language ln)
      Checks if the specified language is supported.
      Parameters:
      ln - language
      Returns:
      true if language is supported

    • equals

      public final boolean equals(Object obj)
      Overrides:
      equals in class Object

    • compareTo

      public final int compareTo(org.basex.util.ft.LanguageImpl o)
      Specified by:
      compareTo in interface Comparable<org.basex.util.ft.LanguageImpl>

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object