Class CatalanWordTokenizer

java.lang.Object
org.languagetool.tokenizers.WordTokenizer
org.languagetool.tokenizers.ca.CatalanWordTokenizer
All Implemented Interfaces:
org.languagetool.tokenizers.Tokenizer
public class CatalanWordTokenizer extends org.languagetool.tokenizers.WordTokenizer
Tokenizes a sentence into words. Punctuation and whitespace gets its own token. Special treatment for hyphens and apostrophes in Catalan.
Author:
Jaume Ortolà

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected org.languagetool.rules.spelling.morfologik.MorfologikSpeller
    speller
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    CatalanWordTokenizer()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    List<String>
    tokenize(String text)
     

    Methods inherited from class org.languagetool.tokenizers.WordTokenizer

    getProtocols, getTokenizingCharacters, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • speller

      protected org.languagetool.rules.spelling.morfologik.MorfologikSpeller speller

  • Constructor Details

    • CatalanWordTokenizer

      public CatalanWordTokenizer()

  • Method Details

    • tokenize

      public List<String> tokenize(String text)
      Specified by:
      tokenize in interface org.languagetool.tokenizers.Tokenizer
      Overrides:
      tokenize in class org.languagetool.tokenizers.WordTokenizer
      Parameters:
      text - Text to tokenize
      Returns:
      List of tokens. Note: a special string CA_APOS is used to replace apostrophes, and CA_HYPHEN to replace hyphens.