JarCastingjava-data-text
Package provides java implementation of various text preprocessing methods such as tokenizers, vocabulary, text filter, stemmer, and so on
Install
Add the following dependency to your POM file:
<dependency>
<groupId>com.github.chen0040</groupId>
<artifactId>java-data-text</artifactId>
<version>1.0.3</version>
</dependency>
Features
-
Porter Stemmer
-
Punctuation Filter
-
Stop Word Removal
- Xml Tag Removal
- Ip Address Removal
- Number Removal
-
English Tokenizer
Usage
To use any text filter, just create a new text filter and then calls its filter(...) method.
Porter Stemmer
import com.github.chen0040.data.text.TextFilter;
import com.github.chen0040.data.text.PorterStemmer;
TextFilter stemmer = new PorterStemmer();
List<String> words = Arrays.asList(
"caresses",
"ponies",
"ties",
"caress",
"cats",
"feed",
"agreed",
"disabled",
"matting",
"mating",
"meeting",
"milling",
"messing",
"meetings"
);
List<String> result = stemmer.filter(words);
for (int i = 0; i < words.size(); ++i)
{
System.out.println(String.format("%s -> %s", words.get(i), result.get(i)));
}
StopWord Removal
import com.github.chen0040.data.text.TextFilter;
import com.github.chen0040.data.text.StopWordRemoval;
StopWordRemoval filter = new StopWordRemoval();
filter.setRemoveNumbers(false);
filter.setRemoveIpAddress(false);
filter.setRemoveXmlTag(false);
InputStream inputStream = FileUtils.getResource("documents.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String content = reader.lines().collect(Collectors.joining("\n"));
reader.close();
List<String> before = BasicTokenizer.doTokenize(content);
List<String> after = filter.filter(before);
Punctuation Filtering
import com.github.chen0040.data.text.TextFilter;
import com.github.chen0040.data.text.PunctuationFilter;
TextFilter filter = new PunctuationFilter();
InputStream inputStream = FileUtils.getResource("documents.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String content = reader.lines().collect(Collectors.joining("\n"));
reader.close();
List<String> before = BasicTokenizer.doTokenize(content);
List<String> after = filter.filter(before);