Title: Improve feature selection method of web page language identification using fuzzy ARTMAP

Authors: Choon-Ching Ng, Ali Selamat

Addresses: Faculty of Computer Science and Information Systems, University of Technology Malaysia (UTM), 81310 Skudai, Johor Bahru, Johor, Malaysia. ' Faculty of Computer Science and Information Systems, University of Technology Malaysia (UTM), 81310 Skudai, Johor Bahru, Johor, Malaysia

Abstract: The information available in languages other than English on the World Wide Web and global information systems is increasing significantly. Different languages can be produced by using one particular script such as Arabic, Persian, Urdu and Pashto that use Arabic script letters. The issue is how to produce reliable features of a web page that is to undergo language identification. Incorrectly identifying the language results in garbled translations as well as faulty and incomplete analyses. The aim of this study is to enhance the effectiveness of feature selection method of web page language identification. We have investigated total N-grams, N-grams frequency, N-grams frequency document frequency, and N-grams frequency inverse document frequency of web page language identification. From the experimental results, it is proven that N-grams frequency gives the most promising result compared to other feature selection methods.

Keywords: feature selection; N-grams frequency; fuzzy ARTMAP; web pages; language identification; web page languages; language translation; web page analysis; reliability; neural networks; adaptive resonance theory.

DOI: 10.1504/IJIIDS.2010.036897

International Journal of Intelligent Information and Database Systems, 2010 Vol.4 No.6, pp.629 - 642

Received: 23 Nov 2009
Accepted: 11 May 2010
Published online: 15 Nov 2010 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article