CSE256 Assignment 1: Text Classification

Yi Rong <hi@rongyi.ai>

April 11, 2022

Introduction

In this report, we discuss the various ways of data pre-processing and feature engineering

for a text classification task. We first start by giving an overview of the classification task,

the model used, and the given baseline implementation in Section 2. Then we iterate on

top that version guided by the project documentation to use TF-IDF for token weighting to

achieve better accuracy, detailed in Section 3. Finally we present our various approaches for

visualization: we divide the process into four main parts: Data IO, Tokenizer, Vectorizer,

and Classifier. Among those, we will mainly focus on improving the accuracy by tweaking

Tokenizer and Vectorizer, and won’t focus too much on modifying classifier or data reading.

In the baseline implementation provided along the project documentation, a simple word

tokenization with whitespace is used to split the sentences, after which the tokens are fed into

a Bag of Words vectorizer to generate a feature vector of vocabulary occurance frequencies

to represent the review.

Labels are encoded using the scikit-learn’s default LabelEncoder, which just produces

the categorical (binary) label, and the Labels will always be encoded as such across all the

experiments we perform, as we mainly focus on transforming the features.

by running the previous CountVectorizer on tokens to calculate the BoW representation,

and then perform a TfidfTransformer to return weighted frequency vector.

Using this approach, we can already see accuracy improvement compared to the baseline

with just the default hyperparameter. The TF-IDF has a parameter n, which is the maximum

n-gram feature length ( in scikit-learn, it’s ngram_range ). And in the classifier part, we

can change the regularization strength C to control how much we want the regularization

to be, or how much we trust the training data ( in scikit-learn, C is set to

1

λ

, the inverse

Since in our experiments, setting the inverse of regularization strength C to 1 and the

our Guide Feature Engineering attempt.

Independent Feature Engineering

and a vectorizer (TF-IDF). For the tokenizer, we use the BPE Tokenizer from Huggingface,

with pre-tokenization set to Whitespace so that sentences in our training data is first split

into words. We then train our tokenizer over the entire training dataset plus the unlabeled

dataset, since we don’t need any labeling data in this stage, and we’d like to learn as much

about the words used in the review as possible.

After we trained our tokenizer, we encode our training data into tokens by running the

BPE Tokenizer over our training set. Then those tokens are fed into TF-IDF for further

ranking similar to the approach in Section 3.

Similar to what we do in the previous section, we also perform a grid search on the

hyper-parameter N for TF-IDF, and get the following result.

and Word2Vec [1] comes to our attention. The idea is that in addition to train our tokenizer

over the dataset to produce better tokens out of sentences, we also train our vectorizer over

the dataset to better produce feature vectors that suits this review database.

Since our BPE Tokenization provides a reasonable improvements over previous approaches,

we keep using it as our tokenizer, and append a Word2Vec implementation after that to turn

the training tokens into features. We use the gensim package for their implementation, and

leave every hyper-parameters by default. The Word2Vec model is trained with 5 epochs of

data combined from the training set and the unlabeled set to help the vectorizer know the

review dataset better. By default it uses the skip-gram algorithm, and produces a key-value

map from every word in the dataset to a fixed-length vector. We also added additional

average of all the word vectors.

feature engineering applied to the text classification task.

Table 3: Dev Accuracy of Our Approaches

Name Tokenizer Vectorizer Classifier Train Acc. Dev Acc.

Baseline space BoW LR (C=1) 0.98210 0.77729

TF-IDF space BoW + TF-IDF (N=2) LR (C=1) 0.91532 0.79257

BPE + TF-IDF BPE TF-IDF (N=3) LR (C=1) 0.92994 0.80349

BPE + W2V BPE Word2Vec LR (C=1) 0.80641 0.80568

The TF-IDF (Assignment part 2.1) achieves better accuracy than the baseline because

of the automatic weighting of tokens helps the regression algorithm to prioritize learning

on the words that actually mean things rather than common words like “the”. The third

BPE approach (Assignment part 2.2) performs better than the second TF-IDF one, because

we use a brand new tokenizer that better understand the dataset and can do sub-word

tokenization rather just spliting a sentence by whitespaces. The final one BPE + Word2Vec

(Assignment part 2.2) got the highest accuracy percentage, due to the fact that both its

tokenizer and vectorizer part has some understanding of the underlying dataset, enabling

other approaches.

Feature Engineering approaches here.

In this report, we attempt to improve the feature extraction for text classification, and

of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).

[2] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation

of rare words with subword units. arXiv preprint arXiv:1508.07909 (2015).