Sentiment Analysis of Work from Home Activity using SVM with Randomized Search Optimization

Government policy on a problematic topic can lead to pros and cons, including the implementation of work from home during the COVID-19 pandemic in Indonesia. Lots of social media users express their opinions through social media, such as Twitter. Using Twitter API, data on Twitter can be obtained freely, so it can be utilized for sentiment analysis. Therefore, this study contains an analysis of public sentiment on the work from home policy using various preprocessing methods and Support Vector Machine with randomized search optimization. The result shows that the use of the acronym expansion method, slang word translation, and emoji translation in the preprocessing stage can increase the F1 Score value. The best F1 score results obtained were 83.362%. The results of the preprocessing method are used to predict unlabeled data. Prediction results show that 62.35% of tweets have positive sentiments, on the contrary, 37.65% of tweets have negative sentiments. So, it can conclude that most netizens support the policy of work from home.


Introduction
Government policy on a problematic topic can lead to pros and cons, including the application of work at home activity during the COVID-19 pandemic.This policy applies to all fields of work, from the economic, education, manufacturing, and government sectors.Implementing new work patterns, many of the affected communities need to adapt to the new environments.Frequently, some workers must be terminated without any severance pay.The existence of these pros and cons is against the government's desire to see the public sentiment, whether the policies implemented are appropriate or not.Of course, the results of this sentiment analysis will be very meaningful in applying other policies.Therefore, the issues raised for this research will greatly contribute to the government.
One of the social media that is often used to express opinions, complaints, or open criticism is Twitter.Through the tweet feature, users can send whatever is going on.Unlike other social media, Twitter has a unique and instant way of interacting, such as retweet, retweet with comments, like, and reply.So, if there is a hot topic being discussed, information will spread quickly.To answer the problems that have been mentioned, what needs to be done is to analyze public sentiment or extract opinions.Quoting from the book Sentiment Analysis and Opinion Mining, the notion of opinion mining is a field of study that analyzes opinions, sentiments, evaluations of a person, attitudes, and emotions towards an entity [1].
There have been several studies that have conducted sentiment extraction of social media users, such as Twitter.The methods and approaches used are also quite diverse, so the accuracy results obtained vary.Research conducted by Santoso et al. [2] used the SVM classification model to analyze sentiment about lecturer evaluation.The dataset to be classified needs to be carried out in a series of preprocessing stages, such as tokenization, case folding, filtering, and stopword removal.Then, the data is classified using the Support Vector Machine (SVM).The accuracy results obtained are 67.83% by applying cross-validation at the evaluation stage.In another study [3], the SVM method was developed by adding a lexicon-based method.However, the accuracy results obtained are not good, only 40%.accuracy results with cross-validation between the two algorithms.The result is that the SVM algorithm works better than KNN, with an accuracy value of 67.90%.The use of the k value in cross-validation, if it is too small, will reduce the value of accuracy, but if the k value is too large will result in a large accuracy value.Based on this study, the most optimum value of k-fold is 10-fold.
To increase the value of accuracy, this study will use the preprocessing method suggested in the research conducted by Jianqiang Z. et al. [5].This study compares 6 types of preprocessing methods.As a result, deleting URLs has less effect on classifier performance.This is because the URL does not contain information useful for sentiment classification.This study also discusses the appropriate preprocessing method applied to the SVM classifier model.The performance of this model will improve if you remove numbers, replace negation words, and expand acronyms.A study conducted by Wijayanto, et al. [6], compared various preprocessing methods to analyze sentiment using supervised data.The amount of data used greatly affects the results.The larger the dataset used, the better the results, because a large dataset can represent all the possibilities.In addition, handling slang words will help increase the accuracy.
To optimize hyperparameter, Ahmad M. et al [7] conducted a study that focuses on SVM optimization for sentiment analysis.This method helps improve the performance of the classification algorithm, by changing the parameters used.This study implements grid search to tune the hyperparameters and cross-validation to evaluate the performance of the classifier.The result shows, grid search performs well for hyperparameter exploration.In another study conducted by Bergstra, J. et al [8] compare the performance of grid search and random search.The result shows that random search performance is more efficient than grid search because grid search allocates too many trials for the exploration of less important dimensions.
Several studies that use the SVM method showed poor results.On the other hand, various methods have been proposed to help improve accuracy, such as by selecting preprocessing methods and optimizing hyperparameters with Random Search.The purpose of this study is to analyze public sentiment related to the work from the home policy implemented by the government during the COVID-19 pandemic using Support Vector Machine with Randomized Search to obtain maximum accuracy.
In addition, this study also compares the appropriate preprocessing methods used for the built models.

Research Method
This study proposes a sentiment analysis system that uses Support Vector Machine (SVM) as a classifier with Randomized Search as an algorithm to tune the hyperparameter.Figure 1 shows the several stages to build a good machine learning model for predict the sentiment.After completing several stages above, the best model can be determined from the evaluation stage.The best model is used to predict sentiment from the testing data.Figure 2 shows the stages to predict the data testing sentiment.

Data Collecting
At this stage, the tweets data from Twitter will be collected.The method used is using the python programming language by utilizing the Twitter API.The data taken is in the form of Indonesian-language tweets containing various keywords related to work from home (WFH) during the COVID-19 pandemic in Indonesia.Table 1 shows the keyword used for data collecting.The results obtained will be grouped into two types of data, namely training data and data to be predicted.The training data is collected as much as 5,000 data, then the data will be labeled manually.As for the data to be predicted as much as 50,000 data.This data is referred to as data testing.This data will predict the sentiment by using the built model.

Data Labeling
After getting the data from the data collection stage, the 5,000 data will be used for the training phase.This data is referred to as data training.Thus, the data needs to be labeled.At this stage, the data is manually labeled into two types of labels, positive and negative, for each data in training set.Labeling is done using predetermined labeling criteria.Table 2 shows the labeling criteria.After determining the labeling criteria, the training data is labeled according to the criteria mentioned before.Table 3 shows some samples of the results.To find out the comparison of the results of data labeling, Figure 3 shows the comparison of the number of data labeled positive and negative.At this stage, various preprocessing methods will be implemented.Quoting from the book "Data Preprocessing in Data Mining" [9], preprocessing is a series of steps performed to perform data cleaning, data integration, data transformation, and data reduction.This stage is important because if you analyze data that has not been processed carefully, it can provide misleading information.If there is a lot of irrelevant information or has a lot of data noise, the data quality will decrease.To produce high-quality data requires understanding and applying the appropriate preprocessing methods used for each case study.
The first step in preprocessing stage is the acronym expansion.An acronym is an abbreviation of several combinations of letters, syllables, or words that are commonly used in everyday life.At this stage, the acronym will be expanded into certain phrases.The acronym expansion is based on a dictionary developed by Prakoso [10].The dictionary consists of 4,085 acronyms and they are expanding phrases.Table 4 shows the sample acronym dictionaries used.The second step is converting text to lowercase.This method is called case folding.The purpose of this method is to convert the text into a standard form so that it is easy to analyze.Table 5 shows the difference between tweets before and after case folding.
The next is removing noises that contains in the text, such as numbers, punctuation, excess whitespace, usernames, URLs, and hashtags.After getting clean text, the next stage is slang word translation.Slang is a non-formal language that is usually used in certain social circles.At this stage, the slang words will be converted into formal language.The translation of slang words is carried out based on the "alay" dictionary from the results of research conducted by Ibrohim, et al [11].The dictionary used consists of 15,167 slang words along with their meanings.Table 7 shows the differences between tweets before and after slang translation.

Original Tweet
Preprocessed Tweet @selphieusagi Aku lebih nyaman wfh walaupun skrg full wfo.Kangen wfh walaupun jam 9 malem masih kerja aku lebih nyaman wfh walaupun sekarang full wfo kangen wfh walaupun jam malem masih kerja Selama ini wfh terus, dpt tmn baru jg kenalan online, mau sampai kapan:( selama ini wfh terus dapat teman baru juga kenalan online mau sampai kapan @dodioc enak banget wfh, aku hari ini harus dikantor enak banget wfh aku hari ini harus dikantor Emoji is a pictorial character in the text that is usually used to express something.Emojis are often found in text conversations or chats.To extract the meaning for each emoji, it needs an emoji translation.Therefore, the emoji will be translated into a meaningful word using the emoji dictionary.Table 8 shows the differences between tweets before and after emoji translation. The other method is stopword removal.Stopwords are words that often appear in sentences, but are considered meaningless, such as "yang," "jika," "dari," and "untuk."Stopword removal is a method used to remove words that are less important and save important words (wordlist).At this stage, we will use a stopword list from the Sastrawi library.Table 9 shows the differences between tweets before and after stopword removal.The last step is stemming.Stemming is a method used to transform each affixed word in a sentence into a root word.The goal is to simplify words, so they are easy to analyze.For example, the words "taking," "taking," and "taking" will be changed to the words "take" because they have the same meaning.At this stage, this research will utilize a stemmer from the Sastrawi library.Table 10 shows the difference between tweets before and after stemming.

Term Weighting
Feature weighting refers to the process of calculating the weight of categorical information for each feature in the feature set according to a specified weighting algorithm [12].One type of feature weighting is TF-IDF (Term Frequency-Inverse Document Frequency).According to Wang Y. [13], TF-IDF is a powerful feature weighting Fatihah Rahmadayana, Yuliant Sibaroni Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi) Vol. 5 No. 5 (2021) 936 -942 DOI: https://doi.org/10.29207/resti.v5i5.3457Creative Commons Attribution 4.0 International License (CC BY 4.0) 940 method, which comprehensively measures the frequency of occurrence and distribution of words to determine the weight.TF-IDF consists of TF (Term Frequency) and IDF (Inverse Document Frequency).TF is the frequency with which a word appears in the document.While IDF is the inverse of document frequency.

Support Vector Machine
Support Vector Machine (SVM) is one of the supervised machine learning algorithms used for classification.This algorithm uses a kernel to map data to a higher dimensional space to classify data that cannot be separated linearly [14][15].The use of the kernel requires a set of parameters that can be changed as needed, these parameters are called hyperparameters.This study implements two kinds of the kernel, linear kernel and Radial Basis Function (RBF).So, the optimized hyperparameters are C and gamma.Parameter C serves as a regulatory parameter.While the gamma parameter is the coefficient used in RBF, poly, or sigmoid kernels.

Hyperparameter Tuning using Randomized Search CV
Hyperparameters are parameters that are set before the learning process begins.This parameter is adjustable and can directly affect how well the model performs.To find the most optimal hyperparameters, various strategies are needed to adjust, the most convenient way is to try various combinations of hyperparameters.However, over time many approaches have been proposed to optimize these hyperparameters, such as Grid Search and Random Search.Grid search is a traditional algorithm for performing hyperparameter tuning.This algorithm is a complete search algorithm through a certain subset of the hyperparameter space of a learning algorithm.The grid search algorithm is tested with several performance metrics, usually by using the crossvalidation method on the data training.
Random Search Algorithm refers to an algorithm that uses randomness or probability in the method definition, and in the literature, it can be called the Monte Carlo method or the stochastic algorithm method [14].This algorithm works by iterating the search space randomly [11].In contrast to Grid Search, not all parameter values are tested, but several parameter settings are sampled from a specified distribution [15].Then, the algorithm evaluates the model using cross-validation.According to research conducted by Bengio, et al. [10], the random search algorithm shows a more efficient performance than the grid search algorithm.The grid search algorithm allocates too many trials for the exploration of less important dimensions.The classification algorithm that will be optimized is the SVM.Table 11 shows the hyperparameters combinations of the SVM that will be optimized.To optimize the hyperparameters used in the SVM, this research implements hyperparameter tuning using a randomized search algorithm.This process is done by implementing the RandomizedSearchCV function from the sci-kit-learn library [7].Table 12 shows the customized random search algorithm in this study.The RandomizedSearchCV function will randomly select n_iter hyperparameters and evaluate the results.In this study, the number of iterations carried out was 20 iterations.While the evaluation method is carried out by applying cross-validation.This method is used to evaluate the performance of a model or algorithm where the data is separated into two subsets, namely learning process data and validation data.Based on research conducted by Lidya, et al. [4], the most optimum value of k-fold used is k = 10.So, this study will use 10-fold for cross-validation.To get a good model, the search function will evaluate it using F1 weighted calculations.

Choose the best performance model
This section will explain the results of selecting best model in this study, starting from the data collection stage, data preprocessing, feature extraction, hyperparameter tuning, to the evaluation process.This study monitors the performance of the SVM algorithm which is optimized using the Randomized Search optimization algorithm method.This method selects the most optimal combination of parameters based on the highest F1 Score.
This study also tested several preprocessing methods using data training.The data used for training as much as 5.000 data.The test was carried out by comparing the F1 Score values obtained from the use of acronym expansion preprocessing methods, slang word translation, and emoji translation.The aim is to determine the performance of the preprocessing method in processing raw data into data that is ready to be classified.Table 13 shows the average F1 score obtained, along with the parameters used in SVM, for each scenario carried out.Based on the scenario testing shows that the best SVM parameter obtained from RandomizedSearch with CV.All tested scenarios show same result.The best parameter combination for this research is using RBF kernel, with regularization parameter (C) equals to 1, and the kernel coefficient for RBF (gamma) equals to 100.
Based on the tests that have been carried out, the results show that the use of all preprocessing methods shows the best results, namely with an F1 Score of 83.362%.On the other hand, if the three methods are not applied, it shows a slight difference in the results obtained, which is 82.491%.The three methods synergize well if applied simultaneously, this is shown in the decrease in the value of the F1 Score if one method is not applied.So, it can be concluded that the most optimal preprocessing method is using all preprocessing methods, including acronym expansion, slang word translation, and emoji translation.This method will be used to predict data testing.

Predict the sentiment of data testing
In the previous stage, the model selection has been carried out using training data totaling 5,000 data.The result shows the best preprocessing methods are acronym expansion, slang word translation, and emoji translation.While the best SVM parameter is using radial basis function (rbf) kernel, the regularization parameter (C) is 1, and the kernel coefficient (gamma) is 100.
To determine the level of public sentiment towards the implementation of the work from home policy in Indonesia, this study extracts 50,000 unlabeled data testing that obtained from data collecting stage to represent public sentiment.This data will be predicted using best model mentioned above.Figure 4 shows the results of sentiment prediction on the testing data.The bar chart above shows that 34,921 tweets or 62,35% of tweets have positive sentiments.This can be interpreted that 62.35% of tweets show support for the work from home policy implemented by the Indonesian government during the COVID-19 pandemic.On the contrary 21,086 tweets or 37.65% of tweets have negative sentiments.So, it can be concluded that most Indonesian netizens support the work from home policy.

Conclusion
Based on the research that has been done, it shows that the use of acronym extension methods, slang word translation, and emoji translation in the preprocessing stage can increase the F1 Score value.The best results obtained are 83.362% by applying the three preprocessing methods.If the three methods are not applied, it shows a slight difference in the results obtained, which is 82.491%.The three methods synergize well if applied simultaneously, this is shown in the decrease in the value of the F1 Score if one method is not applied.

Figure 1 .
Figure 1.The stage to build a good model for predict the sentiment.

Figure 2 .
Figure 2. The stage to extract unlabeled data.

Figure 3 .
Figure 3. Value counts of positive and negative labeled data 2.2.Preprocessing

Figure 4 .
Figure 4. Prediction Result of Public Sentiment towards Work from Home Policy

Table 1 .
Keyword used for

Table 2 .
Sample of acronym dictionary

Table 3 .
Case folding processing results

Table 4 .
Data cleansing processing results

Table 7 .
Stopword removal processing results

Table 8 .
Stemming processing result

Table 10 .
Randomized Search Algorithm

Table 11 .
Scenario test results