ML for SS: Embeddings and topic modeling

What are “vector space models”

Different ways of converting some abstract information into numeric information
- Focus on maintaining some of the underlying structure of the abstract information
Examples (from smallest to largest input):
- Word vectors:
  - Word2vec
  - GloVe
- Paragraph/document vectors:
  - Doc2Vec
- Sentence vectors:
  - Universal Sentence Encoder
- Topic vectors:
  - Latent Dirichlet Allocation (LDA)

Word vectors

Instead of coding individual words, encode word meaning
The idea:
- Our old way (encode words as IDs from 1 to N) doesn’t understand relationships such as:
  - Spatial relations
  - Grammatical relations (weakly when using stemming)
  - Social relationships
  - etc.

Word vectors try to encapsulate all of the above implicitly, through by encoding words as a vector based on how features manifest themselves in text

Word vectors: Simple example

words	f_animal	f_people	f_location
dog	0.5	0.3	-0.3
cat	0.5	0.1	-0.3
Bill	0.1	0.9	-0.4
turkey	0.5	-0.2	-0.3
Turkey	-0.5	0.1	0.7
Singapore	-0.5	0.1	0.8

The above is a simplified illustrative example
Notice how we can tell apart different animals based on their relationship with people
Notice how we can distinguish turkey (the animal) from Turkey (the country) as well

What it retains: word2vec

Relations are retained as vectors between points (distance + direction)

What it retains: GloVe

How does word order work?

Infer a word’s meaning from the words around it

Refered to as CBOW (continuous bag of words)

How else can word order work?

Infer a word’s meaning by generating words around it

Refered to as the Skip-gram model

An example of using word2vec

In Brown, Crowley and Elliott (2020 JAR), word2vec was used to provide assurance that the LDA model works reasonably well on annual reports
1. We trained a word2vec model on random issues of the WSJ (247.8M words)
2. The resulting model “understood” words in the context of the WSJ
3. We then ran a psychology experiment (word intrusion task) on the algorithm

The task is to find which word doesn’t belong

Each question consisted of 3 words from 1 topic and 1 intruder from another topic
- Ex.:
  - Laser, Drug, Viral, Therapeutic
  - Supply, Steel, Capacity, Losses
  - Relief, Louisiana, Cargo, Assisted

Results

Loading in word2vec with Gensim

The gensim package comes with the ability to download word2vec and GloVe vectors from a repository
The code below would allow you to download a model trained on Google News
- In this model, each word is represented as a 300-dimensional vector

import gensim
import gensim.downloader

base_w2v = gensim.downloader.load('word2vec-google-news-300')

Note: The model it downloads is 1.7GB

The model will be stored in ~/gensim_models/
- ~ represents your user directory
- You can safely delete this directory after you are done using it

Examining word2vec: Odd one out

base_w2v.doesnt_match(['Queen', 'King', 'Prince', 'Peasant'])

'Peasant'

base_w2v.doesnt_match(['Singapore', 'Malyasia', 'Indonesia', 'Germany'])

'Germany'

base_w2v.doesnt_match(['Euro', 'USD', 'RMB', 'computer'])

'computer'

base_w2v.doesnt_match(['mee goreng', 'char kway teoh', 'laksa', 'hamburger'])

'hamburger'

Examining word2vec: Closest words

base_w2v.most_similar(['Earnings'])

('Pro_Forma_EPS', 0.6441532373428345) ('Diluted_EPS', 0.636042058467865) 
 ('Goodwill_Impairment', 0.6357625126838684) ('Tax_Expense', 0.6289322376251221) 
 ('Reconciling_Items', 0.6285154819488525) ('Restructuring_Charges', 0.6268271207809448) 
 ('Backs_FY##', 0.6254147291183472) ('Raises_FY##_EPS', 0.6230234503746033) 
 ('Restructuring_Charge', 0.6216667294502258) ('FFO_Per_Share', 0.6207219958305359)

base_w2v.most_similar('IASB')

('Accounting_Standards_Board', 0.7211726307868958) ('FASB', 0.6697319149971008) 
 ('IAASB', 0.6319378614425659) ('IAS##', 0.6150702834129333) 
 ('FASB_IASB', 0.593984842300415) ('Exposure_Draft', 0.5892050266265869) 
 ('Board_IASB', 0.5818656086921692) ('IFRS', 0.5813880562782288) 
 ('GNAIE', 0.5802473425865173) ('Solvency_II', 0.574397087097168)

Examining word2vec: Closest words

base_w2v.most_similar(['KPMG'])

('PwC', 0.8044512867927551) ('PricewaterhouseCoopers', 0.8032213449478149) 
 ('Deloitte', 0.7856791019439697) ('Grant_Thornton', 0.7815379500389099) 
 ('PriceWaterhouseCoopers', 0.7609084248542786) ('KMPG', 0.7575340270996094) 
 ('PricewaterhouseCoopers_PwC', 0.7438496351242065) ('Pricewaterhouse_Coopers', 0.7163813710212708) 
 ('Delloitte', 0.7009097337722778) ('KPMG_LLP', 0.7008424401283264)

base_w2v.most_similar(['Arthur_Andersen'])

('Arthur_Andersen_LLP', 0.7720072269439697) ('Peat_Marwick', 0.6542829275131226) 
 ('Price_Waterhouse', 0.6524070501327515) ('KPMG_Peat_Marwick', 0.6093755960464478) 
 ('Peat_Marwick_Mitchell', 0.6006763577461243) ('&_Lybrand', 0.5949062705039978) 
 ('Arthur_Andersen_accounting', 0.559570848941803) ('auditor_Arthur_Andersen', 0.5569155812263489) 
 ('KPMG', 0.5496521592140198) ('Price_Waterhouse_LLP', 0.5493941903114319)

Examining word2vec: Analogies

man : King :: woman : ?

Mathematically: \(King - man + woman = ?\)

base_w2v.most_similar(positive=['King', 'woman'], negative=['man'])

('Queen', 0.5515626668930054) ('Oprah_BFF_Gayle', 0.47597548365592957) 
 ('Geoffrey_Rush_Exit', 0.46460166573524475) ('Princess', 0.4533674716949463) 
 ('Yvonne_Stickney', 0.4507041573524475) ('L._Bonauto', 0.4422135353088379) 
 ('gal_pal_Gayle', 0.4408389925956726) ('Alveda_C.', 0.4402790665626526) 
 ('Tupou_V.', 0.4373864233493805) ('K._Letourneau', 0.4351031482219696)

The sleight of hand behind this

Word2Vec implementations usually bar a word in the analogy from being an output
- E.g., it will never report man : King :: woman : King
  - But this is actually the mathematical answer

analogy = base_w2v['King'] + base_w2v['woman'] + base_w2v['man']
analogy = analogy / np.linalg.norm(analogy)
print('King', np.linalg.norm(analogy - base_w2v['King']))

King 1.9888592

print('Queen', np.linalg.norm(analogy - base_w2v['Queen']))

Queen 2.7364814

It’s still pretty good though!

Note that since word2vec’s original answer was Queen, this implies it was second best
- If Queen is the closest word to King, then this would be mathematically uninteresting
  - It’s actually 7th though!

base_w2v.most_similar('King')

[('Jackson', 0.5326348543167114), ('Prince', 0.5306329727172852), ('Tupou_V.', 0.5292826294898987), ('KIng', 0.5227501392364502), ('e_mail_robert.king_@', 0.5173623561859131), ('king', 0.5158917903900146), ('Queen', 0.5157250165939331), ('Geoffrey_Rush_Exit', 0.49920955300331116), ('prosecutor_Dan_Satterberg', 0.49850785732269287), ('NECN_Alison', 0.49128594994544983)]

What is this good for?

You care about the words used, by not stylistic choices
- Abstraction
You want to crunch down a bunch of words into a smaller number of dimensions without running any bigger models (like LDA) on the text.
- E.g., you can toss the 300 dimensions of the Google News model to a Lasso or Elastic Net model
  - This is a big improvement over the past method of tossing vectors of word counts at Naive Bayes
You want synonyms for a set of words that are selected in a less-researcher-biased fashion
- You can even get n-gram synonyms this way
- A popular method for augmenting small dictionaries

Exercise: Trying out word2vec

Colab file available at https://rmc.link/colab_w2v

This set of exercise is to help you understand about what word2vec is good at
- As well as what it isn’t good at