Topic modeling the “Dispatch” folder
In this lesson we learn how to find the most important topics in order to analyse data. However, the jupyter notebook with python should help us to organize our data in order to prepare data for our analysis. Our task is to run topic modelling on the “Dispatch” folder with using the provided R-script.
Jupyter notebook:
Dispatch 1864 20 topics:
Please find the python code for 20 topics.
# %%time
# # Build LDA model
# number_of_topics = 20
# lda_model = gensim.models.LdaModel(corpus=corpus,
# id2word=dictionary,
# num_topics=number_of_topics,
# update_every=20,
# passes=100,
# alpha='auto')
# print("-"*50)
# path = "./models_new/"
# lda_model.save(path+'dispatch_1864_20.lda')
In order analyse our data, we should make some adjustment in the python srcipt for 30 and later for 40 topics.
Dispatch 1864 30 topics:
# %%time
# # Build LDA model
# number_of_topics = 30
# lda_model = gensim.models.LdaModel(corpus=corpus,
# id2word=dictionary,
# num_topics=number_of_topics,
# update_every=20,
# passes=100,
# alpha='auto')
# print("-"*50)
# path = "./models_new/"
# lda_model.save(path+'dispatch_1864_30.lda')
Dispatch 1864 40 topics:
The anylsis shows that most important topics in 1864 were enemy, army and general for 20 and 40 topics. For the 30 topics the most dominant topics were states, south etc. In my observation I analysed just year 1864. The entire Dispatch (1860-1865) was analysed by Rob Nelson (http://dsl.richmond.edu/dispatch/Topics). However, Nelson organized all 40 topics in to a series of themes in order to present topics that had some relation to one another. According to Nelson, the most important themes were slavery, nationalism and patriotism, soldiers and military conflict. This data for 1864 can be also organized into themes just like by Nelson.