|
|
|
|
|
|
by sixhobbits
29 days ago
|
|
|
I do some similar charting etc with telegram data dumps that you can still get from the "telegram lite" desktop app even though they removed the export functionality from the main app. For removing noise you might want to look into TF-IDF instead of the manual method described in the post that I didn't understand. It basically looks for words common across the whole corpus as noise or ones that appear within a specific chat much higher than the whole dataset as interesting. You can also do some fun stuff by finding phrases used asymmetrically eg more by one person in the convo than the other, or over time. Wordclouds per person are also fun! |
|
|
TF-IDF was the first thing I tried - it works great for stopwords but it doesn't handle cross-language bleed of filler words well, and the short life-event messages ("he died", etc) use common words and get aggressively down-weighted.
I had some asymmetry analysis when looking at directional sentiment and per-person question rates - that's fun indeed!
I also went with the Jaccard convergence and the endearment categories instead of wordclouds, so that I could see how word choices are changing across time.