Ask HN: Predict Stocks with Tweet Sentiment

[alphameta]

Hello everyone,

I am new to DS/ML and currently trying my head on this project. I would like to predict the stock price for one or two day ahead based on the historic price and the tweet sentiment for that particular stock.

Currently, I have around 350,000 tweets dataset for a particular stock and did sentiment analysis using Vader Lexicon analysis, and the stock dataset from yfinance. Plotted both the charts to see some correlation and found out twitter sentiment does have affect on the stock price.

I am confused with how to use this sentiment to predict the stock or how should i proceed further with this project or is this something useless if I put it on my Resume.

Thanks

[kratom_sandwich]

Have you tried looking into academic literature? There's lots of papers on text analysis, including several surveys. Start with [1] for example, which creates a word list suited for applications in finance, and then work with forward citations.

[1] DOI: 10.1111/j.1540-6261.2010.01625.x - "When Is a Liability NOT a Liability? Textual Analysis, Dictionaries, and 10-Ks" by Loughran and Mcdonald

[anigbrowl]

Why don't you just give it $1000 and let it trade for you, if the effect is strong maybe you won't need to care about your resume.

[alphameta]

I will surely give it $1000 but first i have to figure out on how to predict the stock. thanks

[sigmaprimus]

Possibly weighting sentiment against frequency of tweets?

I would say that You also need to include swings in the general marketplace. I would do this by observing comparables or simmilar stocks and analyze them using your system, if they all line up or one stands out that might be the key.

The more data points You can use the better, especially if you are using ML to create the model.

Robinhood might be a good test case, they are planning an IPO so You can watch it right from the beginning. It would be ironic if all the Gamestop investors took their profits and hedged Robinhood into the ground! If that happens I bet preceding Twitter tweets will be a harbinger of sorts.

Just don't get caught up in a "Red/Black" gambling system and lose your shirt!

Maybe looking 2 days into the future is a bit too ambitious of a target right now, a couple hours or even less time would be a huge advantage and should make it faster to verify.

Good luck!

[zora_goron]

I don't have any experience in this domain, but perhaps one thing to try is formulating this challenge as taking in Tweet sentiment and predicting the delta in stock price. For example, using the stock data, generate the % change in stock price every day compared to the prior day. Then, using Tweets from a date _n_ (or range of dates), try to predict the % change of the stock price on date _n+1_.

[almostkorean]

Sounds like a perfect fit for https://signals.numer.ai. If you have any questions there's an active chat community at https://community.numer.ai

[giantg2]

Yes, it can be done. It likely won't produce the results you hope for because other people have already done it (the game has already changed to account for this product). It could still be a good resume booster.

[visox]

Maybe something interesting to you https://insightrend.com/?term=gme

[yvino]

can you make it online lets validate it.

[alphameta]

I still have no idea how to use this to predict one day ahead, I have tried ARIMA and LSTM both maps the chart perfectly but other than that i have no idea, any referneces to study ?

[xkeysc0re]

Get historical price data as well as historical sentiments / tweets from the past N years or so that reference the stock or company. Create sentiment + pricing pairs (so find a way to sum, average, or otherwise normalize sentiment - you might also want to make the price a tuple or vector) and then train your model on that: sentiment score and current price as input, new price as output. You'll probably have to write some additional tooling to munge and prepare this data and then feed it into the model pipeline