Hacker News new | ask | show | jobs
by actusual 31 days ago
150,000 rows of data, where will I store it all?!
3 comments

The biggest misconception that people have when modeling using tabular data is that more data = better model.
There's a story sharing the front page right now about scaling laws, so it's not an unreasonable assumption.

https://news.ycombinator.com/item?id=48689744

You don't need to. Just sample 1% of data, let the model do the feature engineering, ask model model to replicate everything in Pandas (or R, or whatever else), so you can run it for a full dataset.
Big Data is terrifying and only solvable with opaque deep learning