| HN Mirror

What things are new/unique? Hmmm, I'll bet someone could implement almost all of what I do inside excel sheets one way or another, but it would just be super-cumbersome to do.

One big example is regression. All Excel really does is simple linear regression. Add more variables you get a better r2'd and that's it. Comparatively, with scikit it's trivial to avoid all horrid overfitting problems that come with simple regressions in excel. I use of lasso-cv model, and better practices like cross-validation, etc.

Another big improvement is I replaced Excel -> Tableau visualizations pipeline with using Altair-Viz. Altair makes almost-Tableau-like charts pretty easily and cuts out lots of intermediary steps -- jump right from dataframe to a chart.

It's much easier to add sanity/safety assertion checks in python than in Excel. Everyone knows examples of excel spreadsheets that are subtly poisoned.

I took someone else's technical charting ideas and reimplemented them in python, which then just alerts me for key events (changes in trend, potential exciting issues, etc). Now I don't have to look through the data myself, which is a chore, and which is susceptible to my mood. It just spits out its objective 'truth'.

One thing about (at least commodities trading) that outsiders might not realize is -- none of the sexy FAANG big data stuff is particularly useful or appropriate. The data is way too sparse and the quality varies immensely and is normally quite lagged. As a concrete example perhaps you think the price of oil = f(oil demand in china, oil production in saudi arabia). Great. At best you get monthly data, probably lagged, and probably manipulated by the government. So that's 24 pieces of data per year....not great candidate for machine learning ideas. Perhaps very high-frequency, ultra short term automated trading can use tick data for things like this, but not for general prop trading that holds risk for days or weeks.

I also expect outsiders imagine oil price = f(factor_a, factor_b, ...factor_z) and it's just the job to figure out that ka, kb, ... kz are. But markets aren't physical phenomena like a spring where repeated measurements of length and weight can eventually can estimate what the K-factor is. If markets are springs, then their K-factor changes all the time. Factors that sometimes matter for one month don't matter for the next six or twenty four. That's why it's important to monitor the many spaces that can drive a market -- underlying fundamentals, 'flows' from the financial/hedging markets, and myriad non-specialist investors who behave based totally different drivers than oil specialists. As much as oil trading could be trying to accurately count barrels of oil, it's also important to understand the 'meta' of who is buying/selling and why.

To be clear, my trading is not automated. At end of the day I'm making a bet myself, not a machine deciding. It's more like I have machine exoskeleton helping me to see a little bit more clearly and maybe a little less emotionally, which hopefully leads to slightly smarter bets.