Hacker News new | ask | show | jobs
by hn_20591249 1113 days ago
If you are using Tensorflow, Spark and Keras in your hiring decisions I’d say you are already behind the curve on technologies.
2 comments

Sure. But I'm a software engineer who is finishing a Ph.D. in applied informatics (coincidently in an area of time series prediction using ML). I'm not a manager.

When I mentioned "AI Winter" in front of them they didn't know what I was speaking about. But they created a nice corporate ladder which anybody can climb and it was based on years of experience with the aforementioned frameworks. Python experience needed, Scala + Spark was an advantage.

I don't know what are you planning to do. But ... I'm buying a huge load of popcorn and I will laugh my ass off when this bubble bursts in a couple of years.

Never underestimate the ability of corporations to make a multi-decade business venture out of selling AI-bullshit to other corporations.
what are the current relevant technologies?
(not parent but -) It depends on area, generally: Provider APIs(ChatGPT through OpenAPI etc), langchain, huggingface transformers, pinecone/vector DBs are absolutely taking off.

Lots of specialist ML models which required specific data collected carefully for a business task are no longer needed. Most ML roles until now, and research time, was spent collecting data and training specialist models.

General models like ChatGPT or pretrained image models do better than a fresh model trained from scratch or even a finetuned small/medium model (e.g. BERT/T5) ever will these days.

The special ML (pipeline of data --> train --> deploy --> manage/mlops/drift) used tech like PyTorch, Tensorflow, MLFlow.. and for more applicable levels (e.g. deployment), transformers, sci-kit learn, keras. However these are being replaced wholesale at many companies by langchain, huggingface inference API (for vision tasks) and pinecone/other vector DBs.

Langchain is just a smart way to wrap and order API calls to OpenAI/ChatGPT/other providers really, with some prebuilt use cases. Right now there is less on the metrics/output side than with lets say "bring-your-own-data" ML models, which you could measure things like precision.

Now, the old guard of ML (PyTorch, Tensorflow) is still used for training new models, open source replication attempts etc. But newer frameworks like JAX have not really taken off, as they have been entering the community as the community switched to using providers, rather than training their own models.

There remains a subset still powerful for communication with C-suite: Using simple models like K-means to show clusters with readable axis. They tend to use sci-kit learn or R. But this is more classic data science than ML.

There are also areas of AI so far relatively unaffected by ChatGPT etc - time series prediction (like OP, so it's less surprising they are using the old guard technologies), game engine AIs, non-discrete data, recommendation algorithms, some computer vision algorithms (especially Active learning). Some like HuggingFace (a commercial company running transformers Python module) are sort of inbetween given they serve both data-trained and the newer models.