Y
Hacker News
new
|
ask
|
show
|
jobs
by
7moritz7
168 days ago
The scale. How many tools do you know that can query the
content
of all arxiv papers.
1 comments
eamag
168 days ago
Doesn't look like the scale is there, even for HN:
> Currently have embedded: posts: 1.4M / 4.6M comments: 15.6M / 38M That's with Voyage-3.5-lite
link
Xyra
167 days ago
The scale is there. I'm scraping, cleaning, token efficientizing dozens of sources every single hour. The lack of monies for embedding everything was a temporary problem.
link
> Currently have embedded: posts: 1.4M / 4.6M comments: 15.6M / 38M That's with Voyage-3.5-lite