|
|
|
|
|
|
by JimmyRuska
1067 days ago
|
|
|
This is great, love it! Crawling sites to index the FAQ's and knowledge bases, into the vector search, isn't as intimidating as it sounds, at least on linux systems. Sometimes a thin wrapper function over plain old wget will get you 99% of the way wget -rnH -t 1 --waitretry=0 'https://{{domain}}' -P '{{domain}}'
|
|
|