Hacker News new | ask | show | jobs
by tossandthrow 804 days ago
why not Ai?

And even if we accept the premise (as flawed as it might be) that Ai is not able to create original knowledge, most of what's online is dessimination and dies not represent new information but just old information rewritten to be understandable by a certain segment.

something LLMs excel at.
1 comments
poyu 804 days ago
> Ai is not able to create original knowledge

The current state of LLMs do hallucinate though. It's just not a very trustworthy source of facts.

link
tossandthrow 804 days ago
just like my first teachers said I should absolutely not use Wikipedia.

LLMs was popularized less than 2 years ago.

I think it is safe to assume that it will be as trustworthy as you see Wikipedia today, and probably even more as you can embed reasoning techniques into the LLMs to correct misunderstandings.

Wikipedia cannot self correct.

link
howenterprisey 804 days ago
Wikipedia absolutely self-corrects, that's the whole point!
link
tossandthrow 804 days ago
it does not. it's authors corrects it.

unless you see Wikipedia as the organisation and not the encyklopedia?

in that case: sigh, then everything self corrects

link
howenterprisey 804 days ago
It is incoherent to discuss Wikipedia as some text divorced from the community and process that made it, so I'm done here.
link
pksebben 804 days ago
There's an important difference between wikipedia and the LLMs that are actually useful today.

Wikipedia is open, like completely open.

GPT is not.

Unless we manage to crack the distributed training / incremental improvement barriers, LLMs are a lot more likely to follow the Google path (that is, start awesome and gradually enshittify as capitalist concerns pollute the decision matrix) than they are the Wikipedia path (gradual improvement as more eyes and minds work to improve them).

link
tossandthrow 804 days ago
this is super interesting!

it also carves I to the question what constituted model openness?

most people agree that just releasing weights are not enough.

but I don't think it will ever be feasible to say that reproducing model training is feasible. especially when factoring in branching and merging of models.

for me this is an open and super interesting question.

link
pksebben 804 days ago
Here's what I envision (note: impossible with current state of the art)

A model that can be incrementally trained (this is the bit we're missing) hosted by a nonprofit, belonging to "we the people" (like wikipedia).

The training process could be done a little like wikipedia talk pages are now - datasets are proposed and discussed out in the open and once generally approved, trained into the model.

Because training currently involves backpropagation, this isn't possible. Hinton was working on a structure called "forward-forward" that would have overcome this (if it worked) before he decided humanity couldn't be trusted [1]. It is my hope that someone smarter than me picks up this thread of research - although in the spirit of personal responsibility I've started picking up my old math books to try and get to a point where I grok the implementation enough to experiment myself (I'm not super confident I'm gonna get there but you can't win if you don't play, right?)

It's hard to tell when (if?) we're ever going to have this - if it does happen, it'll be because a lot of people do a lot of really smart unpaid work (after seeing OpenAI do what it did, I don't have a ton of faith that even non-profit orgs have the will or the structure to pull it off. Please prove me wrong.)

1 - https://arxiv.org/abs/2212.13345

link