[zapperdulchen]

I played along the example of the chancellor question.

If you deviate a little from the examples in the article GPT-4 gets it all wrong:

Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel)

https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f...

Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Helmut Kohl (wrong, Angela Merkel)

https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f...

Interestingly whether you put is or was into the question does make a difference.

Even when allowed to surf the web, ChatGPT gets it wrong:

Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder (wrong, Angela Merkel)

https://chat.openai.com/share/867cb3bc-f642-4d2a-b80d-fe8dc0...

Though it's referring to the right Wikipedia page: https://en.m.wikipedia.org/wiki/List_of_chancellors_of_Germa...

[dragonwriter]

> Even when allowed to surf the web, ChatGPT gets it wrong:

> Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder

Its probably counting Walter Scheel, who was Acting Chancellor in 1974, and you are probably not.

When I asked ChatGPT to list the chancellors in order and identify the eighth, it listed eight ending in Schröder, with Scheel and his ten day Acting Chancellor tenure in May 74 as number 5. (Scheel’s tenure is in the timeline on the Wikipedia page you cite, though not in the numbered list in that page, which is why there is a dicontinuity in thr dates on the numbered list.)

[littlestymaar]

It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong…

[dragonwriter]

> It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong

Schröder is the only answer it gave when using functionality that would bring some representation if a list into its context first, and 10it did it with different prompts and mechanisms for bringing a list into its context.

That LLMs are bad at counting-related tasks without doing that is well-known, and not a point I felt needed belaboring.

[RandomLensman]

Walter Scheel was never Chancellor, he was fulfilling the duties of the Chancellor while being Foreign Minister, but he was not a Chancellor. So counting him is wrong.

[Kiro]

Imagine reading this comment thread a few years back. Just getting an answer to this question at all from a generic bot would be considered science-fiction and now we're dismissing it based on technicalities.

[RandomLensman]

Not a technicality. It's like inventing a proper head of government that wasn't there.

Also, knowing German chancellors was always easy, so that fact that a machine falls behind any normal dictionary or the German Chancellor's website is poor form.

It's like a calculator getting basic addition wrong. I don't want a future full of poorly performing machines.

[Kiro]

He was acting Chancellor. It's like saying a red apple is not an apple. It's a technicality. The fact that we have a completely reasonable explanation to why GPT-4 includes him in the list and you guys still make it sound like the AI is a complete moron is just hilarious to me.

[dllthomas]

A "red apple" is an apple that is red. An "acting officeholder" is not an officeholder who is acting, but a person who is acting as if they have the office (... with institutional support - Norton was not acting Emperor).

It's a technicality, but a valid technically and it's a technical question. I agree that it's wrong in a relatively small and surprisingly human way, but it's wrong in a way that including a red apple amongst apples is not.

[RandomLensman]

There is no "Acting Chancellor", that is a poor translation of what the job is. There is only a Chancellor, not an "Acting Chancellor" - you have to trust that the Germans and their institutions kind of know who was Chancellor and who was not.

The machine is a complete moron for not being able to get basic data from basic sources right.

[OJFord]

> Walter Scheel was never Chancellor, he was fulfilling the duties of the Chancellor while

I think this is really an issue of semantics & translation, because that is just what 'acting X' means? (Admittedly not always while still doing something else too, but I don't see that as significant - if it had gone on long perhaps a junior minister in the foreign office would've been named acting FM in his place too.)

[RandomLensman]

That is not semantics. The law is very clear: Scheel was not a Chancellor (and indeed he is not counted as one in Germany: https://www.bundeskanzler.de/bk-en/chancellery/federal-chanc...).

Article 69 Grundgesetz has a sort of "Caretaker Chancellor" that has the function but - importantly - not the office. They way to have the office is through article 63.

It works differently in Germany to the US, for example, where the vice president could become the president, while the vice chancellor only ever gets the function, not the office (unless through a proper vote for Chancellor).

When there is actually ground truth, machines should be able to recover that, not take weird turns.

[OJFord]

What distinction are you making between 'acting X' and 'caretaker X'?

I'm British, not American, so I'm not assuming something like the vice president automatically becomes president, we don't have that either. If anything it's even less than Germany since (at least in theory and history, modern media etc. makes it a bit different in practice) there's nothing special about the PM, it's just the governing party's leader. I suppose though you could say the heir apparent to the throne immediately becoming the monarch on the death of the previous one is like vice taking over - but I don't think that detracts from my point because nobody would call that 'acting monarch', they just are.

'Acting X' means doing the necessary duties, but not any major decisions that can be avoided/deferred, while the 'real' replacement is found. Sometimes it ends up being the same person, e.g. the head of some division is 'acting CEO' for a while as the board searches for a new CEO, ultimately ends up going with that person and title changes to just 'CEO' - or they don't, and go back to old job, or maybe quit in a huff, and the person they found is just 'CEO' taking over from the 'acting'.

[RandomLensman]

The real issue here is that people stick to the word "Chancellor", not the "acting" as such.

There is no named role of "Acting Chancellor" created in the basic law, instead someone is just tasked with performing certain duties (and people might sometimes call it "acting Chancellor" or whatever). However, the position "Chancellor" is a well defined & special role (unlike perhaps PM) and whatever that other "acting" role is, it isn't a Chancellor. It is a bit like adding Oliver Cromwell to list of English Kings or Kamala Harris to the list of US Presidents - you can do it but you then move beyond the "standard" definitions.

[dragonwriter]

It is exactly what “acting X” means in English.

[RandomLensman]

You can clearly see it is confusing the heck out of people here by seeing "acting X" as a version of X when it is not in this case here.

[dllthomas]

Yeah, but those people are just wrong. To be fair, it's not something most people have much direct experience with.

[postexitus]

If you think ChatGPT is "counting" anything, you have massively missed the point.

[getpost]

I asked for a list of Federal Chancellors of the Federal Republic of Germany and GPT4 included Scheel ("acting"), which makes Schröder the 8th. I often troubleshoot GPT4 replies by asking it for more context. "Is" vs "was" is interesting an find!

https://chat.openai.com/share/ddd2800e-630c-4ff7-9992-14b265...

[zurfer]

as others pointed out, it's important how you ask and what you think the truth is. https://chat.openai.com/share/cc688592-236a-499b-82e1-81bdaa...

[carlmr]

>Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel)

Technically there's no right answer to this question. "Is" implies present. But Angela Merkel isn't the present chancellor. Olaf Scholz is. But he's not the eigth.

[pwdisswordfishc]

Technically, Angela Merkel never stopped being the person who served as the eighth chancellor.

[amelius]

Language is imprecise. LLMs should be able to deal with that.

[moelf]

your first two links are duplicates