[fotcorn]

I can only access https://makersuite.google.com/ when using a VPN to the US. Also, it spams popups that get blocked by Firefox.

Some basic prompts, which are answered correctly most of the time by ChatGPT4:

There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?

> 29 books

Julia has three brothers, each of them has two sisters. How many sisters does Julia have?

> Three

If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?

> Under the plate in the kitchen.

So, not great.

[tr3ntg]

Since the Maker Suite doesn't have the same prompting as something like Bard or ChatGPT, I gave Gemini Pro a bit more context:

> Consider the following user chat. Reason out loud as needed to provide a reasonable response. User message: "Julia has three brothers, each of them has two sisters. How many sisters does Julia have?"

Gemini Pro:

> Julia's brothers have nothing to do with how many sisters she has. Julia's sisters are also her brothers' sisters, so the number of sisters Julia has is the same as the number of sisters her brothers have. Julia's brothers have two sisters each, so if we add up the number of sisters for each brother, we get: 2 + 2 + 2 = 6. Therefore, Julia has 6 sisters. The answer is 6.

[staticman2]

Mixtral 8x7b only gets the sister question correct when I ask it to show it's work, but also shows that it's aware of the brain teaser:

https://poe.com/s/5gtLvuWT52o9LEBAJouo

GPT 4 gets it right.

GPT 3.5 gets it hilariously wrong if I change the question slightly:

User: Julia has 8 brothers, each of them has 5 sisters. How many sisters does Julia have?

Assistant:

Julia has 8 brothers, and each of them has 5 sisters. Since Julia is one of those sisters, she has 8 brothers + 1 (herself) = 9 sisters.

https://chat.openai.com/share/3a49800f-7747-44d9-90e7-c1e037...

Also on 3.5 I tried this:

User: Julia has 8 brothers, each of them has 5 sisters. How many sisters does Julia have? Show your work step by step.

Assistant: Julia has 8 brothers, and each of them has 5 sisters, including Julia herself. So, Julia is one of those 5 sisters for each of her 8 brothers.

To calculate how many sisters Julia has, you can simply multiply the number of brothers by the number of sisters each brother has:

8 brothers * 5 sisters per brother = 40 sisters

Therefore, Julia has 40 sisters in total.

[angm128]

Mistral 7B (mistral-7b-instruct-v0.2.Q6_K.gguf): Let's break down this problem step by step. First, we know that Julia has three brothers. Then it is stated that each of her brothers has two sisters. This means there are two sisters for each brother. Since Julia is one of the sisters for her brothers, she also counts as one of their sisters. However, when counting the number of sisters for herself, she should not be included in the count. Therefore, Julia has three brothers and two sisters (excluding herself), so she has a total of five sisters.

[hatsunearu]

Isn't 6 still incorrect? Or am I dumb?

[tr3ntg]

This made me laugh. It's very wrong. Unless I'm dumb, the answer we're looking for should be 1.

Sounds like the confidence of Gemini worked on you, though.

[gduplessy]

I tried in Bard just for comparison's sake and it got to 1 while still saying a different number. It's a bit strange, here:

Julia has two sisters!

Here's the reasoning:

1. *Julia's brothers have sisters*: Each of Julia's three brothers has two sisters. This means in total, her brothers have 3 brothers * 2 sisters/brother = 6 sisters.

2. *Those sisters are not additional*: However, it's important to remember that these sisters belong to Julia's brothers, not directly to Julia herself. Julia and her sisters are siblings, not counted among the sisters her brothers have.

3. *Therefore*: Julia has 2 sisters (herself and one other).

So, while Julia's brothers have a total of 6 sisters, only Julia and one other sister are Julia's own sisters.

[pomtato]

.

[hluska]

It’s way more complicated. As an example, my kid has one half sister and had one step sister. My kid referred to both of them as their sisters, but the step person didn’t acknowledge my kid as such.

Even if we err on the side of the step, take the half sister entirely out of the equation and only focus on the step relationship, one would have claimed 0 sisters and the other would have claimed 1. They were talking about each other and they were both telling their truth. But the answer is simultaneously 0 and 1.

It gets even more interesting because neither my kid nor their step sibling are cisgender. I don’t know what to call their step so just use their name and pronouns now. The step still won’t acknowledge they’re siblings and I respect that a lot.

When these smart young folks take over, the world will get really cool.

Edit - This was way more complicated to explain than I anticipated.

[vasco]

The sum is incorrect and the statement of "Julia's sisters are also her brothers' sisters, so the number of sisters Julia has is the same as the number of sisters her brothers have" is also incorrect. The brothers each have one more sister than Julia, since she is one of the sisters so she only has one sister. It's just reworded but still wrong.

[skywhopper]

You're not dumb. But it's an ambiguous question as written. Presumably, we are to assume that all siblings involved are still alive, that all all full siblings (ie no half- or step-siblings), that everyone is cisgender, and that Julia is female. In which case, Julia would have one sister. Each of her three brothers would have two sisters, one of whom was Julia herself.

[shwouchk]

My 6yo can answer this correctly, making all of your stated assumptions above. Edit: Clarification - I am in no way implying that someone who was confused by the math is dumb, everyone makes silly mistakes sometimes.

I think It’s only ambiguous to a machine. To a human, even in 2023, these are likely default assumptions for a question phrased as above, unless being overly pedantic or attempting to “get around” the puzzle.

If you ask a model - or a person - “what does 1+1 equal to?”, it is also ambiguous, since we haven’t defined what “+” means, did not specify that we are dealing with natural numbers rather than eg elements in Z//2Z which could be similarly denoted.

[shawabawa3]

Yes, making it reason made it even more wrong

[ajhai]

Gemini Pro compares to GPT 3.5.. their biggest model that competes with GPT-4 is called Gemini Ultra and they say that's coming early next year.

[isoprophlex]

Asking these to GPT3.5 has been an utterly frustrating experience, lol. I guess gemini is at this level of intelligence right now, not GPT4... rigged demos notwithstanding;)

[FergusArgyll]

I asked this to mini-orca 3b and here was it's brilliant answer.

> If you read 2 books over the weekend, then there are 31 books in your house. However, if you only read one book, then there would be only 25 books left in your house.

[BlindEyeHalo]

No joke, even if trying to correct GTP3.5 it still gives a nonsense answer:

> I apologize for any confusion caused by my previous response. Reading a book doesn't physically remove it from your house. The assumption I made was a misunderstanding. If you read a book, it is still in your house unless you lend it, give it away, or otherwise remove it.

> So, if you started with 31 books and read 2 over the weekend, you would have 31 - 2 = 29 books still in your house.

[la64710]

To the second question chatGPT answered : The orange is in the kitchen, as that is where the plate, along with the orange, was moved.

Now a human can reason because a human can visualize in their mind that a plate is being moved without moving the orange. How is that kind of reasoning possible in LLMs which in my very basic understanding is predicting the next token.

[tagraves]

Here's the response chatGPT 4 gave for me:

> The orange remains in the living room. Moving the plate to the kitchen does not affect the location of the orange, since it was placed below the plate but not attached to it. Therefore, the orange stays where it was originally placed, which is in the living room.

You don't need to visualize it in your mind to understand the relationship between being _below_ and being _moved with_. Keep in mind that many people cannot visualize anything in their mind!

[la64710]

I have wondered how do people blind from birth create their mental world? Is it all dark with no color or light and only sound? No shapes? Or do they still form mental images from non visual sensory inputs?

[luma]

Make sure you're including the GPT version, there's a wide gulf between GPT4 and prior models. GPT-4 answers:

Q. If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?

A. The orange remains in the living room. Moving the plate to the kitchen does not affect the position of the orange.

----

Q. There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?

A. There are still 31 books in your house. Reading books does not change their quantity

----

Q. Julia has three brothers, each of them has two sisters. How many sisters does Julia have?

A. Julia has one sister. Each of her three brothers has two sisters, and Julia is one of them. The other one is her sister, making it a total of one sister for Julia.

[lern_too_spel]

In order to predict next tokens in text with greater and greater accuracy, some parameters of the language model must represent a model of the world.

[amf12]

FWIW, Gemini Pro is equivalent to GPT 3.5, so expected

[EZ-E]

GPT4 gets the second question wrong for me, but in a weird way.

> Julia has three brothers, each of them has two sisters. How many sisters does Julia have?

Julia has two sisters. Each brother having two sisters refers to the same two sisters, Julia and another sister.

Then if I ask to verify its answer, it realizes the issue.

> Apologies for any confusion. If Julia has three brothers and each of them has two sisters, then Julia is one of the sisters. The other sister is common among all three brothers. Therefore, Julia has only one sister.

[ryoshu]

Mistral 7b gets the first two wrong, but gets the third one right: "The orange is still in the living room. When you moved the plate, you didn't move the orange. It remains in its original location until you pick it up and move it to a different place"

[strobe]

mistralai/Mixtral-8x7B-Instruct-v0.1 responses - which seems much better:

>>There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?

You still have 31 books in your house. The number of books doesn't change because you finished reading two of them; they're just no longer unread.

>>Julia has three brothers, each of them has two sisters. How many sisters does Julia have?

Julia has one sister.

Here's the logic behind this:

1. We know that Julia has three brothers. 2. Then we are told that each of her brothers has two sisters. 3. Since all four children (Julia and her brothers) share the same parents, they also share the same siblings. 4. Therefore, if each of Julia's brothers has two sisters, then there must be only one other sister besides Julia herself.

>> If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?

The orange is still in the living room, as moving the plate does not affect the location of the orange.

[robertlagrant]

Wow - that is good.

[dhoe]

Great questions. Gpt4 got the second wrong in my test, but corrected itself when I told it to review for errors. It got the other two right at first try.

[vbezhenar]

GPT4 insisted on wrong answer in the second test: https://chat.openai.com/share/301cc597-929b-43d6-9685-2aad3e...

[kuratkull]

My attempt had GPT4 fail on Julia, it was adamant that Julia has two sisters. It took couple of back and forths for it to understand.

[bool3max]

Why would you expect an LLM to answer thoe questions correctly?