I have my own benchmark: a photo of filament boxes in my drawer. The task is to list (inventory) the filaments. It's not an easy task: the letters are small, the boxes are in various orientations, the labels have lots of writing on them, etc.
This is what qwen produces (basically perfect):
Fiberlogy (Blue Boxes - Left Side)
ASA Black: 750 g (0.75 kg)
Easy PLA Graphite: 850 g (0.85 kg)
Ref Easy PLA Gray: 850 g (0.85 kg)
Ref Easy PLA White: 850 g (0.85 kg)
Ref Easy PLA Inox: 850 g (0.85 kg)
Bambu Lab (Brown Boxes - Right Side)
ASA White: 750 g (0.75 kg)
PLA-CF (Burgundy Red): 1,000 g (1 kg)
PAHT-CF (Black): 500 g (0.5 kg)
PET-CF (Black): 500 g (0.5 kg)
PETG-CF (Black): 1,000 g (1 kg)
----------
And this is gemma-4-26b-a4b (basically useless):
### *Fiberlogy Filaments*
All Fiberlogy items listed below share these common specifications: Made in Poland, $1.75\text{mm}$ Diameter, $0.75\text{kg}$ Weight, and Print Temperature of $200-230^{\circ}\text{C}$.
* *PLA Black* (Barcode: 5903281997986)
* *Easy PLA Graphite* (Barcode: 5903281994057)
* *Ref Easy PLA Grey*
* *Ref Easy PLA White* (Barcode: 5903281994386)
* *Ref Easy PLA True...* (Text is cut off)
**
### *Bambu Lab Filaments*
All Bambu Lab items listed below share these common specifications: Made in China, Removable Spool (Do Not Remove).
* *PLA-CF (Carbon Fiber Reinforced)*
\* Color: Burgundy Red
\* Diameter: $1.75 \pm 0.02\text{mm}$
\* Weight: $1.0\text{kg}$
\* Suggested Drying Conditions: $45^{\circ}\text{C}$ for $6-12$ hours
* *PAHT-CF (High Temperature Polyamide with Carbon Fiber)*
\* Color: Black
\* Diameter: $1.75 \pm 0.02\text{mm}$
\* Weight: $0.5\text{kg}$
\* Suggested Drying Conditions: $80^{\circ}\text{C}$ for $6-12$ hours
Thanks. Did you set the image min/max tokens for Gemma4 to 1120 for this? This might not be a fair comparison without that, to the differences in architecture.
I did not, and I had no idea such a setting even existed. This could definitely change things. However, I don't see a way to set this in LM Studio, which is what I currently use to run models.
This is what qwen produces (basically perfect):
Fiberlogy (Blue Boxes - Left Side)
ASA Black: 750 g (0.75 kg)
Easy PLA Graphite: 850 g (0.85 kg)
Ref Easy PLA Gray: 850 g (0.85 kg)
Ref Easy PLA White: 850 g (0.85 kg)
Ref Easy PLA Inox: 850 g (0.85 kg)
Bambu Lab (Brown Boxes - Right Side)
ASA White: 750 g (0.75 kg)
PLA-CF (Burgundy Red): 1,000 g (1 kg)
PAHT-CF (Black): 500 g (0.5 kg)
PET-CF (Black): 500 g (0.5 kg)
PETG-CF (Black): 1,000 g (1 kg)
----------
And this is gemma-4-26b-a4b (basically useless):
### *Fiberlogy Filaments*
All Fiberlogy items listed below share these common specifications: Made in Poland, $1.75\text{mm}$ Diameter, $0.75\text{kg}$ Weight, and Print Temperature of $200-230^{\circ}\text{C}$.
* *PLA Black* (Barcode: 5903281997986)
* *Easy PLA Graphite* (Barcode: 5903281994057)
* *Ref Easy PLA Grey*
* *Ref Easy PLA White* (Barcode: 5903281994386)
* *Ref Easy PLA True...* (Text is cut off)
**
### *Bambu Lab Filaments*
All Bambu Lab items listed below share these common specifications: Made in China, Removable Spool (Do Not Remove).
* *PLA-CF (Carbon Fiber Reinforced)*
* *PAHT-CF (High Temperature Polyamide with Carbon Fiber)* * *PETG-CF (Carbon Fiber Reinforced)*