[DHaldane]

My gut agrees with you that LLMs shouldn't do this well on a specialty domain.

But I think there's also the bitter lesson to be learned here: many times people say LLMs won't do well on a task, they are often surprised either immediately or a few months later.

Overall not sure what to expect, but fine tuning experiments would be interesting regardless.

[cjk2]

I doubt it'd work any better. Most of EE time I have spent is swearing at stuff that looked like it'd work on paper but didn't due to various nuances.

I have my own library of nuances but how would you even fine tune anything to understand the black box abstraction of an IC to work out if a nuance applies or not between it and a load or what a transmission line or edge would look like between the IC and the load?

This is where understanding trumps generative AI instantly.

[DHaldane]

I doubt it too, but I notice that I keep underestimating the models.

Do you have a challenge task I can try? What's the easiest thing I could get an LLM to do for circuit board design that would surprise you?

[cjk2]

Make two separate signals arrive at exactly the same time on two 50 ohm transmission lines that start and end next to each other and go around a right hand bend. At 3.8GHz.

Edit: no VSWR constraint. Can add that later :)

Edit 2: oh or design a board for a simple 100Mohm input instrumentation amplifier which knows what a guard ring is and how badly the solder mask will screw it up :)

[DHaldane]

Right - LLMs would be a bit silly for these cases. Both overkill and underkill. Current approach for length matching is throw it off to a domain specific solver. Example test-circuit: https://x.com/DuncanHaldane/status/1803210498009342191

How exact is exactly the same time? Current solver matches to under 10fs, and I think at that level you'd have to fab it to see how close you get with fiber weave skew and all that.

Do you have a test case for a schematic design task?

[cjk2]

Yeah. But you need $200k worth of Keysight kit to test it.

The point is there’s a methodology to solve these problems already. Is this better? And can it use and apply it?

[bmicraft]

It would seem to me that the majority of boards would be a lot more forgiving. Are you saying you wouldn't be impressed if it could do only say 70% of board designs completely?

[AdamH12113]

Not the GP, but as an EE I can tell you that the majority of boards are not forgiving. One bad connection or one wrong component often means the circuit just doesn't work. One bad footprint often means the board is worthless.

On top of that, making an AI that can regurgitate simple textbook circuits and connect them together in reasonable ways is only the first step towards a much more difficult goal. More subtle problems in electronics design are all about context-dependent interactions between systems.

[nurple]

I hate that this is true. I think ML itself could be applied to the problem to help you catch mistakes in realtime, like language servers in software eng.

I have experience building boards in Altium and found it rather enjoyable; my own knowledge was often a constraint as I started out, but once I got proficient it just seemed to flow out onto the canvas.

There are some design considerations that would be awesome to farm out to genai, but I think we are far from that. Like stable-diffusion is to images, the source data for text-to-PCB would need to be well-labeled in addition to being correllated with the physical PCB features themselves.

The part where I think we lose a lot of data in pursuit of something like this, is all of the research and integration work that went on behind everything that eventually got put into the schematic and then laid out on a board. I think it would be really difficult to "diffuse" a finished PCB from an RFQ-level description.

[cjk2]

No because it’s hard enough picking up an experienced human’s designs and work with them. A 70% done board is a headache to unwrap. I’d start again.

[nurple]

This is how I am with software. There's usually a reason I'm arriving at 70% done, and it's not often because it's well designed and documented...

[LeifCarrotson]

Really? Most of the time?

I find I spend an enormous amount of time on boring stuff like connecting VCC and ground with appropriate decoupling caps, tying output pins from one IC to the input pins on the other, creating library parts from data sheets, etc.

There's a handful of interesting problems in any good project where the abstraction breaks down and you have to prove your worth. But a ton of time gets spent on the equivalent of boilerplate code.

If I could tell an AI to generate a 100x100 prototype with such-and-such a microcontroller, this sensor and that sensor with those off-board connectors, with USB power, a regulator, a tag-connect header, a couple debug LEDs, and break out unused IO to a header...that would have huge value to my workflow, even if it gave up on anything analog or high-speed. Presumably you'd just take the first pass schematic/board file from the AI and begin work on anything with nuance.

If generative AI can do equivalent work for PCBs as it can do for text programming languages, people wouldn't use it for transmission line design. They'd use it for the equivalent of parsing some JSON or making a new class with some imports, fields, and method templates.

[scld]

"Looks like you forgot pullups on your i2c lines" would be worth a big monthly subscription hahaha.

[oscillonoscope]

There are schematic analysis tools which do that now just based on the netlist

[makapuf]

This totally didnt happen to me again recently. But next time I surely won't forget those. (Cue to a few months from now...)

[DHaldane]

I've found that for speeding up design generation like that, most of the utility comes from the coding approach.

AI can't do it itself (yet), and having it call the higher level functions doesn't save that much time...

[anonymoushn]

We have 0 y/o/y progress on Advent of Code, for example. Maybe we'll have some progress 6 months from now :) https://www.themotte.org/post/797/chatgpt-vs-advent-of-code

[DHaldane]

Have you tried using more 4000x more samples?

https://redwoodresearch.substack.com/p/getting-50-sota-on-ar...

[HanClinto]

> But I think there's also the bitter lesson to be learned here: many times people say LLMs won't do well on a task, they are often surprised either immediately or a few months later.

Heh. This is very true. I think perhaps the thing I'm most amazed by is that simple next-token prediction seems to work unreasonably well for a great many tasks.

I just don't know how well that will scale into more complex tasks. With simple next-token prediction there is little mechanism for the model to iterate or to revise or refine as it goes.

There have been some experiments with things like speculative generation (where multiple branches are evaluated in parallel) to give a bit of a lookahead effect and help avoid the LLM locking itself into dead-ends, but they don't seem super popular overall -- people just prefer to increase the power and accuracy of the base model and keep chugging forward.

I can't help feeling like a fundamental shift something more akin to a diffusion-based approach would be helpful for such things. I just want some sort of mechanism where the model can "think" longer about harder problems. If you present a simple chess board to an LLM or a complex board to an LLM and ask it to generate the next move, it always responds in the same amount of time. That alone should tell us that LLMs are not intelligent, and they are not "thinking", and they will be insufficient for this going forward.

I believe Yann LeCun is right -- simply scaling LLMs is not going to get us to AGI. We need a fundamental structural shift to something new, but until we stop seeing such insane advancements in the quality of generation with LLMs (looking at you, Claude!!), I don't think we will move beyond. We have to get bored with LLMs first.

[pton_xd]

> If you present a simple chess board to an LLM or a complex board to an LLM and ask it to generate the next move, it always responds in the same amount of time.

Is that true, especially if you ask it to think step-by-step?

I would think the model has certain associations for simple/common board states and different ones for complex/uncommon states, and when you ask it to think step-by-step it will explain the associations with a particular state. That "chattiness" may lead it to using more computation for complex boards.

[HanClinto]

> > If you present a simple chess board to an LLM or a complex board to an LLM and ask it to generate the next move, it always responds in the same amount of time.

> Is that true, especially if you ask it to think step-by-step?

That's fair -- there's a lot of room to grow in this area.

If the LLM has been trained to operate with running internal-monologue, then I believe they will operate better. I think this definitely needs to be explored more -- from what little I understand of this research, the results are sporadically promising, but getting something like ReAct (or other, similar structures) to work consistently is something I don't think I've seen yet.

[visarga]

> I just want some sort of mechanism where the model can "think" longer about harder problems.

There is such a mechanism - multiple rounds of prompting. You can implement diverse patterns (chains, networks) of prompts.

[sweezyjeezy]

Some research to the contrary [1] - tldr is that they didn't find evidence that generative models really do zero shot well at all yet, if you show it something it literally hasn't seen before, it isn't "generally intelligent" enough to do it well. This isn't an issue for a lot of use-cases, but does seem to add some weight to the "giga-scale memorization" hypothesis.

[1] https://arxiv.org/html/2404.04125v2