[jcr]

Do you think your camera system could (eventually) support some combination of Computer Vision and Machine Learning to identify (some) assembly mistakes in real-time and provide feedback/warnings to the assembler?

My thought is improving yields by reducing defect rates caused by those hopelessly imperfect humans (like me) might be a great selling point for some types of manufacturing.

[kg4lod]

That's a really neat idea!

You are describing a type of machine called an AOI (automated optical inspection). It is a common piece of equipment in electronics assembly lines. But AOI equipment is very expensive (~$250k+) and only works on a specific set of products. For example, it doesn't work well when the product contains soft or flexible surfaces, or has a highly non-planar shape.

GetScale terminals can be interfaced with AOI machines in the factory to collect and store the findings along with our human-led inspections and catalog everything by serial number and we do have ambitions to add AOI-like capabilities to our hardware down the line.

Sounds like you have a project in mind. What are you working on? ;-)

[jcr]

I was trying to avoid running off into the endless weeds of details and acronyms and talking, but oh well... ;)

I'm not very familiar with the more advance microscopy based AOI systems used on silicon and chip level manufacturing (e.g. wire bonding checks). I'm also not familiar with advanced AOI used in robotic assembly. On the bright side, I am familiar with the comparatively "simple" form of AOI used on PCB assembly lines.

With PCB's most of the AOI identification and spatial orientation issues are solved through Fiducial Marks [1] and pattern recognition. Some PCB AOI systems even read/record chip/board lettering with OCR, as well as handling Bar and QR codes. Most of the rules and routines controlling the AOI are consistently derived directly from the PCB design files, so there is little need for custom setup programming on each new PCB run.

That last point is critical; The rules and routines are automatically and consistently _derived_ from the authoritative data sources. Manually writing every tiny step in the AOI automation routines for each PCB design would never be time/cost effective, so it has never been done.

Given sufficient example 2D images, 3D scan models, or CAD files, doing the object recognition and spatial orientation with OpenCV is reasonably straight forward, even without fiducials. The extremely tricky part will be automatically deriving rules and routines for the GetScale AOI-like capabilities from authoritative data sources. If the human assembly task is to attach the top cover and secure it with four screws from the other side, you should _never_ need to manually write the AOI routine for checking the existence of a screw in each of the four corners when viewing the bottom of the product.

I'm sure you're really busy, so I think I'll stop here. ;)

If you want to know about the projects I'm working on, email would be a lot better for me. My address is on my HN profile page. Also, I spotted a few places where your website could use a little TLC, and it's better to point them out through email rather than publicly.

[1] https://en.wikipedia.org/wiki/Fiducial_marker#PCB

[kveykva]

We've discussed this possibility of this and I would really like to make an attempt when we have more data!

The practical application is this is sort of an 80/20 proportion though. Currently we see fairly significant improvements in some situations just by the system being there. Factories will pre-test units before passing them over to our system in order to demonstrate better quality, when prior to installation they weren't testing a majority of units at all.

It'd still be really cool to try, haha.

[jcr]

There are metric-truck-loads of existing research on behavior changes due to known surveillance. Though they might be pre-testing units off-camera for the sake of giving a good "demonstrating quality" stage show while being recorded by your system, there's also the effect of better behavior when knowingly being monitored. Either way, you win. ;)

The problem with pre-test is it can both decrease defect rate _and_ decrease throughput rate. Depending on how much each decrease is, you can actually decrease yield/time rather than increase it. This exact problem exists on PCB manufacturing. It's certainly possible to pre-test every component as it comes off the reel and before it's used on the PCB. The component pre-test does decrease defect rates of the finished boards, but it also decreases throughput rate, so it can negatively impact yield/time. The decision then becomes an investment/accounting problem. Often, post-mfg circuit/component test automation and reworking failed boards makes more sense financially.

As far as I know, no one does hard or soft real-time verification of the human side of manufacturing and lab experimentation, but _my_ knowledge of the current state of the art is admittedly outdated. Using CV and ML to spot human manufacturing errors (or inefficiencies) seems feasible if you throw enough compute power at it, but whether or not it's practical and financially viable is another (more important) question altogether.

Though CV+ML may not be reasonable in the real world, I agree with you; it sure does sound like fun to try.