[ohazi]

PSA: Networking on microcontrollers is a giant pain in the ass.

Don't get me wrong, boards like this (and ESP/Realtek boards with WiFi) are great for hacky prototypes. But if you're building an embedded device that will be deployed in any sort of volume, a small application processor running Linux will make your life dramatically easier. You can keep it asleep most of the time if your power budget is tight.

The hard part is not cramming in an Ethernet PHY or a 2.4 GHz radio, it's the mountain of software that you need to run on top of it to get the kind of reliable, secure communications channels that we've come to expect. Bare metal networking stacks like LwIP have a reputation for being buggy, and are nowhere near as battle-tested as the Linux/BSD networking stacks security-wise. Some of the more memory constrained devices can barely fit a complete certificate chain. Are you really prepared to roll out updates on a system like this when the next Heartbleed comes along?

Also, the layers tend to pile on, so while you might get away with a bare-bones wire protocol initially, you'll be starting your project close to the limits of what is practical. Somebody will eventually ask you if you can connect to a websocket. I know buildroot and yocto look a little scary at first, but they're better than designing yourself into a corner before you even get off the ground.

[freefriedrice]

That's a matter of opinion.

If you don't know jack about network programming, then of course you can retreat to Linux. But Linux is not an edge / IoT platform by any stretch of the imagination. Micro RTOS is closer, but still overkill for embedded Wi-Fi.

Specifically for Wi-Fi, LWIP is -the- stack to use for embedded TCP/IP. It is under 30K with full TCP/IP and offers everything you need. Then there is COAP or MQTT, which are miniscule compared to HTTP. mbedTLS adds another 80K for full TLS compatibility on 40MHz processors. On top of this every major wifi vendor has an SDK that fits nicely into LWIP (some even ship it), some are small (CC5000) some are larger (like the WF200), but the APIs are pretty straightforward. Now, leaving TCP/IP and 802.11 will add headaches (BLE, ZigBee, Zwave) due to additional learning curves, but only because most people in embedded networking already understand TCP/IP and 802.11 phy.

If you want to play in the embedded space, you gotta learn about it. Running back to linux might work in some cases, but not constrained IoT space.

EDIT: I had my #s reversed, LWIP is under 30K, mbedTLS is around 80K, TLS is most of that (esp. RSA stuff, I could have disabled everything I didn't need [mbed has highly granular config] based on the ciphersuite and shrunk it considerably since ECC is smaller and faster than RSA). Just checked my map file, and I was in -O0 optimization mode.

EDIT2: On second thought: if people want to complain that they need linux to do networking on the edge, please complain! I'll interview after you and demonstrate MQTT with TLS v1.3 running off an MCU with a 2032 coincell for months (eh, I'm weirdflexing a little, but still...)

[ohazi]

I think it's less a matter of opinion and more a question of expectations around the device you're building.

To illustrate, let's say we build the embedded device your way. Six months later, we get a support request from a customer that needs to get the device onto the captive portal at a hotel. The portal is a javascript monstrosity full of CORS failures.

What is your immediate response?

a) Oh, shit, I didn't think of that. I can probably come up with a workaround, but I'll need at least a month, and it'll end up being a huge hack that involves tethering with a phone or laptop (but not via WiFi, since the radio we're using can't do AP and STA modes simultaneously), and the UI will be such a disaster that the customer is immediately going to want to find a better product.

or

b) This is an industrial CNC controller... WTF are you doing on a hotel WiFi network?

If your answer is (a), then your approach is almost certainly the wrong one. If your answer is (b), your approach might be reasonable, but there are still other factors to consider.

It's possible to know everything there is to know about network programming but still not want to debug or re-implement 30 years worth of accumulated cruft around firewalls, MTU sizes, MS-CHAPv2 bugs, malformed packet crashes, anything IPv6, etc. And that's just the table stakes... leaky abstractions mean that sooner or later, everything connected to the internet is going to need to do something higher up the stack that's just barely out of reach for your microcontroller, even one that's beefy enough to run Windows 95. What are you going to do when there's some new web standard where the only implementation is in the Chrome source code as of three months ago?

You'll be able to make it work, but it'll take thousands of person-hours to make it work well. Or you can just "run back to Linux."

Separately, if you're running a company that builds embedded devices, do you want to pay your engineers to fix network stack bugs that were solved in 1999, or do you want to pay them to develop your company's value-add? Weirdflex all you like, it's still (usually) a terrible business decision. I don't think pennypinching on the BOM is a valid justification when things like [1] exist.

[1] https://www.seeedstudio.com/MT7688AN-BIT3-1-Module-p-4011.ht...

[aswanson]

True. Software is soft for a reason. There's always a new requirement or vulnernability waiting to be exploited that needs to be updated.

[pingswept]

That Mediatek module is nice! Thanks for that, and for the wisdom.

[freefriedrice]

You're very good at inventing creative scenarios. Well done.

I don't really know where to start, you clearly want to defend your comfort zone. So I will be supportive: go work on projects that require Linux. It will be much more rewarding to you than working on constrained edge devices that are more challenging due to resource constraints.

You're also free to pretend that you can just swap in platform that requires watts of power in a scenario that requires an edge device to last a year on a battery.

[ohazi]

Look, if you really need to get a coin cell onto the internet, then go for it. But I don't think your comments on this thread represent good advice for most engineers building typical devices. Insinuating that anyone who doesn't agree with your view is technically challenged is also kind of pathetic.

Expectations around what an internet connected device should look like and be capable of changed permanently with the launch of the iPhone in 2007. There will always be special cases like yours, but they are increasingly (sadly?) relegated to smaller corners of the internet.

I did it your way for years. Every company I've ever worked for that started off with a microcontroller based networking solution eventually migrated to an application processor because users and non-technical management kept demanding iPhone-like behavior that was either too difficult or too time consuming to deliver any other way. People expect a lot more functionality than just TCP sockets and web interfaces nowadays.

[nitrogen]

I'll add that it's not just capabilities that matter, but also the open Internet gets more and more hostile over time. In the absence of proof otherwise, eventually someone is going to transmit something that makes your device misbehave.

[nine_k]

There are devices where both BOM and size constraints matter. Do you want to run Linux on every smart power socket, smart light switch, smart light bulb in your house? A light bulb in particular has tight volumetric and thermal budgets.

This is where stuff like ESP8266, and the controller from the post, makes sense.

[ohazi]

I don't own any smart bulbs, but yeah -- this is a good example of a case where using a microcontroller is probably reasonable. I can't imagine people enjoy paying >$25 per bulb, and there's evidence that users are willing to put up with some truly appalling UX to get those things connected [1].

Amusingly, the most reliable smart outlet I own runs Linux on that MediaTek SoC I linked above (~500 MHz MIPS, 32MB RAM). I mean, power is definitely not a constraint when you're an inch away from the prongs, right? ¯\_(ツ)_/¯

[1] https://www.belkin.com/us/support-article?articleNum=116178

[LeifCarrotson]

I completely agree, it depends on your expectations.

With the caveat that it's not a one-way trade-off of "better" vs. "good enough for the expectations". If your customer calls you and says "My industrial CNC controller crashed and ruined a $20k prototype, is this my fault or yours? Here are the logs." and your answer is "the browser was running bad javascript that locked up too many threads for too long so it stopped counting the encoder for 20 ms", they're going to tell you that a servo amplifier has no business doing any of that stuff.

Some devices do need to be on the Internet and should therefore be running general-purpose operating systems. They may need to interact with other devices that run real-time operating systems (or bare metal, like the Teensy). Some IoT devices try to bridge the gap. Knowing and deciding which side you're on is critical.

And I think the best answer to pennypinching on the BOM is to separate your concerns into separate devices/apps. Make the CNC HMI handle the Internet connectivity. Give it one NIC that hooks up to the Wifi, let it run whatever Javascript you want. On a separate NIC, talk whatever flavor of industrial protocol you prefer on the machine. Do not bridge the two. In the servo amp, use a bare-metal/RTOS microcontroller. That device doesn't need to be burdened with Linux, and in fact is made worse by it: Let it be what it is and no more. In 20 years, it should still run reliably, regardless of whatever newfangled holographic VR HMI is loading in motion profiles.

[ohazi]

Exactly, use the right tool for the job.

FPGAs and microcontrollers are really good for meeting (and verifying that you're meeting) hard real-time constraints.

Network connectivity (by definition) can't be real-time, so you wouldn't want to run a network stack on the real-time core anyway, even if you could.

Instead of adding another non-real-time microcontroller for networking and making it sweat, go with an application processor and just treat it like a dumb modem that happens to know how to handle VPN traffic without exploding.

There are application processors like the i.MX7 and the STM32MP1 that combine a Cortex-M4 and a Cortex-A7 on the same die for precisely this reason (and similar FPGA fabric + PowerPC / ARM Cortex-A? products from Xilinx and Altera for even longer).

[Jonnax]

Is that why they say the S in IoT stands for Security?

[cactus2093]

On a similar note, does anyone know of good resources for beginners for prototyping all the way through small-scale manufacturing of an embedded device?

As a web & backend developer who likes to tinker with Arduino/Teensy style projects, I've always been curious what all would be involved in actually getting some units manufactured, like for a small kickstarter run or something. It seems pretty doable these days to get pcb boards printed and even assembled once you have a final design. What I haven't been able to find much of, is advice along the lines of going from a prototype board like an Arduino or Teensy system, to a productionalized system in terms of both the software and hardware - what platform is easiest to build on (like this advice to use a Linux-based system if you need any networking), how to hook up a bootloader, how to build a system that allows users to update firmware for future bugfixes or features, etc. plus I'm sure there are a million other unknown unknowns I would never even think of.

[ChrisGammell]

I made a video course for getting up and running with KiCad, the open source electronics CAD program: https://www.youtube.com/playlist?list=PLy2022BX6EspFAKBCgRuE...

If you're looking at broader focus on building hardware, Contextual Electronics helps to fill out some of the other elements you're looking to do. As luck would have it, I just started a new course today (no joke) where I'm showing how I'm designing a Raspberry Pi HAT with cellular and Bluetooth capabilities, which I'll be selling and using as a reference design for my hardware design consulting business.

[OJFord]

How does KiCAD compare to Eagle? I find it being built in to Fusion 360 now (since Autodesk bought it) pretty compelling, but I haven't actually used any schematic/PCB CAD software since that I don't think.

[ChrisGammell]

I think they have been keeping apace for the most part, but the integration of F360 is a really nice feature if you do a lot of mechanical/electrical integration. I think it might be a bit overplayed, since my designs are usually, "Here's the mechanical space you're allotted, it's probably not going to change", but that's not the case for everyone.

As for KiCad, the most recent release (5.1.x) have been really stable and introduced a ton of features over the series 4 release. Like anything, once you're used to the methods, it's hard to get an objective look on how things work but I can tell you that I use KiCad for all of my professional design work and it has served me very well.

One thing I continue to be excited for is the extensibility of KiCad, since Python scripting is a main feature of KiCad (though the program is written in C++). This has developed a wide range of user plugins and scripts that have opened up some awesome new features.

One of my favorites is the Interactive BOM plugin, it's really quite amazing. Here's a user showcasing it on twitter: https://twitter.com/arturo182/status/1254376168322670593

[scottlawson]

Chris, I love your work and the AmpHour podcast! Thanks for working to share your knowledge with others, I especially enjoy the episodes where you spend the entire episode interviewing someone in the industry

[ChrisGammell]

Glad to hear it! I love making The Amp Hour and am flattered the people listen on a regular basis. Thanks for being part of the community!

[joshvm]

Take a look at Particle. They sell dev boards, but also have excellent documentation on how to scale (at least they used to, I'm not sure where it's gone now - have a look in the datasheet section) - obviously they want to get you into their cloud fleet management platform, but it would be a tempting solution for me. For example, you buy the dev board (fairly cheap), they also sell the SoC (cheaper when you buy in bulk) and have design guidelines for making your own boards.

The boards themselves are quite nice. It's an Arduino-like stack and you can always code in C++ if you need the STM32 capability underneath. The main stack is also open source I think.

Doesn't really cover much on the mechanical side, e.g. enclosure design. That's a whole different game and that's where people lose a lot of money - if you mess up an injection mold for example.

https://www.particle.io/

[throwaway9d0291]

I've done a small run of ~100 devices.

It was basically the same process as looperhacks: hook stuff up on a breadboard, turn that into a prototype PCB design, iterate on prototypes if there are problems, then just put 100 on the order form instead of 5 when you send it to the manufacturer.

You asked in the other comment about flashing and that depends. For me, my project was basically just a motherboard for an ESP32 dev board, which would serve as the logic controller. Since it has a micro-USB port and there were only 100 of them, I just flashed it from my laptop the same as any dev board.

For larger scale manufacturing, you can have the flash written in advance, before manufacture or put some exposed contacts on your board that can be reached with pogo pins and write your flash through those pins.

You can also outsource all of this to a contract manufacturer like Seeedstudio [0].

As for updates, the ESP32's software has built in OTA functionality that you can configure however you please. My boards are connected with MQTT over WiFi anyway so I added a simple "update now please" command that triggers the OTA. Aside from that you could add a USB port or stream the OTA over bluetooth from a smartphone.

[0]: https://www.seeedstudio.com/propagate/

[looperhacks]

Lately I've been prototyping a small board with my goal of maybe 10 units built. For me, the process is literally putting it together on a breadboard to check if it works, then design a PCB and send it to a Chinese manufacturer. Repeat if your PCB design is faulty. (Do double check your design though, you don't want to pay for new boards and shipping every other week)

[cactus2093]

I guess one big gap for me is just the build process of how to even load code onto the device to be executed, and what type of EEPROM or flash storage or other components are required for that, once you've removed the training wheels of the plug and play dev environment.

[cowboysauce]

It really depends on what you're using. Most microcontrollers don't require external memory to store their program because it's stored in on-chip flash memory. Additionally, you can generally order microcontrollers from a distributer that are already flashed with your program. Though this may require ordering 1000+ units. If that isn't an option then you'll have to ensure that the pins used for programming are exposed in some fashion and then you'll flash it. The exact details are dependent on exactly which microcontroller and toolchain you're using.

It mostly comes down to reading the documentation for the specific microcontroller that you're using.

[im_down_w_otp]

The power budget isn't the only reason you go for an MCU. Often you're doing it for the real-time guarantees or the hardware qualification (e.g. for applications where you need something with a safety rating). That said, yes... TCP/IP based networking in the traditional embedded world isn't comparatively pleasant.

[CamperBob2]

Look at the NetBurner products. For the most part they Just Work, and their uCos-derived OS is a pleasure to use. It has a distinctly simplistic old-school vibe, but they have updated it to support WebSockets and numerous other new(ish) features. The end result is a good compromise between LwIP and full-blown Linux/BSD.

No association with the company, just a satisfied user.

[realharo]

How are things on the commonly used RTOSes? I'm not really into hardware much, only fairly recently learned basic electronics as a hobby, but I've seen ARM push their "mbed" (https://os.mbed.com/) thing in many places recently (most recently saw their logo printed on a nRF52 BLE dev kit). I've also seen Amazon get more involved with (in charge of?) FreeRTOS (https://aws.amazon.com/freertos/) quite some time ago.

Don't such things solve most of these issues?

[adammunich]

The RTOS world is much less of an 'operating system' and more so a bare-bones task scheduler. It's not like a POSIX environment at all.

[realharo]

Right, but doesn't it also provide some very basic level of abstraction that things like network stack implementations can target? And then many projects also come with their "standard" (not bundled, just designed for) set of basic libraries? That's the impression I'm getting from skimming through the docs. (https://docs.zephyrproject.org/latest/guides/networking/net-..., https://www.mbed.com/en/technologies/security/mbed-tls/, https://docs.aws.amazon.com/freertos/latest/userguide/freert..., etc.)

[jnwatson]

You are correct.

[nikofeyn]

that's not really true anymore. it may apply to things like pharlap or vsworks, but there are more modern options that give you a nice real-time linux environment.

https://access.redhat.com/documentation/en-us/red_hat_enterp...

https://www.ni.com/en-us/innovations/white-papers/13/introdu...

[numpad0]

Linux with RT patches isn’t exactly RTOS, it’s Linux with such patches.

[nikofeyn]

how is it not an RTOS?

and national instruments and red hat are both large companies that would seem to disagree with you. national instruments has completely replaced pharlap and vxworks in their newer products with their linux real-time OS, so them and their users writing real-time applications would also disagree with you.

[numpad0]

See how Red Hat words it? “Red Hat Enterprise Linux for Real Time” not “RHEL RTOS” as if they don’t think Linux is technically not an RTOS. Or maybe they do think that way.

And see that chart down that page? Shows RHEL for RT achieve 3.5 to 85uS variance. I don’t know what application needs it but 80uS is 24km at the speed of light.

LabVIEW is NI’s bread and butter and LabVIEW connects to peripherals via cables and buses. NI not opting for conventional RTOS just means that 80uS errors are probably good enough for their usage.

RTOS also often refers to specific group of lightweight OS and libraries more so than capabilities. People want small no-bullshit runtime, not always a precision scheduler. From that point of view, Linux is just a bloatware towards extreme side so don’t qualify as one.

Overall, no, I still don’t think Linux is RTOS, I don’t think Red Hat says it is, and I think NI just don’t care.

[comboy]

I wonder what's your use case for RTOS? I don't really know any problems that couldn't be solved with micros communicating with linux. Seems like even SpaceX gets away with it.

I mean there seem to be the market so clearly there are some people who need it, I just would like to learn more about where is it really needed.

[monocasa]

SpaceX uses FreeRTOS and VxWorks for chips that need hard real time guarantees from what I've heard.

[HeyLaughingBoy]

It's not that the problems couldn't be solved that way, it's that they don't necessarily get any simpler. Adding an external micro and all the support it needs is added cost, maybe added real estate, power budget and very often, it makes the solution more complex and less reliable.

[numpad0]

RTOS are mainly for micros communicating with Linux.

Lots of time they are just a giant C library that goes into the same a.out, that for devs look close enough to let himself pretend there will be an OS running his code.

[realharo]

Stuff powered from a coin cell battery?

[comboy]

From coin cell battery you can power a microcontroller that is sleeping most of the time, not a real-time operating system.

[dekhn]

You can do exactly this with an ESP32. It's got an RTOS (FreeRTOS) which will spend most of its time sleeping https://learn.circuit.rocks/esp32-deep-sleep

[comboy]

Debatable. Personally I would not try to do a project that uses wifi on a cell battery. And if you don't need wifi then you have much better options that use low current not only while sleeping (e.g. nrf series)

But my main point which I should have included in the previous reply was - what does real time OS have to do with the project being run from a cell battery?

[rwmj]

It has a 32 bit processor and at least 1 MB of RAM and 8 MB of flash. This makes it considerably more powerful than the first PC I owned. Could it run Minix 3?

[ChuckMcM]

Yes. And in fact a 32 bit microprocessor with a meg of flash and 256K of RAM can run a number of useful OSes that have networking capability.

What I find interesting about the situation is the Arduino environment. The Teensy originated as an "Arduino" type dedicated controller board. The Arduino environment originated on something that had 32K of flash memory and 2K of RAM. (Think Altair 8800 or IMSAI 8080).

It was designed so that people new to embedded controllers could get something running quickly on a very simple processor.

The Cortex-M architecture (which the Teensy uses, as does the STM32 parts) is somewhere between the 68000 and the 80286 in terms of its capability (no MMU though).

On at $30 dev-board (STM32F429-DISCO) from ST-Micro I've run a full shell/loadable program environment with networking based on a FreeRTOS kernel and a shell of my own devising.

It is the Arduino environment that cripples these things, if they ran the equivalent of MS-DOS none of these complaints would make any sense at all. (and they are very capable of running that level of OS)

[Hello71]

as I recall, PCs from the 90s also had extremely poor networking support.

[baybal2]

> But if you're building an embedded device that will be deployed in any sort of volume, a small application processor running Linux will make your life dramatically easier.

Can you name some? I think of most embedded network stacks as being easier than raw socket APIs on Linux or Windows.

Programming network on Linux only looks that easy because of megatons of ready to use abstraction libraries available.

[ex3ndr]

I was puzzled finding a good linux board for cheap device. Decent linux is like 10$ price minimum and you probably need something better. It could easily sky rocket to 30$ and this is out of budged of almost any devices. 30$ makes your device retail cost ~100$ already.

While you can get M4 for under a 1$ having all built in and very power efficient.

Also no need to hustle with linux deployment and update rollouts.

[jnwatson]

There are several vendors of embedded RTOSs that will give you a decent TCP stack. Especially on the lower end, Linux is not a good option.

[MrBuddyCasino]

But what to use? The Pi Zero can't be used for mass-production, because you can't get it in quantity. Other options exist, but they are even more expensive, or obscure chinesium for which you can't get documentation or someone to talk to.

[joefourier]

If you want an decently peforming SBC that can match the Pi Zero in price, unfortunately you're out of luck. But if you're willing to put in a bit of work, make your own board and suffer most things not working out-of-the-box, you have a wide variety of cheap SoCs and MPUs. E.g. the STM32 MPU line, which has a decent enough ecosystem and plenty of devboards available, and has some chips under $5/unit in bulk.

[MrBuddyCasino]

None of which have wifi, as far as I can see?

[numpad0]

The original ESP8266 ESP-01 was among such devices to add Wi-Fi to host CPU over SPI or UART...

[joefourier]

You need an external module for WiFi, but the standard Linux networking tools and drivers work.

[throwaway9d0291]

Development kits like the Pi Zero are usually intended solely for development and experimentation. When it comes to production, you either take the product the dev kit is wrapped around or there's some kind of module you can use.

For example Raspberry Pi has the Compute Module, ESP32 comes as a solderable module and for the Teensy I suspect you'd just put a IMXRT1062 in your design.

[chillacy]

Is particle.io considered one of the more expensive ones? I've used them for one-off projects but it looks like they do volume discounts.

[chapplap]

Yes, this is a very good point. Low-end application processors capable of running Linux are also very cheap now and it is easy to build a minimal system for around a few bucks (eg. Allwinner F1C100s, although it does not have an Ethernet MAC built in). Not great for hard real-time stuff, but if the requirements there are limited a separate microcontroller could work.

[closeparen]

>Are you really prepared to roll out updates on a system like this when the next Heartbleed comes along?

Implying that embedded systems get updated...

[HeyLaughingBoy]

Lots of them do. I have a bigass board right next to me with a couple dozen wires sticking out that can get updates by flash drive. We ship tons of hardware that gets updated over Ethernet or OTA.

[subhobroto]

> The hard part is not cramming in an Ethernet PHY or a 2.4 GHz radio, it's the mountain of software that you need to run on top of it to get the kind of reliable, secure communications channels that we've come to expect

Have you looked into PJON?

[swiley]

I totally get what you're saying.

Although I'll add that Linux has it's fair share of networking bugs and quirks (less now than before, but still plenty.)

[vaxman]

Instead of that, how about an nRF5340 running Linux Foundation's Zephyr Project OS on its application core?