This makes me very excited! I have Spinal Muscular Atrophy type 2 [1] and I have lost most of my physical capabilities save for 2-3 fingers and of course speech. Although I am now on Nusinersen [2] treatments I am still becoming weaker over time, albeit extremely slowly.
It brings me comfort to know that such a fallback will eventually exist, should I need one.
Note that specialists are saying that another promising drug from Scholar Rock [3] would probably prevent any further weakening if used in conjunction with my current treatment. Unfortunately, the FDA takes a long time to approve new medications and I have heard this one is particularly special because there is potential for abuse by athletes.
I hope you can get on that promising drug ASAP and that it works wonders. I think the future looks bright for medicine with all these new breakthroughs coming out.
Doesn't that headset require you to be able to move your head in various directions, and be able to use gestures and such? I just looked at some videos during the release, and those things seems to be a pretty fundamental part of using it.
From what I understand the headset can be configured to be used without moving the head and even to use eye tracking, only, instead of the minimal finger gestures.
You don't necessarily need to move anything other than eyes. It's the default input method, but it doesn't require it. It's a computer, it can connect to other devices or use APIs.
So if you use a headset meant for spatial movements, but you're unable to do spatial movements, can't you just use a display and get exactly the same experience?
A very close experience yes. It works just fine with zero head movement and you can move and adjust your windows to only be directly in front of you. And they are by default
I love that one of the first things he said he did was binge on Civilization 6 until 6am. Welcome back, buddy!
Strangely that simple example was the most powerful part for me. I've done that so many times and it was such a fun experience. Now he gets to re-live that joy (and follow up shame) again!
Some clarity for ppl.
- Other techniques would likely work for this patient. Eyegaze technology is pretty readily available. He has proximity switches for driving his chair and lets not forget voice control. So.. what does BCIe offer significantly? I think this is the BIG problem BCI has. The gains are not enough for a lot of people compared to what the AT sector can already offer. Please remember this. This guy could use eyegaze or voice control on pretty regular hardware. And no surgery needed..
- These types of BCI are effectively an array of switches. You typically map a motor thought eg. "Move your arm up" => Moving the cursor up. This maybe how then you control a game such as chess if it has keyboard shortcuts. Eye movement could be done in the same way but there are easier ways. Interestingly to measure these motor commands you dont really need intracortical BCI. You can do it with surface EEG. Sticking it inside your head - closer to centres where you can measure intentional thought makes the signal cleaner and more reliable
- The big breakthroughs is really making this intracortical stuff safer and long term. Its getting there. But this isnt it
The big wins out there - are in speech BCI. Thats hardcore. Even the two main studies doing this - each of the participants requires a LOT of training time to make a Machine model work efficiently.
Eye tracking does not work nearly as well as people imagine. It cannot directly replace a mouse pointer the way you want it to. The accuracy and reliability are not good enough and never will be due to physical constraints. This system is likely already working better than an eye tracker would for cursor control, and it will certainly improve.
Apple has done great stuff with eye tracking on Vision Pro, but it required completely rewriting the UI for literally everything. Not something we have the luxury of doing for accessibility for quadriplegics.
Source: built an eye tracker and eye-controlled UI at a startup and got acquired by Google
I hear you. I literally cant use any commercial tracker because of a alternating strabismus and the midas touch effect is a nightmare. I'd argue Apple did that because its such a mess on regular OS where the primary input method isnt eye gaze. (They did that with iOS too - match the OS and UI to the primary access method)
I'm not sure I can believe this "This system is likely already working better than an eye tracker would for cursor control" - the training to use this stuff isnt just 'magic'. I do agree though with "and it will certainly improve" - yeah iterate.. iterate..
He notes these other forms of HCI in the video, and I think you’re really underselling the main point of this being ease of use. All of those methods are significantly degraded experiences compared to normal ways of interacting with a computer. The potential for a quadriplegic to interface with a computer at a higher ease of use compared to a human without disabilities is huge.
Sure. The problem is this video somewhat misses how that interaction actually works. You need a control technique of thinking to action. What that actually is and how you learn to use that is missing from the demos. Thats a tad misleading right now IMHO.. (NB: It could be intentional speech eg "Move x to y" or it could be intentional motor control "move my hand right and up" You would also need a stop command. Not sure.. )
But sure - "the potential" is the key thing in all this. Just the cost to benefit ratio is pretty dramatic right now.
I don't think the point for him was to simply find a way to control his mouse. He said at the end of the clip he specifically wanted to help out with Neuralink. He also said the surgery was easy and he was released a day later.
He appears to just think about where the mouse should go and then be able to click and click-and-hold. Seems like multiple inputs which an eye tracker wouldn't do. Unless maybe its just configured to click when the cursor pauses on a spot?
Also the user experience seems better than attaching electrodes to your head. It seems to just work wirelessly. It is always there and sometimes he has to recharge it.
Theres a lot of patients who "want to help out" Eddie Chang's team in SF have one patient who moved right next to the hospital to equally do this.. (NB: they should be getting paid too by the way - although thats not clear)
Yeah - this is wireless. Better than some systems which have been, no joke, a box at the top of your head with a HDMI cable in it..
> He appears to just think about where the mouse should go and then be able to click and click-and-hold. Seems like multiple inputs which an eye tracker wouldn't do. Unless maybe its just configured to click when the cursor pauses on a spot?
This is really the key question. The dwell technique you note is what most eyetrackers do (although far better to use a binary technique - eg a blink - to select because of the midas touch effect). Its built into to Windows/MacOS and iOS now. I have a sneaky feeling the reason why its chess is you can encode the positions "X1 to Y2". You can then do a transformer model to decode intentional speech..
If that is the case - then if a person actually speaks whats the benefit right now for this indvidual? (yes - that he doesnt have to say it. BUT sub-vocal speech is already achievable without invasive surgery..)
> You can do it with surface EEG. sticking it inside your head - closer to centres where you can measure intentional and thought makes vy signal cleaner and more reliable.
Further clarification: when doing conventional EEG, the signal quality is so fragile that even blinking can produce recording artifact.
Also, there's the whole "put a shower cap with conductive gel" things that makes it very impractical for every day use.
You don't need a full EEG to do what the parent mentions. 3 electrodes around each eye (EOG) + ground is all that is needed. Even cheap commercial electrodes will work for this. Blinks can be filtered online with PCA-like techniques.
I think once this tech can be used to connect to a robot / exoskeleton, then it will be very useful for someone like him. Imagine him thinking that he wants an apple from the fridge in another room and the robot goes and grabs it for him.
Yeah I agree. In reality though - right now - I'd say you might be surprised to what people actually want and need. For example I could assess someone for say a curtain opener or door opener. I have fallen into the trap of assuming that someone would want this. In reality though its very few people who do. Why? Its totally overkill and pointless when 90% they are with someone who will just open the door anyway. (NB: This isnt Everyone - the ubiquity of home control systems on the regular commercial marketplace is changing this I think)
Sorry but what you're saying simply is not true from my experience. The potential of this technology is the ability to perform multiple actions simultaneously e.g. having your character strafe while zooming in while firing a trigger. With eye tracking you are limited to (for the most part) one action at a time, which is what I do now.
You are correct in that one could add more inputs but that only works if you can use the inputs. The individual in the video has full control of his head which many people do not. All I can do, for example, is use like two fingers.
There’s a ton more that these implants can do other than cursor control. And not needing external visible hardware like other forms of input is a bigger deal than you might think, and especially in terms of the user wanting to feel more “normal” and less of a robot in a bundle of contraptions
I haven't followed Neuralink too closely since it was announced, so I was not expecting to see what I just saw. I've seen a handful of breakthrough moments in my life - I think this will be remembered as one.
Why would this be remembered as a breakthrough? Playing games with a BCI is many years old at this point. Here is an article from 2020 talking about playing Sonic the Hedgehog, amongst other games [1]. Here he is fist bumping President Obama in 2016 with a brain controlled robot arm.
I don't think the device itself is a breakthrough, the issue beforehand was that tissue in the brain tries to heal from the implant, it can be lethal, I don't know what they're doing to have a PERMANENT implant, completely stop that area from "healing" so tht the implant doesn't become a legitimate hazard.
The main issues with previous BCIs were that the electrodes were larger, less flexible, and not as biocompatible. This caused scar tissue to form around the electrodes, degrading the quality of the signals and eventually making the device useless. Neuralink has apparently reduced scar tissue formation enough that the implant receives usable signals for years. Neuralink also uses far more electrodes, improving signal quality and adding redundancy. The number of electrodes necessitates the use of a robot to place them. The robot is programmed to avoid blood vessels, which also reduces bleeding and scarring.
It will be quite a while before we know if they've managed to mitigate the issues enough to last a lifetime (and support upgrades), but it's better than previous devices.
Can you point to a study showing that Neuralink has achieved any of those things you mentioned? What evidence is there that the implants have reduced scar tissue formation and what supports the assertion that they would be usable for years? What other previous devices are you comparing it to that allows you to positively assert that it is better than them?
You can not just reel off every bullet point on the marketing slides without support as if it is the truth.
the only development (and it wasnt invented by neuralink) is flexible and smaller electrodes seem to take longer for scarring to take over than larger and more rigid ones. Right now, it seems that their only path is to help people with severe disabilities get better quality of life for a few years, which many people might take.
Personally, I think the most exciting part of Neuralink and other companies working on BCIs is the fact that they're trying to keep these implants in long-term, and scale the deployment significantly. Most academic BCI research thus far has just been trials, without patients getting to keep the implants long term.
Moving the mouse wasn't that impressive. That he could turn of the music just like that while the game was open was impressive. And Civ 6 is way more complex to operate compared to chess. I assume that's not mouse driven.
Still early days for this tech but it seems impressive.
> And Civ 6 is way more complex to operate compared to chess. I assume that’s not mouse driven.
Civ 6 is drivable completely with the mouse, and other than editing gold amounts in trade offers a little more quickly there’s not much reason to use anything but a mouse for it.
I never could get into Civ. Do you have any tutorials or a specific game edition to recommend? Maybe my issue is that it's too slow to get started with, but it seems like I'd enjoy it if I stuck with it.
Why do you say “5/6”? its an open-source clone of 5, and the Readme notes that given the time it took them to get done with 5 (i.e., not yet, as the base mechanics are only “mostly done” and there’s a bunch of stuff from the expansions still to do), implementing Civ 6 mechanics is an express “won’t-do”.
You either like it or you don’t. The inbuilt tutorial is pretty self explanatory and took me 15 mins to really get into it from scratch. It’s slow and turn based and that’s not for everyone.
Factorio on the other hand took a while for me to start liking but that’s because it has a huge learning curve. I eventually got into it but it’s not something I crave playing.
Not a big Civ fan. I had way more fun with Paradox games like Crusader Kings. You can learn them by watching a Let's Play on YT. After watching a few hours of somebody play in a roll play fashion, I jumped in.
A key differentiator of Neuralink is that the implant can both read and write through ~1000 channels (each of which is a tiny wire into the brain). So it's not really the same thing as external devices that read electrical activity from outside the brain, because those cannot write data. Not sure if the initial implant supports much of this, obviously you'd start with the simplest use cases.
Which is why I will never get anything like that installed unless I'm paralyzed or effectively so. Humans have no mechanism for recognizing that a sensation, thought, or emotion, which arises within themselves, is actually inserted by malware. No thanks.
That's pretty amazing, the fact he's able to click the pause button with his brain alone is insane to me - that's like Apple Vision Pro without the gigantic goggles.
This is a good point. Seems like a likely candidate for an S curve in tech development. i.e. next 15 years of VR are improvements to camera, display, and tracking technology. Following 15 years are brain implants.
The sheer joy on this mans face to be able to freely control a mouse again, and engage with general technology. If they’re able to make this into a generally safe procedure, a lot of people will be interested in just that.
As soon as you can stimulate tactile impressions it's over. You can put on your VR Headset and be in a completely different world. Eventually the interface for eyes and ears will improve, but tactility would be a huge step towards being in a completely virtual world.
Semi related tangent incoming: I’m reminded of a book I read last year named Semiosis by Sue Burke.
Tiny spoiler warning I guess though not really, it’s just background world building that was used as motivation for side character’s growth. In the book, there was a Hitler-esque villain who existed long before the characters were born. The villain killed many billions of people. But through cloning, the societies of Earth punish this villain for their entire life by feeding them torturous scenarios through their brain implants. These were scenarios like being chased and eaten by a tiger, running naked through a frozen tundra, execution, etc.
The clone thought it was entirely real because it was all in their brain implant, even though they were safe in a jail cell. And as an extra Black Mirror-y twist, anyone in that society could tune in with their own implant to watch the clone being tortured.
I’m not really trying to cast doom and gloom on this brain implant tech, I think it’s neat. I was just reminded of the book I read when you mentioned simulating tactile impressions and virtual worlds. Pleasant simulations would be great, but even “benignly” scary ones like a virtual haunted house in your brain could be terrifying. (As someone who hates haunted houses.)
The inspiration for this (as well as for SpaceX) comes from the “Culture” series of novels by Iain M. Banks, which most people are apparently unfamiliar with. Specifically the BCI interface is called “neural lace” there and it grows along with the brain from a seed and covers its entire surface. There it serves as an interface to access superhuman AIs and information in general, on demand, and only the hopeless luddites choose not to have it
Sorry but surely the idea of having a brain computer interface predates the Culture novels by many decades. Similarly, how are these books the inspiration for SpaceX? Those ideas of traveling the stars (well, planets) _absolutely_ predate the culture series.
Seems pretty obvious to me that these ideas originated long ago in the scientific world and were (beautifully) expanded upon by science fiction authors (again, many decades ago).
That’s fair I suppose but it’s probably giving more credit to Musk than he deserves. The real work is all being done by researchers who are drastically more knowledgeable about all of this than he is. He’s wise to invest in these sorts of ideas though and deserves credit for that.
Yeah those researchers would be aimlessly rotting in academia if he didn’t fund risky, out there stuff like this. I can’t name anyone else among his peers who spends their wealth on anything even remotely civilizationally beneficial.
They probably interviewed him to see if he’d be a good person for PR reasons. And I imagine he got it for free. So he has plenty to be happy about with the situation
I'd like to know if they're doing 'online training' - ie. Do the weights of the neural net which converts raw signal data into mouse movements update themselves every few seconds using historical data?
Such online training might be necessary to deal with brain plasticity - ie. The optimal set of neurons to read to determine X/Y mouse movement right now might not be the same set it was an hour ago.
Such plasticity can be seen in regular humans too when they say 'whoa, I haven't used a pen for months - let me get used to writing again!'.
But giving it the benefit of the doubt, this looks mind blowing to me !
It was kind of known that the research and tech is almost there for a while already, but seeing the demonstration live like that - incredible !
But then it takes me back to those Musk companies - maybe it's just a repackaged already available research presented in a nice way - making us believe it could be 'deployed' in real world, while in reality it can only be done in a very controlled environment. And we are led to believe that we are '2 weeks away' from it being widely available. Hope we're wrong here.
While not an Musk company, it is worth remembering Theranos demos being convincingly faked as well. I don't know if it is happening here but it does happen.
That's not faked though.. according to your link they tried multiple times and only showed the success video. But the statement that the software drove with no intervention was accurate, and the video was authentic.
How about that time they claimed the Cybertruck beats a Porsche in a 1/4 mile race, but they only did a 1/8 mile race and then just made up a 1/4 mile time based on their feelings? [1]
Or that time they lied about 120,000 defective suspensions that broke causing crashes and fatalities and then blamed their customers and covered it up so they would not have to issue a recall in the US (they were forced to issue a recall for the issue in China)? [2]
A decade of unequivocal lies about delivering autonomous vehicles [3] while knowingly killing tens to hundreds.
But other than that, Mrs. Lincoln, how was the play?
The link says that the route was pre-determined, pre-mapped, tried multiple times and crashed once during the trials.
By the same logic, this video would not be faked even if in fact it's the same chess move he tried for 2 months and it's the only one working, and also another patient died.
Not saying it's the case - clearly not. But yeah, let's be real.
That's at least misleading. By that standard, the current video wouldn't be fake if the subject were primarily operating his computer through eye-tracking software, since no explicit representations were made about the mechanism of control.
Doesn’t have to be faked either, I remember the Emotiv made similar claims and had similar demos where to the best of my understanding eventually people figured out it wasn’t really reading your brainwaves but rather facial muscles or something (a friend of mine works in brain computer interfacing and explained it to me once but I don’t remember the details — it wasn’t that they were actively trying to mislead necessarily, more akin to leaking validation data into neural network training than to outright deceiving).
I don't think it's fake in this sense - the technology already exists for quite some time - there are devices like that even allow people to talk - check Synchron for example.
But it may be fake in the sense that it's not viable for a long term use and / or necessitates a very controlled environment. That would indeed match the pattern for Musk companies - articulate a grandiose vision, show a working demo to amaze the world, then show progress on some scale but without disclosing the issues that actually would make to product not viable or not financially viable in the long term, and sell the hell out of it to investors.
If you look closely, all his companies work like that. Yes, including Tesla ! Their car business is at the point where they may start loosing money again - competition, slowdown in EV subsides, but also factors that allowed them to reach an apparent profitability - for example, their service is nonexistent, while it used to be great. Basically they sold millions of cars, while their service centers are still scaled for thousands, not millions. This will a big financial liability, but not really visible in their financial for now.
But hey, they already transitioned to another business that promise to make even more money in the future with other promising tech ! AI anybody ?
This is some weird stock market doom and gloom astroturfing and is a massive stretch to discuss given the context of the OP
Also after reviewing your post history, it’s overwhelmingly you shitting on the same stocks with really ridiculous hyperbole and not much else. And a warren buffet quote thrown in for good measure.
FSD being good isn't really the same as the Robotaxi idea. Robotaxi was where you could rent out your car and it could completely autonomously drive other people for a fare. Your car would make money.
I haven't seen 12.3 but other versions had some frightening mistakes in videos. It feels like it will eternally be 90% there but never actually reach the end goal of completely autonomy.
Personally I love Autopilot even if I only use it to drive on the freeways for me when I'm tired.
The headline is wrong, the video was not faked (nor edited). The guy just said what was in the video was not currently shipping to customers, i.e. it was a tech demo.
I want to know how big the GPU crunching all the numbers is to make this work...
The mouse seems to move very nicely and smoothly (60 FPS?) which presumably means the neural net which converts raw sensor data to mouse movements is running in ~15 milliseconds.
Most neural nets don't do a forward pass in 15ms unless they're either tiny or the GPU is very powerful.
This video kinda looks like the patient really likes their new implant and it's abilities, but is pretty frustrated they now need to do more trials/make marketing videos for the implant when they just want to get on with their life...
not exactly it is a bunch of wires measuring differential voltages inside the brain not on surface of the skull. So you get a much better specificity in the signals (think of it as resolution). even if you out 1000 probes on a skull you wouldn't get much information.
It is fascinating. But looking at this guy condition, playing games would be IMHO not the prio. I would hook up the remote controlling to a robotic exoskeleton, for just be back and function normal. I guess that also can be detected ( intention to move feet or hands in any direction ).
Your idea is that the priority should have been, instead of first getting a mouse working on his laptop, to hook him up to some kind of robotic exoskeleton?
Of course, but that’s going to take so much more time and effort. You can see how much joy this man is in just having the autonomy to move a mouse again. That’s honestly amazing.
I don't think it can simultaneously control the dozens of degrees of freedom that a robot would have. Yet. You only need three degrees of freedom for a mouse with button. That alone is transformational for a quadriplegic.
It brings me comfort to know that such a fallback will eventually exist, should I need one.
Note that specialists are saying that another promising drug from Scholar Rock [3] would probably prevent any further weakening if used in conjunction with my current treatment. Unfortunately, the FDA takes a long time to approve new medications and I have heard this one is particularly special because there is potential for abuse by athletes.
[1]: https://en.wikipedia.org/wiki/Spinal_muscular_atrophy
[2]: https://en.wikipedia.org/wiki/Nusinersen
[3]: https://scholarrock.com/our-pipeline/spinal-muscular-atrophy...