[larme]

People who are interested in this application should check synplant[0]. It has a ML technology called "Genopatch" which gives you 2 functionality:

1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags

2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.

My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)

[0]: https://soniccharge.com/synplant

[aw123]

How far are we from getting a general model that can resynthesize any instrumental audio sound without fiddling with any knobs, so that we can recreate instruments we hear from any song? Seems like it should exist by now?

[larme]

For me creating the exact sound is not very interesting from sound designing perspective. You can always sample the real instrument.

Like physical modeling synthesis, the interesting part is to compress the sound to some parameters that you can tweak and generate new sounds

Another approach is VAE, which also you give your some latent embedding, you can tweak the embedding to generate new sound. However the meaning of this embedding is not explicit.

[lightedman]

"You can always sample the real instrument."

This doesn't really work on instruments like guitars. Open D sounds way different than fretted D on the E string. Timbre changes with position and it's one of the ways I determine where a player's hands are on the neck when I'm trying to play their song.

[beepbooptheory]

I'm not doing fancy AI stuff but I have worked a lot with my own bespoke supercollider system where I record whole fretboards of guitars and then play alternative notes based off of certain rules. For whatever dumb reason though, the most natural sounding thing is really just playing, e.g., any random D4 from its possibilities at any given moment.

Timbral differences also exist depending on force, the manner plucked, the already ringing overtones... It's hard to know what you want, but the most natural thing is always going to be some organic variation in the notes in general.

If you have a good ear, you aren't, I don't think, hearing so much the timbral diff in the individual open or fretted notes as much as the fact that a barre chord and an open chord is a different voicing of the same harmony.

[lightedman]

No, I'm going off the timbral differences - same way I identify which pickup position is being used. There's a specific 'thickness' I cue in on to determine pickup and specific note placement.

[beepbooptheory]

Huh, got it. That's pretty cool!

[ofalkaed]

That is not something inherent in guitars themselves, it is the norm in steel string guitars and the fan-braced/Spanish guitar but mostly because that is the norm for all those mass produced guitars which make up the bulk of guitars. On steel string you can often greatly decrease this quality just by switching to flatwounds, this is part of the flatwound sound, it shifts the timbrel content into the players technique but if you want much timbrel content with flatwounds you need heavy strings and a high action, and the hand strength and technique that sort of setup requires.

Before the rise of the steel string and the Spanish guitar, guitars tended to be more even across their range and also had less bass which helped even them out, and now that sound is what we are used to. There have always been niches that wanted that more even sound, but for most that just makes it more difficult to play all that music that developed around these quirks, so they remain niches.

[mark_something]

It doesn't even work for most instruments, nearly every instrument can sound differently depending on how you play it. A violin sounds different depending on how close the bow is to the bridge, a piano sounds different depending on which pedals are pressed and whether the lid is open, a trumpet sounds different depending on whether there is a cup in front of it and where. Experienced musicians know how to use these effects to create the right feeling.

And that's only based on how little I know about this.

[aa-jv]

In the early days of FM synthesis it was not uncommon to hear the refrain "FM synthesis can reproduce any sound you can hear on the radio" from some of the wilder-haired synth nerds of the time.

Aphex Twins' MIDI Mutant came pretty close (quite a few years ago now) to delivering on that promise:

https://magazine.raspberrypi.com/articles/aphex-twin-midimut...

I'm pretty sure another pass at this problem would prove quite fruitful - as others have noted elsewhere in this thread, there are tools like SynPlant which promise this kind of functionality, although - for my needs - I much prefer AT's approach, having it all in a single box.

Seems to me that the Zynthian/Monome[1] folks might have something like this in their toolkits, somewhere. Might be time to catch up with those projects...

[1] = https://zynthian.com / https://monome.org

[tashian]

Fiddling with the knobs is the fun part. https://www.youtube.com/watch?v=la2u4VlGwbQ

[zhinit]

SUNO is pretty close. It still has some weird things going on with high frequency artifacts and phase between left and right channels but if you aren't listening on a good system (like a phone) most people probably wont notice.

[zhinit]

synplant is a great synth!