Chrome 149.0.7827.200 (Official Build) (arm64), macOS Tahoe 26.0.1
It doesn't make it any less impressive to those who know what hardware requirements for LLMs usually is/are but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse and yet are unable to upgrade because of said LLMs stealing all the hardware in the world all while RAM/memory/storage manufacturers manipulate the market(s) against them.
If you are curious I used a NVIDIA GeForce GTX 1660 SUPER So to be exact, it came out 7 years ago (I upgraded at some point on this desktop a long time ago and didn't remember the exact year) (I updated the article to reflect this now)
This cost $230 new and you can get one now for $100 which I don't think is too out of reach.
Interestingly, now the #1 GPU is the GeForce RTX 4060 Mobile version, which I believe is the first time the top has been a laptop chip instead of desktop chip.Items #2 and #3 on the list are the 2 generation old RTX 3060, followed by the 1 generation newer RTX 4060. 4th and 5th are RTX 5070 and RTX 3050.
I don't think "those with no idea" spend much time thinking about their hardware at all. They respond to marketing and peer-pressure influences, but most of them are not upgrading phones or laptops because they can't run AI on it.
Most people I know have been wanting upgrade cycles to slow down for quite some time, now. I think that those people will survive deferred retail therapy for a few years.
That I didn't get from the text.
0: 128 x 173
1: 64 x 87
2: 32 x 44
3: 16 x 22
4: 8 x 11
Then i used 4 separate channels.
This was somewhat arbitrary due to the local training constraint. This would be a hyper parameter worth tuning if I had time to dig into this more.
I trained this a few month ago and don't remember exactly what I tried before I arrived here, but I only ran the whole process 2 or 3 times because of how long it took to train. Hope this answers your question!
1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags
2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.
My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)
Like physical modeling synthesis, the interesting part is to compress the sound to some parameters that you can tweak and generate new sounds
Another approach is VAE, which also you give your some latent embedding, you can tweak the embedding to generate new sound. However the meaning of this embedding is not explicit.
This doesn't really work on instruments like guitars. Open D sounds way different than fretted D on the E string. Timbre changes with position and it's one of the ways I determine where a player's hands are on the neck when I'm trying to play their song.
Timbral differences also exist depending on force, the manner plucked, the already ringing overtones... It's hard to know what you want, but the most natural thing is always going to be some organic variation in the notes in general.
If you have a good ear, you aren't, I don't think, hearing so much the timbral diff in the individual open or fretted notes as much as the fact that a barre chord and an open chord is a different voicing of the same harmony.
Before the rise of the steel string and the Spanish guitar, guitars tended to be more even across their range and also had less bass which helped even them out, and now that sound is what we are used to. There have always been niches that wanted that more even sound, but for most that just makes it more difficult to play all that music that developed around these quirks, so they remain niches.
And that's only based on how little I know about this.
Aphex Twins' MIDI Mutant came pretty close (quite a few years ago now) to delivering on that promise:
https://magazine.raspberrypi.com/articles/aphex-twin-midimut...
I'm pretty sure another pass at this problem would prove quite fruitful - as others have noted elsewhere in this thread, there are tools like SynPlant which promise this kind of functionality, although - for my needs - I much prefer AT's approach, having it all in a single box.
Seems to me that the Zynthian/Monome[1] folks might have something like this in their toolkits, somewhere. Might be time to catch up with those projects...
Sometimes the "by me" is the interesting / fun / instructive part.
This would be a cool project to work on. Ideally you would buy some vintage gear and then run the audio through both, but that would be very expensive. You could may be find some vst emulations though and get decent results.
I think it might be that it's missing a large part of the lower frequencies, not that entire bass sound is missing. And I guess it'd be hard to faith-fully regenerate those, if we simply don't have a lot of samples.
Source: have degree and postgrad in jazz and used to be a bass player. Have made transcriptions of early bass players from original recordings. (by ear without any kind of fancy tech)
[1] and the playing technique for various reasons.
The Mooche
* https://www.youtube.com/watch?v=BPJ5vNmVL3I (The Duke, 1928)
* https://www.youtube.com/watch?v=tLdCq2PKM4o (John Barry, 1984)
Sugar Foot Stomp
* https://www.youtube.com/watch?v=qEdIWVsfPXs (Henderson, 1925)
* https://www.youtube.com/watch?v=35foefTrdLo (Nighthawks, 1993)
Doctor Jazz
* https://www.youtube.com/watch?v=HTYAaX7lqjQ (Morton, 1926)
* https://www.shazam.com/song/1687138286/doctor-jazz (Asaro & Fat Babies, 2016)
I was wondering if anyone was working on this (i.e., using pure software). Period equipment, real musicians, and such would be time consuming, arduous, and cost prohibitive.
I could take my sequencer and crank the tempo up to a level on a Chopin etude that would smoke Yuja Wang too.
Who cares? The performance that is interesting is a human performance under these artistic constraints.
We didn't need transformers for algorithmic jazz or algorithmic composition in general.
It is also the bullshit of algorithmic Bach. Bach produced 1,100 works and most people haven't listened to even 1% of arguably the greatest artist who ever lived. What is the point of generating more?
One, a ubiquitous restoration model. Find degraded copies of music in the wild, old YouTube's, transcodes, vinyl rips, bad masters, half destroyed tapes... Pair them with modern pristine lossless encodes of the same music, train. Then use that model on music we don't have pristine copies of.
The second is similar but more specific. There are so many stems floating around from popular music. My idea is to compare individual stems against the results from MVS/Spleeter(same song, same instrument). This would surely stand a chance of pushing that tech forward, so we can treat the FFT artefact heavy sound of new efforts.
Thirdly, from a creative point of view, I wanna do the equivalent of image to image on my tracks... But I actually want it to hallucinate in the manner of the early deep dream images, I want to be able to play with that space..
I can knock out musak to spek in minutes already, gen music is just reducing low effort to nearly no effort, preventing people with needs from networking with creators.... Uhh.. but I think that's a very general issue with Gen AI away from the corporate/entrepreneurial dev space
For 2) SUNO's stem separation is pretty good. The open sources ones (like spleeter) are also not bad, but they are pretty hit or miss.
For 3) This is a really cool idea. The deep dream images produced the fur and eyeball textures because they used imagenet data which had a bunch of dogs and other animals in it. The trick would be finding a good data set to use for this. May be there is a public source of animal sounds floating around you could try this with.
And yeah think people 100% generating songs is pretty wack, but I could see why it would be fun to someone who is not a musician.
I love these one-liner explanations - the absolute minimum information one needs to move forward. Great writing.
Did you publish the actual trained model anywhere? I see how in the code there is python for how individual samples can be generated, but the model checkpoint pulldown comes from a directory that... I don't see.
I then went through the code of how this runs on the web and- I'm not a web dev guy- so I'm pretty confused at all the bits bolted together to make this into a web app. It seems like there is a WASM bit that is compiled from a typical C++ audio plugin that is doing the stuff like conv reverb and limiting and distortion in the web app - all that is oldschool, non-generative AI, DSP being applied to the samples. Then the samples are just... a few default generated samples, to start- where are they pulled from, physically? And you have a login requirement to spool up the actual generative AI part to generate new samples to run into the DSP (because that needs a GPU on the backend to do, so, a login to help rate limit this)
How big is the actual generative model? Did you ever think about building the generation engine into the WASM bundle, using maybe WebGPU in the WASM to accelerate in a platform agnostic way, so that the entire app would run offline in someone's browser window?
I'm having fun just playing with the kick program without a login, which, again, am I right in saying in that mode there is no gen AI of sampled happening server side, it is just playing starting with some pre-made samples?
It was a lot of work to get a good DSP to work on the web hahah. Yes, I am writing the DSP in C++ and compiling to WASM. Im using multithreading so the audio work is done in the AudioWorklet while the UI is run in the main thread. I was thinking of writing another article on this because it's pretty interesting and a bit complex.
I sound designed some of the kicks myself and some of them are from sample packs. I just renamed them all to have German names.
If I wanted the model to run completely in the frontend I would need the user to download 300MB of weights and it would probably be tricky making sure it works on everyone's hardware. So I though about this but it didn't seem like the best option. I'm pretty sure it is possible though.
And yes I put rate limiting so no one goes crazy on my credits.
Im glad you're enjoying it!
This is the hot topic in AI ethics.
Is it just learning the same way that a human learns from hearing songs on the radio and playing them, or is it just a compression algorithm? There is not a clear answer here but it seems like the legal system so far is letting things slide and agreeing with the former argument.
(This can also be applied to other areas like open AI reading the NYT)
I was looking into a way around this and I don't think it exists without building out an entire phone app.
Looks like this has been possible since iOS 17 but is not well documented.