This was initially a project I had started to get better acquainted with the Web Audio API. I had done some preliminary work with the Media Source Extensions API and piping output via to the Web Audio API, but I wanted to see how well the scheduling capabilities were. Throughout the effort, I learned quite a bit about signal processing and FFT.
The current process is rather simple: I utilize M3U8 for storing segment information, and manually parse the file when a play request is received. I grab the first few (Opus-encoded) segments, pass them through an OfflineContext to maintain a specific sample rate, and store the resulting buffer in a wrapper object. Each object is stored in an array, and is responsible for re-creating the AudioNode when stopping or starting.
The hardest part about this (for me at least) was fetching and scheduling accurately while the player is running. I group segments requests together, and schedule after each group has finished downloading. It's not perfect, as I occasionally get pops between each segment: Chrome seems to be less noticeable than Firefox in this regard. I schedule based on the buffer duration, but there seems to be subtle differences between browsers.
In any case, I've had quite a bit of fun working on this project.
Looks great, also thanks for giving technical specifics!
Could you elaborate a bit more on what the site does, perhaps in an ELI5 way (explain it like I'm 5 years old)? It's not entirely obvious at a glance, i.e is it like SoundCloud?/How does it differ?/Can it be used to create music?
Sure. The site is billed as a delivery service, thus you're able to upload music (that you have the rights to) and monetize it. You could, of course, opt to give away your content for free; there's no requirement to login or perform an action to download a free tack. The site does allow remixes and mixes, but they have their own rules.
In terms of interaction, it's similar to SoundCloud. I've been using SoundCloud for years - just right after they transitioned to their V2 site. I've also used Bandcamp for selling my own music, and always felt their approach was transparent and fair when it came to monetization. When I started this project, a little bit of both went into the design and structure.
I like the look & feel and the technology seems to work just fine.
But I don't really understand what the texts want to tell me.
- Is this for people who like to listen to music (a music player)?
- Is it for people who like to create music where they can upload and sell it (like a store)?
Somehow I have a feeling it is both, but 'music delivery service' feels to me like an abstract term (what is the difference to 'website for sharing music'). Might be worth iterating over those parts again.
I'd label it as a music player and a store. You, as a listener, can browse the site and purchase music to add to your collection. Artists are listeners to, so they can do the same thing, but they also can upload their own music and monetize it. Aside from that, it has the basic functionalities you would expect from a site like this: favoriting tracks, creating sets, commenting - the typical stuff.
Honestly, I wasn't sure how to label it myself: I'm terrible with describing things. I'll be sure to give it some thought though.
Looks great! You've offered several simple descriptions in these comments that don't appear to be on the site. It would be good if you included some of this language on the landing page so the purpose is immediately obvious...
Unfortunately, Safari does not support Opus-encoded audio using the Web Audio API. I actually had to push a hotfix earlier this week since Safari uses webkitAudioContext as opposed to AudioContext (which I wasn't checking for) - the site wouldn't even load.
I've recently been doing some client-side media recording, and ... could not get chrome to record in anything BUT opus for days. Didn't want Opus because... FFmpeg couldn't deal with it (nor can aws media converter, or other cloud-based converter services). opus may have some benefits, but it doesn't seem to be well supported in various tools.
Can you say something about the specifics? Like – can I use it like a locker service similar to Google Play Music? If so, how many tracks can I upload?
You have a Collection, which is where music you've purchased resides. If an artist decides to remove their track, but you purchased it, it's "soft-deleted" - the artist themselves cannot access the track, but you still can.
At the moment, the upload limit is 15 tracks a month. Since this is a subscription-less service, there's no way to subscribe to a plan for unlimited uploading. We can always override the limit on a per user basis, depending on your needs.
The backend is pure Java, and is composed of four separate microservices. The web server (we use Javalin) is nothing special; the transcoder works as follows:
* An POST request is sent to the web server. If the file is valid, it is moved via FTP to a local server. I then create a message for RabbitMQ to add to a transcode queue.
* A worker listening on the channel gets the job, and does all the needed work to make the file "streamable", e.g. segmenting and encoding file parts, creating the M3U8, generating the waveform, storing to S3, etc. Throughout the process, progress is saved in Redis, which is made available to the client for visual feedback
* Once the process is done, an identifier is generated, which the client refers to when saving any modifications to the track information. We primarily use MariaDB.
The transcode process is the most complex piece of the puzzle, of course. Everything else is pretty much CRUD operations.
> all the needed work to make the file "streamable", e.g. segmenting and encoding file parts
Pardon the probably stupid question, but is that (splitting into several files) really necessary? Isn't supporting HTTP range requests enough?
Or is it because clients tend to download the whole file, even when the listener is still at the very beginning and might skip, meaning wasted bandwidth?
Not a stupid question at all! I actually looked into byte-range requests as opposed to downloading individual segments, indeed Amazon S3 does support byte-range requests. I would think it's similar to implement, and wouldn't have as many moving parts.
When I had started the project, I saw how other streaming sites fetched their audio, and I saw a pattern in individual GET requests for segments. I thought, "Hey, they're doing this for a reason, and even though I don't fully understand this reason, maybe I should do it to." At the time, I think I justified it by thinking that someone who wants to rip the stream file would have to piece the track together versus just requesting the full track.
In the end, it really doesn't matter: anyone who is willed enough can get whatever is being sent to their client. It's one of those moments where I didn't bother to really think it through and factor in the practicality aspect. I'd like to go back and re-visit byte-range requests, though, just to see how it would work differently.
MPEG-DASH and HLS both use segments. I think one reason is to allow for dynamic quality switching: if the bandwidth drops, the player can simply start pulling segments encoded at a lower bitrate. This is harder to do with byte-range, since bytes don't map to time cleanly.
As someone else who has built streaming things, HTTP range requests aren't enough all of the time.
Quite a number of older Android clients, like TVs, cheap phones, etc. don't have an up-to-date browser on them, and they will make the first range-request correctly (with a byte range that reflects the browser knows the full content-length, etc)...
But you'll have to build your own scheduling system to pull in further requests, which is painful, clumsy and doesn't always work.
This is doable. If implemented, what would you like to see? I imagine whatever the notification system would show, e.g. uploads from other users, comments, etc.
The current process is rather simple: I utilize M3U8 for storing segment information, and manually parse the file when a play request is received. I grab the first few (Opus-encoded) segments, pass them through an OfflineContext to maintain a specific sample rate, and store the resulting buffer in a wrapper object. Each object is stored in an array, and is responsible for re-creating the AudioNode when stopping or starting.
The hardest part about this (for me at least) was fetching and scheduling accurately while the player is running. I group segments requests together, and schedule after each group has finished downloading. It's not perfect, as I occasionally get pops between each segment: Chrome seems to be less noticeable than Firefox in this regard. I schedule based on the buffer duration, but there seems to be subtle differences between browsers.
In any case, I've had quite a bit of fun working on this project.