[Tokkemon]

I work for Sibelius so I'm heavily involved in this world. MusicXML is a great standard and offered a solid basis for data interchange between music notation programs. But now there's a new group working to build a successor standard, MNX: https://w3c.github.io/mnx/docs/

It was originally going to be in XML but they recently switched to JSON, which is a good move, I think. I can't wait for it to be adopted as it will give so much more richness to the data set.

[TedDoesntTalk]

Hopefully they will use JSON5 so comments can be included. The loss of comments when switching from XML to JSON was a disaster in other domains.

[usrusr]

I'm all for JSON-with-comments, and I even have a bit of a soft spot for the idea of JSON-with-expressions, but considering how far I expect this one to be on the "interchange" side of the spectrum between interchange formats and authoring formats, I doubt that comments will be missed a lot.

Certainly less, less by several orders of magnitude, than in the depressingly ubiquitous JSON-as-configuration use case...

[microtherion]

Another concern with comments is that apps might try to (ab)use them to store app specific information, which hurts the goal of an interchange format.

[throw0101b]

> Another concern with comments is that apps might try to (ab)use them […]

There's no "might" about. This is exactly what happened in the early/beta days of JSON, per Douglas Crockford, creator of JSON:

> I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I know that the lack of comments makes some people sad, but it shouldn't.

* https://web.archive.org/web/20190112173904/https://plus.goog...

* https://archive.today/2015.07.04-102718/https://plus.google....

[coldtea]

There is a big but though: whether the loss of interoperability for those cases was any real loss. Those people wanted their own files parsed a specific way by their own infrastructure. Their files already weren't meant for every regular JSON parser: if they wanted wider interoperability they'd had dropped those comments.

And nobody stops them from continuing to use those meta-parsing semantics even if JSON-standards prohibits comments.

I think Crockford was wrong in his objection. It shouldn't have been his concern.

[NoMoreNicksLeft]

"Oh no! Bad actors might abuse this absolutely necessary feature, so it's better that we leave it out and let everyone suffer to avoid that! You'll just have to cope!"

[throw0101b]

> "Oh no! Bad actors might abuse this […]

There's no "might" about. This is exactly what happened in the early/beta days of JSON, per Douglas Crockford, creator of JSON:

> I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I know that the lack of comments makes some people sad, but it shouldn't.

* https://web.archive.org/web/20190112173904/https://plus.goog...

* https://archive.today/2015.07.04-102718/https://plus.google....

> […] absolutely necessary feature […]

It's so "absolutely necessary" that JSON has found hardly any success and is struggling to find a use case or a niche… /s

Or it seems that JSON works just fine without comments, especially as a data exchange format, which contradicts that claim that it is "necessary" (absolutely or otherwise).

[NoMoreNicksLeft]

> Or it seems that JSON works just fine without comments

Code that can't be annotated might as well be machine code. And guess what? Json largely is just that.

> It's so "absolutely necessary" that JSON has found hardly any success

I am almost 50, and since as early as I can remember, the worst technology always wins when there is some sort of standoff. That json succeeded doesn't mean anything, other than that it was the worst.

I'm old enough to understand why this is usually true... the best technology is usually more expensive, and economic forces favor the cheapest. Json though? Best I can figure is that imbeciles just got used to picking the worst, even when there was no expense tradeoff.

Snarking about how it won... I guess that means you think it was perfect on its first try? When the fuck has that ever happened with software?

You're just plainly wrong, and I don't know why or how you'd bother to be. Do you have a few hundred million in JSON Inc. shares about to IPO and somehow I missed it?

[microtherion]

I've missed comments in JSON myself, when writing stuff like configuration files. But interchange formats are a different beast, not really meant to be read by humans. And if they need comments to be comprehensible, one might wonder whether the scheme chosen was insufficiently self documenting in the first place.

[vilunov]

They might do it even without comments by adding unused fields.

[usrusr]

Which, if dropped, are a clear change to the document in question. Whereas a dropped comment would have to be considered equal, even if the "special comment aware software" might not agree at all. I'm not happy with the decision, but I can't refuse the argument some merit.

[adrianh]

Hi, I'm the person running the MNX project! Contributions welcome — we're making good progress and the new format is going to be really fantastic.

[im3w1l]

As different standards proliferate it makes me worried that we will have less interoperability not more, as programs only have limited resources and pick one format they focus on supporting and perhaps half-assing another. (How) do you plan to adress this? Are there plans for some library for dealing with these files to help adoption? What will be the best way for users to convert from MusicXML to MNX and back? Is losslessly roundtripping MusicXML an explicit goal? (I assume losslessly roundtripping MNX will not be possible in general as you intend to add new features to MNX that MusicXML doesn't have and will never have).

[adrianh]

Yes, I've already begun work on an open-source utility that converts from MusicXML to MNX: https://github.com/w3c/mnxconverter

Eventually it'll work both ways. I'm hoping this is a big help for adoption, as gives you two formats for the price of one (just write an importer for MNX, and you can get MusicXML import support "for free" if you use the library).

[ronyeh]

Awesome project (as always) Adrian!

Will MNX allow for inline comments?

I don’t see any comments on the examples page:

https://w3c.github.io/mnx/docs/mnx-reference/examples/

I know JSON doesn’t have comments, but JS and JSON5 allow for comments. It would be super nice to allow for comments because you can hand annotate sections of the MNX file for the purposes of teaching.

[adrianh]

Thanks! We're not planning to support inline comments at this time; this was a tradeoff we knew we'd have to make when we decided to use JSON.

Given the choice between supporting comments and supporting a wider variety of implementations/libraries ("plain" JSON as opposed to a comments-supporting variant), I think the latter is a more practical priority.

With that said, we'd like to add a standard way to add vendor-specific information to an MNX document — which is definitely a must-have, for applications that will use MNX as a native format — and I could see a comments-ish thing appearing in that form.

Regarding that examples page, I'm actually planning to do something along those lines anyway. The MusicXML docs and the MNX docs use the same system (a Django app), and the MusicXML part uses a custom XML tag to define "this part of the XML example should be highlighted in blue" (example: https://w3c.github.io/musicxml/musicxml-reference/examples/a...). It's on my to-do list to implement the same thing for the JSON version — which is essentially like inline comments(ish), if you squint.

[ronyeh]

Thanks! I’ll definitely follow the MNX project. Seems exciting.

[antoinebalaine]

Looking forward to discovering this standard. After 2 years working on parsing ABC, I realize how difficult it is to represent notation. Kudos on this effort!

[microtherion]

I've written two music apps that use MusicXML as their native representation (https://woodshed.in is the newer one), so I've been involved in this world as well.

MusicXML is a great effort to tackle a very difficult problem, but some of the details can get rather hairy (e.g. having to represent many concepts twice, once for the visual aspect, and once for the performance aspect; or how exactly to express incomplete slurs). Interoperability in practice seems to be fairly limited (Possibly because for many music programs, MusicXML import/export is an afterthought).

One of the biggest contributions a new standard could make is to provide as complete a test suite as possible of various musical concepts (and their corner cases!) and their canonical representation. It looks like MNX has already made good efforts in this direction.

[Exoristos]

> they recently switched to JSON

Considering this data is machine-generated and machine-ingested, moving away from XML seems like a big step down.

[bastawhiz]

In what way is JSON a step down versus XML? Frankly I get nervous and sweaty every time I need to deal with XML, because of the inherent performance/security issues it brings into my codebase.

[Exoristos]

XML is much more precise and much more flexible. It also benefits from much more powerful and mature tooling. The few comparative downsides it has include verboseness, which doesn't matter to machines, and that younger devs don't know how to work with it, which again shouldn't be much of an issue in this use case.

[feoren]

XML is less precise, because it's more flexible. Powerful and mature tooling only matters to people creating and editing XML, not computers. XML Schemas are there to support human editors, not computers. Verboseness means larger files and longer processing times, which does matter to computers; the verbosity is explicitly and only there for human readers and editors. JSON is a much better format for something humans only occasionally look at.

[Exoristos]

You're wrong in almost every point. Flexible definition of data allows for greater precision of data structure. Something like XSLT isn't for human interaction; it's for super-powerful machine transformations. I'll grant XML files are larger, but not enough to make much difference when opening a music notation file. Any time a dev pushes back against XML, it's been my experience they are uneducated on the subject.

[slaymaker1907]

The reason why you don’t need some big, formal JSON schema (though they do exist) is because you can notate most of the constraints people care about in TypeScript. It’s just a bunch of structs, arrays, string enums, etc. XML doesn’t really have a nice mapping to type systems like that so it needs schemas.

My coup de grace against XML is that it is wholly unsuitable for serializing arbitrary strings in most programming languages. It’s defeated by quite simple strings like “hello\0world”. You can’t just escape the null using &#; because the standard, in its infinite wisdom, forbids it. Instead, you’re just expected to come up with some completely non-standard way like <null-char /> or just interpret “\0” specially in your application code. Meanwhile, JSON just lets you put pretty much whatever Unicode you want into a string with a standard way of escaping characters like the double quote.

[feoren]

> Flexible definition of data allows for greater precision of data structure.

Other way around. If there are 3 different ways of doing something, the data structure is less precise. Say you have an object with a Name as a string. You know exactly what this is going to look like in JSON:

    {
        "name": "Whizzbang"
    }

How is it going to look in XML? It might look like this:

    <Foo name="Whizzbang" />

It might look like this:

    <Foo>
        <Name>Whizzbang</Name>
    </Foo>

It might look like this:

    <Foo>
        <Name value="Whizzbang" />
    </Foo>

XML is less precise because it's more flexible. "But you just define an XML Schema to disambiguate" -- so now you're doing more work in a separate file that you have to publish and link in just to solve a problem that JSON doesn't have at all.

> Something like XSLT isn't for human interaction; it's for super-powerful machine transformations.

It's only "super-powerful" considering XSLT as a markup language. Considering it as a programming language, it rather sucks. If you're writing ETL scripts to transform data around, use an actual programming language. The fact that XSLT is Turing Complete only drives home the point that it's not a "powerful markup language", it's "poorly-designed programming language". Sure, if you're literally only transforming the exact same data from one XML schema to another, as some sort of adapter step, then XSLT beats general-purpose languages; but you're never just doing that, are you? You're linking in other data sources, validating things, sending things over the wire, etc. You already need the code to save and load your XML into a format you prefer in memory; just use that format for these tasks.

> Any time a dev pushes back against XML, it's been my experience they are uneducated on the subject.

Many people who push back against XML are not uneducated, but rather jaded on it, having worked with ambiguous formats, buggy schemas, and 4000-line long configuration behemoths that should have just been code. You can use XML parsimoniously, but there's not much overlap between the people doing that and the people who love XSLT.

[ctxc]

My experience has been similar to the immediate parent although I think xml looks better than json here. All fun and games till you forget to turn transforms and the 10 other knobs in your parser off.

The dev will be like "okay what are transforms though?" as they watch new calculator.exe instances popping up on the screen.

[bastawhiz]

> It also benefits from much more powerful and mature tooling.

JSON is over two decades old. XML is only five years older. Both have powerful and mature tooling; I'm not sure how you can suggest otherwise.

[account-5]

Never heard of either before but having looked at the comparison [0] I think I prefer the XML version.

[0] https://w3c.github.io/mnx/docs/comparisons/musicxml/

[poulpy123]

I'm with you. I usually prefer JSON to XML but in the example the XML is more readable

[jefftk]

What do you like better about the XML version?

[account-5]

It's computer readable and (with highlighting) human readable. The JSON brackets are visually jarring. I feel that the XML is almost self documenting whereas I'd probably need the schema to understand the JSON.

I like JSON for data transfer but for describing documents XML is decent.

[gnulinux]

How about MEI?

https://music-encoding.org/about/

This is what MuseScore 4 will soon start using.

[Kye]

How well does MusicXML (and MNX) represent the full range of notation? It seems like an exceptionally hard problem.

Related: Can it handle non-Western notations?

[Tokkemon]

MusicXML only works with conventional western notation, but MNX is attempting to expand that significantly. How far they will get with that, I don't know.

[odyssey7]

How would you explain the relationship between MNX and SMuFL?

[adrianh]

They're totally different things, though the standards are maintained by the same people.

SMuFL is a font layout specification. It solves the longtime problem of "I'm making a music font. Which Unicode code glyph should I use for a treble clef?" For many years, this was a Wild West situation, and it wasn't possible to swap music fonts because they defined their glyphs in inconsistent ways. This problem is basically solved now, thanks to SMuFL.

MNX is a way of encoding the music itself. It solves the problem of "I have some music notation I want to encode in a semantic format, so it can be analyzed/displayed/exported/imported/etc."

[odyssey7]

Thanks. If it isn't too many questions, are any layout concerns encodable in MNX, or is it scoped to semantic information only?

[adrianh]

Yes, the goal is to allow people to encode layout information optionally — as a separate layer from the semantic information.

One particularly cool thing is the ability for a single document to have multiple layouts (!), which is useful for parts vs. conductor scores. See here for an example: https://w3c.github.io/mnx/docs/mnx-reference/examples/multip...

[odyssey7]

Much appreciated! This is excellent work.