A little piece of me died today when I read of the existence of XRNS, a music tracker format that uses XML. A music tracker format that uses XML! Can you imagine? If you can’t, Google for “filetype:xrns” to find plenty of samples.
[xml]
< ?xml version="1.0" encoding="UTF-8"?>
…
…
[/xml]
And on it goes. It’s difficult to articulate why this feels so heretical. It’s like those old MOD/tracker formats were designed to be so pure, so efficient. This completely destroys that. Now your playback engine has to carry the baggage of a full XML parsing library.
There are elements of the FFmpeg development team that would enjoy seeing the program grow to be able to handle all the various tracker-type formats (myself included, obviously). It’s not going to be pretty when XRNS collides with FFmpeg.
Addendum: Share the love over on the Renoise forums.
Luckily there are already quite a few libraries that do that, but I do have my share of shivers because of this.
On the other hand, one can say that at least some sort of XML parsing is already needed by msot audio players, to be able to handle the most modern playlists (which I still think are mostly pointless), and some not-XML parser for Microsoft ASX playlists is needed, too.
On the whole, the best thing is that I haven’t noticed yet any software trying to stuff XML inside ID3 tags or Vorbis comments…
XML as an *interchange* format, sure. As a primary file format? Someone’s been drinking the wrong kool-aid.
MONOTONE, for reference, stores frequency/note/effects into 2 bytes. This is partially due to how limited MONOTONE is at the moment, but I mention it as an example that is at least 100 times smaller than the XML snippet you published.
My favorite 78k .xm expands out to 1720k of XML…
@Peter: And that’s just the pattern data and instrument metadata, right? I suspect that doesn’t count the audio samples.
Seriously, who gives a shit what format it’s in? XML or HTML, fuck it. If it works, let it be.
No reason to cry, everything will be just fine.
Human readable is the best way to store projects.
Maybe you never use a creative software, to see why we need that.
Let me quickly preface my comments here by saying that I have been an avid lover of trackers since I was a child, when I first saw SoundTracker on the Amiga in… 1987? Since then, I have used trackers exclusively my entire life, and could not imagine creating music any other way. I can definitely appreciate the history behind trackers, and the struggles programmers have gone through to get music playing in realtime with the very limited resources (cpu, memory, etc) available to them.
Now, onto the subject of Renoise…
Renoise is a professional-grade music production tool that just happens to be based around a tracker interface. That’s where the similarity between it and older formats such as XM/IT/MOD ends. It is not just another run of the mill, sample-only tracker format with a limited range of basic pattern effects; it is a modern DAW with a wide range of advanced features, aimed at people who are creating music for release on CD, Vinyl, MP3, etc. It has never been marketed as something that is suitable for inclusion in 3rd party realtime playback routines, such as those found in demoscene productions or video games, and there are numerous factors besides its XML-based file format that make it quite unsuitable for that task, such as the CPU-intensive native DSP filters and effects, meta devices, signal routing, automation, etc., all of which would have to be perfectly replicated by any 3rd party player routines (even though the source code for all of these native features is not even publically available), not to mention the ability to host any VST/AU/Lapsda plug-ins the composer might have used. If you strip away all of these things and use XRNS simply as a container for notes and sample data, then really what is the point of using XRNS at all?
It’s true that XML is far from an efficient file format if your main concerns are filesize and the speed at which data can be accessed in realtime – obviously very valid concerns in the older days when CPUs were slow, and memory/storage was very limited, but… this is 2009, not 1987. The primary concerns here are to have a format which is open, easy to maintain, easy to parse, and easy for the community to explore themselves, so in this case XML is *perfect*. It allows the Renoise community to quickly create their own unique tools and utilities to enhance the overall functionality of the application. There have already been quite a few nice 3rd party tools created to manipulate Renoise song data in interesting ways (which the actual Renoise team may not have time for, or even care about in the first place), and many of those tools were created by people who do not necessarily have a great deal of programming experience. If these same users had to deal with the complexities of a non-human-readable, purely binary file format, then it would be a completely different situation.
In conclusion, XRNS is a format that is really only intended to be played back by Renoise itself, and not by any 3rd party player routines. While it would technically be possible for someone to write a player which read the XRNS format and interpreted the data, the other more advanced features of the format make it quite unsuitable for this purpose. When dealing with realtime player routines used in demos or games, formats such as FastTracker .XM, Impulse Tracker .IT, or even the humble .MOD, will always be preferable, and in cases where music with a higher production value is required, then .MP3 or .OGG is used.
amen, dblue!
Kieran / dblue = PWN.
Verbosity != PWNage. Try this instead: “If you think XML is the answer, you definitely misunderstood the question.”
During my experience as a programming engineer, I’ve had enough experience to say that XML plain sucks. Or more precisely: the amount of XML disadvantages very much outweight all its positive aspects.
XML is heavily overused and widely misused. Mostly just to have it, because it’s so popular. But, I guess, this isn’t anti-XML debate.
Back to Renoise. Well, it loads and saves things pretty fast. XML is zipped, most of file size is consumed by sample data. A huge waste of RAM, but completely transparent to the end user.
Keeping in mind the nature of Renoise DAW (as already presented by dblue), I don’t think that XRNS should be a matter of concern to anybody.
The only opposition to dblue – “this is 2009, not 1987”.
Are you familiar with Wirth’s Law then?
@crc: Yes, I’m vaguely familiar with Wirth’s Law, and in fact I tend to agree with it on most levels; there is far too much bloat in today’s software. I think if operating systems were developed exclusively by a bunch of l33t teenage demo coders from Scandinavia, we’d probably be squeezing a hell of a lot more performance from our CPUs these days :) It’s true that Renoise is a bit resource hungry, but despite that I still think that it’s doing an incredible job, all things considered. At the moment, I’m using a 1GHz Pentium M laptop from 2003 with 1GB RAM, and although I can’t use a lot of really heavy/intensive plugins, this machine is still able to handle a surprising amount when using Renoise. I think the extra overhead required to handle XML is neglible, compared to all the other intensive processes that are required in an application like this.
Whoops.. I meant to say “@crs”. I do apologise.
A piece of me died today when I read some blog tard rewrote history as if MOD/XM/IT were open formats instead of what they actually are, the reverse engineering of closed binaries that still do not accurately playback on programs that are not the origina of the source of the material.
Naturally, the best the XML fans can do is call names.
If windows networking is the answer, then SAMBA must be the solution.
Don’t ever change.
A separate player for renoise songs is a waste of time. The problem is VSTs. You can’t assume the machine playing has hundreds of dollars worth of VSTs installed just to play .xrns’s.
I would also add it’s not going to be pretty when the Apple Garageband format collides with FFmpeg. [???]
The XML will be least of your concerns when writing XRNS playback library. XML to C structs parser would take currently less than 10 minutes to write: Open schema and press “generate”…
Let’s even say the external plugins are not the problem and the resulting library will only play back songs without external plugins. Try implementing all those Renoise plugins now with sample precise quality: Phaser, Flanger, Delay, multiple types of reverbs, myriad of filters several distortion algos and lot more. Not to mention advanced routing capabilities and modulation chains.
Renoise was never intended for lightweight dynamic music format. It is the project file for Renoise DAW and intended to be open and interchangeable. For tools to extract data from the format or generate renoise format files.
The XML overhead is hype. The parsing of XML format may take a bit of time compared to direct binary formats, but that doesn’t add overall overhead. The XML tree is parsed into binary structures inside the renoise. It’s not a live streaming format, so whatever the project format looks like, it has no influence on the playback speed or efficiency of the song itself.
Oh, BTW, did you know the FOSS flagship audio product Ardour also uses XML in their product files? Let’s start a blog post about this aswell?
Hey, nice troll, you almost got me with it and I was actually about to take it very seriously, come and insult you about how much you miss the point and must be a poor creative mind who just fail at understanding the use of a program: making music.
But no seriously, very nice troll, you made me laught.
quoting Suva:
“The parsing of XML format may take a bit of time compared to direct binary formats, but that doesn’t add overall overhead. The XML tree is parsed into binary structures inside the renoise”
I find strange that noone has pointed this aspect out before in the comments; clearly an XML-based format would not be the best choice for a 64k intro, but once you have turned it into a binary format (which you could also do by yourself in your demomaking tools set), then there are really no differences. Moreover, XML format allows for fast and reliable data verification on saving, so you are always sure that the data have been saved correctly. Renoise does conversions from and to binary upon saving and loading, so performances are (ridicolously) smaller only at these points. This of course is all apart from the other aspects that have depicted above due to which Renoise is completely useless for demo writing (unless you use render to MP3/OGG of course).
Anyway, since Renoise is sadly almost the ONLY tracker that survived these days, either XML was a right choice or anyone cares about that choice anyway..
>> Now your playback engine has to carry the baggage of a full XML parsing library.
For example the “ezXML” project, ~45k of C code compile into ~27k. That should be small enough for most projects if needed.
Now let’s think further what might be needed for a player…
A ZIP decompression library, GZIP decompression (might be handled by the ZIP library maybe), FLAC decompression, OGG decompression and MP3 decompression just to name the most popular ones, which a player should expect as samples to decode inside a XRNS module. Now where is the most overhead?
Also a typical song project from me is around 10mb to 30mb using FLAC compression for the samples. The added tiny amount of the compressed XML is really not worth to mention. To me it seems pretty efficient.
The only thing one might complain maybe is, that the XML structure itself isn’t optimized that well and contains a lot of redundancy. Lot’s of double entries and few usages of attributes. On the other hand it’s pretty simple to parse because of that.
There are even ways to get the size of the XML smuch smaller if desired: use instrument names with 1 or 2 char length, delete song comment and tags, then extract XML and remove leading and trailing whitespaces, remove linebreaks and recompress it with a ZIP utility using maximal compression.
That’s like claiming that PhotoShop or Flash are bad because its work formats aren’t optimized for web browsers =_= . Renoise is not a old-school tracker, it’s a modern (and better) music-making program only with old trackers look. Why don’t you blame Cubase or Fruityloops or any other music-making program?
@xeche: I agree with your comment; an FFmpeg tracker therefore must also provide its own implementation of popular VSTs.
“It’s difficult to articulate why this feels so heretical.”
No need to, I understand perfectly: A slow news day, this seemed like a good opportunity to seem smart, it backfired horribly. Case closed. ^^
[quote]
It’s like those old MOD/tracker formats were designed to be so pure, so efficient.
[/quote]
Their file formats were not designed to be “pure” or “efficient” as much as small in footprint. Perhaps we should be transmitting HTML in a binary format across the web as well? People cried for years for an open tracker format that was easy to parse, and now that it’s here, all you’ve got to do is complain?
If you think .XM is the answer, you definitely misunderstood the question.
I also love how you discredit dblue’s post simply on the basis that it’s somewhat verbose. If a “reverse engineer” like you can’t see the meaning in what he’s saying, I’m fairly sure you should look into a new career.
[quote]
Naturally, the best the XML fans can do is call names.
[/quote]
Naturally, the best the pretentious blogger can do is overgeneralize about people.
I think that if the FFmpeg team desires to have an own binary format for Renoise, then i would suggest them to indulge themselves with that luxury. I personally have no idea for what reason FFmpeg should ever support any mod format actually.
Lots of tracker formats were not open, but just closed source, so lots of formats are hacked or reversed engineered to get supported. As mentioned elsewhere:the old formats were put up like this just because of lots of limiting reasons; not for pureness or effectiveness but because coders back then had not much other choice. If most of those limits weren’t there, we could have a lot of more sophisticated trackers back then when they saw the light.
Renoise is about progress and moving forward (no matter how slow and how long some feature upgrades take to get implemented) for that you can expect any current old fashioned tracker-method is being improved or replaced for a more professional method to use it and give the tracker the dignity and status that it deserves: a tool that can be used professionally to write music with.
You made your point but i guess but I think this blog is not really advocating the positions that you have tried to argue…. Better luck next time ;)
I see the challenge and predict FFMPEG will support XRNS and emulate 1000 windows VST plugins within 6 months.
Yes, they are that good! Get to it team. You can do this!!! This blog and it’s in-depth understanding of the contemporary MOD scene proves it.
a code critic may actually achieve less relevance than a music critic.
christ sake, worthless blog entry.
@xml apologists: It seems to me you are all saying that because modern computers are 1000 times faster than old ones, we should make them spend 99.9% of their time processing XML, and use the remaining 0.1% to do what we already did with the old, slow computers, but without a hint of the clever hacks of days now long gone. Imagine what today’s computers could do, were the software as cleverly written as it once used to be. Imagine all those gigahertz doing something more creative than converting numbers from binary to ascii and back, over and over again. Imagine all those gigabytes storing actual, useful information instead of endless sequences of angle-brackets and quote marks. Imagine no exceptions, it’s easy if you try…
dear XML skeptic,
noone here is making an XML apology at all. If you knew what Renoise is all about, probably you would not have written the above.. First of all, as said by lots of people above, Renoise converts the XML into binary at song loading stage, and this usually take a couple of seconds or less, then it’s all about loading hundreds of binary samples (OGG/MP3/FLAC/WAV are supported) and/or VST plugins, which can load samples of their own.
Once the song has been loaded, XML can be forgotten until Renoise saves the song, which will take more or less the same amount of time it took to load.
In the middle, there are tons of DSP, sound mixing, automations and other stuff the “pure”, “clever” and “optimized” Pascal-written Fast Tracker II was not able to do.
we know that XML isn’t the fastest way to treat data, but thanks to this choice, people with only PHP kwnoledge in programing has done scripts to manage XRNS data. Do you authorize people who are musician to use PHP in order to make their own scripts to manipulate XRNS? Please? Can we? Really? Thanks!
if you can’t understand what I believe has been stated quite clearly in this comment, well, I can’t say it more clear than this, I’m sorry. Try downloading Renoise and its beginner’s tutorials and see it for yourself.
Surely a light weight database format would have been more appropriate then XML+ZIP…
..except you could not easily copy=>manipulate=>paste data from/to external application (f.e. Notepad, another Renoise instance) as easily as you do now
So was it a lie that xml is only used as a static storage format?
Wow, question the XML orthodoxy and this is what you get.
Still wondering what possesses a developer to think XML is a good idea. I know it’s eXtensible and all…
[quote]
@xml apologists: It seems to me you are all saying that because modern computers are 1000 times faster than old ones, we should make them spend 99.9% of their time processing XML, and use the remaining 0.1% to do what we already did with the old, slow computers, but without a hint of the clever hacks of days now long gone.
[/quote]
I’m not sure where you got your presumption that parsing xml takes a significant amount of processing power, but it’s just flat out wrong. I think you should back up your accusations with some proof, before you continue making an ass out of yourself.
@BYTE-Smasher: Please read Joel Spolsky’s Back to Basics, particularly the part about XML databases.
[quote]
So was it a lie that xml is only used as a static storage format?
[/quote]
XML is used for any process that does not require severe CPU usages. Copy and paste actions are those, saving / loading are those and perhaps even advanced edit trickery are those.
If you copy and paste wav-data in the sample editor to paste it in an external editor, you don’t copy XML data.
Any internal audio processing is done binary.
[quote]
@BYTE-Smasher: Please read Joel Spolsky’s Back to Basics, particularly the part about XML databases.
[/quote]
The whole debate is out of concept in this sense because Spolsky’s idea targets a complete different area.
You can’t compare the Renoise format to relational database frameworks that has to provide for epic amounts of data!
Yes for very large databases it is not smart to use XML pure for indexing problems alone but as well when you love your harddrive not to get too fragmented, please don’t do so.
A Renoise songfile is not a large database, there is not some engine constantly requireing to seek, write and read keys from it.
[quote]
What this means to me is that you can’t use XML if you need performance and have lots of data. If you have a little bit of data, or if what you’re doing doesn’t have to be fast, XML is a fine format. And if you really want the best of both worlds, you have to come up with a way to store metadata next to your XML, something like Pascal strings’ byte count, which give you hints about where things are in the file so that you don’t have to parse and scan for them. But of course then you can’t use text editors to edit the file because that messes up the metadata, so it’s not really XML anymore.
[/quote]
So, according to Joel, XML is a poor choice if you have 2 conditions coinciding: Large amounts of data and a need for speed. As XRNS’s song.xml files don’t have anything close to a “large” amount of data, XML is more than efficient enough. As Renoise only needs to access the XML data once per Tick, it really doesn’t consume that many cycles at all.
[quote]
As Renoise only needs to access the XML data once per Tick, it really doesn’t consume that many cycles at all.
[/quote]
Renoise does not process XML data during songplay / rendering.
Binary is translated to XML when copied to clipboard and when pasted, translated back to binary.
I just love it when the fanboys fail to agree with each other about the precise nature of their object of worship.
No need to, I understand perfectly: A slow news day, this seemed like a good opportunity to seem smart, it backfired horribly. Case closed. ^^
Considering all the fuss it’s caused, I’m not sure what backfired on who here.
LOOOOOOOOOL is all I can say about all the comments. Really. Do you guys now feel better after attacking Mike like this? Did your poor repressed souls get some sort of relief now?
Again, LOL. Because all you guys did is show your arrogance. And how easy it is to strike the right nerve with you.
People making good music don’t care what format the data is saved in.
Gusar, except for some posts in which an unkind laguage has been used, I can’t see much arrogance in the replies; at least, I don’t see as much arrogance as there is in the initial post, really.
Anyway, the so called “fanboys” (some of whom are part of the Renoise developing team and some not, that’s why someone is more informed about the insides of the sound processing than other) have shown their point about the topic, so that’s all; if you have been convinced, it’s ok, otherwise people who will have the patience to read the comments will jusdge on their own.
@ghostly: I certainly respect that end users of an application don’t care what the underlying data format looks like, as long as the app reliably does the right thing. MS Excel users generally don’t care about how complex the .xls format has grown as long as it saves their formulas correctly.
I’m just interested in this from a developer’s perspective. Take it as a given that FFmpeg will eventually grow to assimilate every known multimedia format in existence. To that end, I always want to know how much complexity that will entail. We FFmpeg devs are a cranky bunch who are entitled to complain about how annoying it is to implement support, even for formats that “no one else was ever supposed to use directly” (which covers a vast chunk of our game-related formats).
Suddenly I wonder how complex XML is compared to Matroska’s EBML?
Matroska EBML has nothing whatsoever to do with XML. It is a tag/size-based format much like MP4. It has a few annoyances, but on the whole it’s pretty easy to parse.
[quote]
Take it as a given that FFmpeg will eventually grow to assimilate every known multimedia format in existence.
[/quote]
….. *epic facepalm*
and you’re lambasting Renoise for being supposed supporters of bloat? Wow. Just…. wow.
“I just love it when the fanboys fail to agree with each other about the precise nature of their object of worship.”
I’m not a Renoise developer, so I was simply making an educated guess based on what limited knowledge I have of its inner workings. That vV has cleared up the processing mystery, has only served to further dismantle your pointless argument against the format’s use of XML in the first place.
I have however, written a utility that parses through a Renoise song’s entire XML tree to return an accurate length of the track, and my utility can scan through a folder of 300 songs in a matter of seconds.
Laugh all you want, BYTE. But one day, YouTube (or whatever eventually supplants it) will support transcoding XRNS files directly to something Flash can deliver. And it will probably be because of FFmpeg.