Console homebrew communities have always had a precarious relationship with console pirates. The same knowledge and skills useful for creating homebrew programs can usually be parlayed into ripping games and cajoling a console into honoring ripped copies. For this reason, the Dreamcast homebrew community tried hard to distance itself from pirates, rippers, and other unsavory characters.
Funny how times change. While I toed the same line while I was marginally a part of the community back in the day, now I think I’m performing a service for video game archivists and historians by openly publishing the same information. I know of at least one solution already. But I think it’s possible to do much better.
Pre-existing Art
Famed Japanese game hacker BERO (FFmpeg contributors should recognize his name from a number of Dreamcast-related multimedia contributions including CRI ADX and SH-4 optimizations) crafted a program called dreamrip based on KOS’s precursor called libdream. This is the program I used to extract 4XM multimedia files from Alone in the Dark: The New Nightmare.
Fun facts: The Sega Dreamcast used special optical discs called GD-ROMs. The GD stands for ‘GigaDisc’ which implied that they could hold roughly a gigabyte of data. How long do you think it takes to transfer that much data over a serial cable operating at 115,200 bits/second (on the order of 11 Kbytes/sec)? I seem to recall entire discs requiring on the order of 27-28 hours to archive.
If only I possessed some expertise in data compression which might expedite this process.
KallistiOS’ Unwitting Help
The KallistiOS (KOS) console-oriented RTOS provides all the software infrastructure necessary for archiving (that’s what we’ll call it in this post) Dreamcast games. KOS exposes the optical disc’s filesystem via the /cd mount point on the VFS. From there, KOS provides functions for communicating with a host computer via ethernet (broadband adapter) or serial line (DC coder’s cable). To this end, KOS exposes another mount point on the VFS named /pc which allows direct access to the host PC’s filesystem.
Thus, it’s pretty straightforward to use KOS to access the files (or raw sectors) of the Dreamcast disc and then send them over the communication line to the host PC. Simple.
Compressing Before Transfer
Right away, I wonder about compiling 3 different compression libraries: libz, libbz2, and liblzma. The latter 2 are exceptionally CPU-intensive to compress. Then again, it doesn’t really matter how long the compressor takes to do its job as long as it can average better than 11 Kbytes/sec on a 200MHz Hitachi SH-4 CPU. KOS can be set up in a preemptive threading mode which means it should be possible to read sectors and compress them while keeping the UART operating at full tilt.
A 4th compression algorithm should be in play here as well: FLAC. Since some of these discs contain red book CD audio tracks that need archival, lossless audio compression should be useful.
This post serves as a rough overview for possible future experiments. Readers might have further brainstorms.
I take it the Dreamcast Broadband Adapter is still not exactly easily obtainable? Having one might make for some interesting changes in the compression level/effective transfer rate as compared to a serial cable.
I wonder if perhaps there might be any compression advantages to grouping similar files (by extension?) together and sending them, rather than the naive sector-by-sector approach.
Consider also QuickLZ, it’s optimized for fast execution and the compression is not too bad.
James – they were even scarce when they first came out. Might even be practical to setup multiple dreamcasts, and perform a serial port rip in parallel…
Also, did anyone look at connecting the GD-ROM drive directly to PC?
@James: Yep, the broadband adapters tend to be pretty rare. Even back in 2001-2003 when I was a bit more active in DC dev, they routinely sold for at least $100 on eBay. They have not gotten cheaper.
Also, I think I would rather go the whole track route rather than trying to sort out individual files.
@igorsk: For academic reasons, I’ll probably hold a little bake-off with as many libraries as I can easily get compiled for the DC. The thing that I’m measuring is which library can compress the most while producing a compressed stream at a rate of 11 kbytes/sec or better.
About connecting the GD-ROM to a PC– great question, I honestly hadn’t considered that. However, I don’t think the GD-ROM has a PC-compatible interface (like PATA), based on pictures and other items I can easily find on the internet.
@Peter: Well, I do have 3 separate DC consoles. I only have one power cable but I could probably get more of those. What I can’t easily get more of are the DC coders cables. If Lik Sang were still in business, I would probably order a few more. Good luck finding one for sale now.
The specs are out there but I lack the hardware talent to build them.
http://dknute.livejournal.com/25454.html
1,5Mbps, works like a charm here.
@bob: That looks almost crazy enough to work. Interesting how far the hardware has come that there’s a simple, pre-made ethernet module like that.
That’s not ethernet, it’s serial.
I can get 384000 bps with the serial adapter I got off Amazon (http://www.amazon.com/gp/product/B000067RVJ/).
@TapamN: Ethernet is also serial, technically. :) But thanks for clarifying.
384000 bps is impressive. I have never gotten higher than 115200. I thought maybe I achieved 230400 from the DC many years ago. But then I realized that wouldn’t make sense if it took me 27 hours to rip a gigabyte from a GD-ROM (115200 bps works out to ~27 hours in that case).
As for the BBA, yes, they are still fairly pricey, but you can always set up email notifications of newly listed ones and try to snipe it for < $100. That's how I got mine a few months ago. I have a feeling that a relative lot of them are going to surface as Dreamcasts are dumped by those who don't want them anymore. Admittedly it probably only has about $10 in actual parts, but it saves a lot of headache trying to make serial transfers work out…
LOLz “A 4th compression algorithm should be in play here as well: “… What are you smoking? I want a hit :-)
[quote] Readers might have further brainstorms. [/quote]
Yeah – buy or build an SD adapter.
There has been a tool for a few years now SDRip by JJ10DM to “archive” a GDI onto the SD in about 35 minutes.