The DDR Research Master Thread - UPDATED 2023-04-15 - Minor Addition - SOME DDR GB1 Memory Addresses

Good question, I admit the SSQ format, and its finer inner workings are still something I am trying to understand.


Related, I'm trying to find the edit data validation functionality that prevents playback of step data with 3 arrows on a beat.

So I think I've identified a pattern that should help me in finding the relevant code.

The 3 arrow combinations are turned into jumps, but it depends on the arrow patten.

It looks like if it's a 3 arrow combination with a L AND a R arrow in it, the other arrow is truncated, creating a LR jump;

If it's a 3 arrow combination with an U AND a D arrow in it, the other arrow is truncated, creating a UD jump.

So perhaps, doing a bitwise operation with the arrow note value byte, and either 0x09 (1001). or 0x06 (0110), and using that to determine what arrow to truncate? like check if (note_value & 0x09) = 0x09, or (note_value & 0x06) = 0x06, since the bitwise AND will preserve one of the two jumps, depending on what jump is present within the arrow tuple?

My phrasing, thinking might be derpy - woke up but a couple hours ago and have a headache, so not thinking optimally haha



_____________________________________________________________________________________________________________________________

Here's something that's been puzzling me a bit.

When it comes to Edit Mode, in the MAX-EXTREME (+ SuperNOVA-X2), the amount of space allocated to the freeze data (kn_e_freeze_seq_work) and non-freeze data (kn_e_seq_work), does vary somewhat from mix to mix - some allocating 2400 (0x960) bytes for each array (Extreme2 for instance), some mixes allocating more, some mixes allocating less.

I wonder why the inconsistency though - as opposed to them remaining consistent across the entire spectrum of PS2 games. It's not like the length of the songs on average changed much at all in that time frame... going from 3 difficulty levels in MAX (L/S/H) to 5 (B/L/S/H/O) isn't something that I see as necessarily affecting space requirements too much either UNLESS I am an idjit and missed something in that regard...

___________________________________________________________________________________________________________________________
OK, WHY THE HELL DIDN'T I THINK OF THIS BEFORE?!?!?!

Wanting to study the game engine's evolution means my also taking interest in pre-4thMIX versions of the engine - I had forgotten that DanceDanceRevolution 2ndMIX and DanceDanceRevolution 2ndMIX CLUB VERSiON 2 were both ported to the Dreamcast - which means I have, essentially, those versions of those games at my disposal to study the engine from that particular point in time in a way that might be a little less odious (in some ways at least) compared to fiddling with their arcade counterparts in MAME.
___________________________________________________________________________________________________________________________
So apparently in edit mode you can not only place three arrows on a beat in 2ndMIX DC, but the stepdata plays back OK (AND you can keep adding note while doing a test play of your edit data).

0 _____ o





😂😂😂


You COULD DO 8 ARROW BEATS IN DOUBLES in this too! What the actual fuck? 😂 🤣😂 🤣😂 🤣


___________________________________________________________________________________________________________________________
MAIN / FIRST POST UPDATED to add some SSQ data structure information.

God, I should have forcibly enabled MAX2 CS's logging output capability years ago, this is proving to be EXTREMELY helpful in filling in existing gaps in the publicly available documentation on SSQ data.

___________________________________________________________________________________________________________________________
Certainly would explain the "01 04 02 01 02 02 02 05 02 03 02 04" pattern of bytes Saxxionpike observed in the "type 2" chunk (the player/player(one of these should be pad)/kind fields being outputted in the log)

Maybe it's a good idea to write up the timing data next (it's fairly straightforward, the only tricky point is the time unit).

What exactly, if I may ask, is tricky about the time unit? (A pre-emptive/heads up kind of question, I guess , preparing my brain for headaches... )

Interesting how even in 2004 they used the sq_standard data structure variation that relies on the struct hack that was needed pre-C99-standards for flexible array members, seeing how by then they had easily the ability to utilize flexible array members.

Right now, I am trying to look for structs other than the ones already known from their being defined in the leftover debugging data
that are used in conjunction with the sq_standard data structure, figuring out how those work/are used, and going from there, am I overthinking how to go about it, or is that a logical approach do you think?

Regarding the timing data, Wan gave the info I was thinking of earlier in this thread about page 16 (I've quoted relevant sections):


This change in the time unit (from 1/75 second to 1/150 second) is what I was thinking of -- something to take note of when comparing pre-Extreme and post-Extreme SSQs.

Ah yes, man I forgot about that post from Wan, definitely good to keep in mind.

(that's the problem with a thread going on so long, haha, so much information that I forget is even posted. )
----------------------------------------------------------------------------------------------------------------------------------------------
Looks like some of that info (such as bar_max) are stored in the ny_mtbl array
----------------------------------------------------------------------------------------------------------------------------------------------
D'oh I'm an idiot, getting ny_mtbl defined helped me understand music_info data structure's verm_num field - in each ny_mtbl entry is a field, code - which for each song, has the same value as that in the song's music_info's verm_num field.
----------------------------------------------------------------------------------------------------------------------------------------------
Anyone here mess about with the source to the utility Dwarf2CPP? If so, has anyone been able to figure out how to get it to output the addresses to global variables? I feel like, damn, this'd help a lot in tracing globals whose definition is in the leftover debugging data for one DDR PS2 title, but not in another PS2 DDR title's debugging data.
----------------------------------------------------------------------------------------------------------------------------------------------
OK, this is gonna be a really stupid question from a programming standpoint, but how different, really, is Konami's LZ77 compression/decompression routines from other implementations?

(Been a while since I looked into it - or did anything substantial with DDR reverse engineering for that matter, lots of shit IRL going on, but I haven't given up, just so's y'all know.)

Edit 2022/12/18: Updated the timing details section.

I've been doing my own research on the background animation scripts for the MAX/EXTREME-era games. As travelsonic said at the top of this thread, the individual scripts for each song are all compressed using Konami's LZ algorithm. I finally managed to build a decrypter, and got to work analyzing the script files. Here are my findings so far:

For this example, I will be using Let's Groove's script from DDRMAX JP. All entries are 4 bytes long and stored in little-endian format.

Header

The file starts with a header, containing the total uncompressed length, chunk type (0x05), and number of entries. Each of these header values are 4 bytes long, unlike the 2-byte chunk type in step data. Also unlike step data, the number of entries (i.e., 0x17, or 23 in decimal) is repeated twice, as we will see below.

Beat entries

The first set of entries lists the beat positions where each new BGA plays. Just like in step data, the whole value represents 1/4096th of one beat -- or, in other words, the first 2 and a half bytes represent the measure. Sometimes the same beat will be listed more than once; this is explained in the next section.

Timing entries

The first byte contains a bunch of flags which affect timing.
- If the lower half of the byte is 3, (i.e., the byte ends in a 3), pause the clip.
- If the lower half is 5, play the clip normally.
- If the lower half is 6, play the clip normally, then in reverse, and so on.
- If the lower half is 9, play the clip in reverse.
- If the lower half is A, play the clip in reverse, then normally, and so on (opposite of 0x_6).

- If the upper half is 1 (i.e., the byte starts with a 2), play the clip at normal speed.
- If the upper half is 2, play the clip at 0.75x speed.
- If the upper half is 3, play the clip at half speed.
- If the upper half is 4 and the third byte is 0x00, play the clip at 1.5x speed.
- If the upper half is 4 and the third byte is not 0x00, play the clip from start to finish for a given length of beats specified in the third byte, adjusting the speed as necessary. In this example, a value of 0x04 means the clip will play start-to-finish within 4 beats at the song's current tempo, while 0x08 will make the clip last for 8 beats, or at half the speed of 0x04.
- If the upper half is 5, play the clip upon finishing the previous one. Used when the accompanying beat entry is a duplicate, for seamless off-beat transitions. If the previous entry has specific timing instructions, they are carried over to this entry.
- If the upper half is 6, start the clip at the frame specified in the third byte.
- If the upper half is 9, display the song's BG image. I don't know what effect, if they, that the other bytes have here.
- If the upper half is A, play the clip at 1.5x speed.

The second byte points to which video clip will be played, as described in the section below. However, there are some exceptions:
- 0x14 represents the song's background image, typically shown at the end of the song.
- 0x15 through 0x1D represent shared generic videos. Most of these were replaced from MAX2 onwards.
- 0x66 and up represent console-exclusive videos, such as those used in Kind Lady or So In Love. Like the common videos, these are not included in the section below.

The third byte controls the video clip's speed. All clips are 80 frames long and, at least for NTSC-region games, play at 30 frames per second by default. This works out to running for four beats at 90 BPM. However, they can also be sped up to match the BPM of the song and play for a specific number of beats. In this example, a value of 0x04 means the clip will play start-to-finish within 4 beats at the song's current temp (about 130.1 BPM).

The fourth byte is not used, at least as far as I have seen so far.

Video list

Finally, the file ends with a list of all the video clips used in the script. Generic videos, as described above, are not included in this list. This chunk starts with the number of entries, each of which are 4 bytes long. Each subsequent dword corresponds to one of the internal video clip titles (the titles used by AeronPeryton in their scripts). They seem to be encrypted versions of their alphabetic names, although I don't know how they are encrypted. I'm putting together a Google Docs spreadsheet logging this and other info on every video clip from the PS2 games: https://docs.google.com/spreadsheets/d/1Iy_oxteY_3CLH8EV60rGdcd7UA3hhLqXpv-PTVlXgZY/edit?usp=sharing

---

I know what the remaining videos are named (after a shit ton of trial and error replacing file names in DDR Festival). When I get back on my desktop I can post them.

This is awesome work!

For a long time, I wondered if what we see as the SSQ format for steps is really a container format that is utilized for multiple sequence types - steps, BG video sequences, etc.

I can't help but feel like this supports that notion - seeing how the header is identical (unless I am missing something) to that of the sq_standard data structure, (definition can be found in the EX2 PS2 prototype's debugging data, and in both US EX PS2 Prototypes' debugging data), and how the beat data seems similar to how the SSQ beat/steps is laid out (only with a dword for each video clip value, as opposed to a byte for each step value).

This notion that there is a superset sequence format, and SSQ is a specialized subset of a more general sequence format, also SEEMS supported, IMO, by the data structures we do have regarding step data in the leftover debugging data being prefixed sq_ (sq_standard, sq_header, etc).

Do any of these data structures, for instance, ring any bells?
struct sq_header{
    unsigned int size;
    unsigned short kind;
    unsigned short division;
};
struct sq_standard{
    sq_header header;
    unsigned int num;
    signed int body[1];
};
(Note that body being a single element array (and at the end of sq_standard) indicates the use of the pre-C99 struct hack from before flexible array members were a thing in the C programming language. Interesting, since that could indicate that the underlying codebase for some of these structures go back to DDR versions from 1998 or 1999, but that is PURELY speculation on my end.)

Not really. I've seen structures like those pretty much only from what I've read in this thread. In fact, I did try compiling and running Dwarf2CPP on some of the games' ELF files, but it kept crashing when trying to write the second file (which always happens to be les_sel_lv.c). I also gave Ghidra a try to decompile the ELFs and/or arcade binaries, but no such luck at the moment.

On the plus side, I have finally figured out how the video filenames are encoded at the end of the scripts. First, since the system reads 32-bit values like these in little-endian, we will need to reverse the byte order before converting the video's hex code to binary (using jrlova/MAX1.avi as an example):


Once we have our binary string, split it up into groups of 5 bits each. There will be two spare bits left over; these can be ignored. It is important to remember that bits are read from right to left, so start from the right end, like so:


Now that we have our binary string cut up like so, we can translate it into letters. Simply convert each binary slice into a decimal number -- again, starting from the right -- and convert that number into a letter, starting with 0=a, 1=b, and so on through 25=z.


And there you have it!

---

Weird, I've not had this problem 0 _ o

Interestingly, the guy who made Dwarf2CPP is making a newer version that will have more robust features like actually being able to note the locations of functions and global variable addresses in output source files, and being able to preserve data types better than you can (for example currently it seems to remove the explicit declaration of a var as signed - what this guy is working on will let you have an output where that doesn't happen). Also, he's gonna be adding the ability to search for DWARF data in a given ELF, in case references to existing section headers w/ DWARF data became corrupted, or were wiped w/o the data itself being wiped for whatever reason, etc.

Right! Sorry I got distracted for a few weeks or so.

These are the missing/unknown-named videos.

jxdhit = dragon1
jxdpop = circle12
jxdrag = dragon2
jxthda = taiqi4
jzcdda = colors1
jzcddb = colors4
jzcita = nyc1
jzdhit = dragon3
jzdiza = text31
jzdizb = ring8
jzdrhi = fire7
jzfloa = sphere4 (plain)
jzflob = sphere4 (glowing overlays)
jzhara = blinds2
jzharb = blinds1
jzkcoa = kl1
jzkina = kl2
jzkisa = kl3 (part 1)
jzkisb = kl3 (part 2)
jzkkia = kl5
jzklia = kl8
jzklib = kl4
jzktia = kl6
jzkwha = kl7
jzmaxa = max3
jzmira = discoball2
jzmirb = discoball1
jzmizu = bubnel2
jzpuka = beads3
jzrasa = beads2
jzrasb = beads1
jzshaa = caramel1
jzsila = moon2
jzsnoa = kaliedoscope6
jzsnob = kaliedoscope9
jzsqua = square5 (part 1)
jzsqub = square5 (part 2)
jzsquc = square5 (part 3)
jzsqud = square5 (part 4)
jzsque = square5 (part 5)
jzsrha = colors3
jzwado = wind2
jzwaic = water5
jzwaku = lady23
jzwoma = lady20 (the woman's skin is transparent)
jzwomb = lady26 (the woman has a visible skin color)
jzzzza = bluey1 (reincarinated MAX ccdrga)
jzzzzb = cube9 (reincarinated MAX cccuba)
jzzzzc = fireworks (...reincarinated MAX ccsaca despite being exactly the same???)
jzzzzd = cards1 (reincarinated MAX ccitaa)
jzarua = circle13a
jzarub = circle13b
jzfuri = lady24
jzneko = lady25
jzsiba = pentagon2
jzsibc = orange2
jzsita = circle14
jdiaha = heart14
jdiahb = heart13

Over the past... however long since my last post, I've been working on a Python script to convert in-game BGA data into Stepmania's BGCHANGE format. I could set up a github repository and share it... or, I could just share the resulting BGCHANGE scripts themselves. (Or both.) These are all based on AeronPeryton's video rips, since their names and timings are more consistent than Beware's and others' versions. Before I go ahead with that, there are some finer points I'd like to discuss everyone's preferences on, but that can be saved for a new thread.

Another project I'm excited about is the prospect of is fixing or replacing the broken BGA scripts in games like DDRMAX NA and modding them back into the game. The only thing I'm hung up on at this point is how to re-compress them with Konami's LZ77 algorithm. The ddr-tools (https://github.com/root670/ddr-tools) and scharfrichter (https://github.com/SaxxonPike/scharfrichter) repositories I used to decompress the game data don't yet have any compression functions implemented. For what it's worth, I did find another repository which does include LZ77 compression (https://github.com/sbruder/konami-lz77), but it's in Rust and my compiler isn't working at the moment. Even my attempts to port the code over to C haven't worked so far. So... if anyone knows about any other LZ77 compression code anywhere, I would certainly appreciate any such pointers.

---

So do you think it's possible for this (actually kinda complex, but still reasonable) video sequence, then, to actually exist in a copy of DDRMAX USA?

Is that a scrapped bgvid setup or is that something you've concocted yourself?

---

It's something I made myself. DDRMAX USA's scripts are all MAX/MAX2 arcade imports, nothing would ever be as complex as that, even though I suspect it's perfectly possible to replicate and put in the game.

Also you can tell it's a fanmade script because it uses this video which doesn't appear anywhere aside from LIVING IN AMERICA in MAX2 and .59 in Ultramix. Even though it's a way cooler video than some of the others that get used a lot. Sad!



I sadly don't suspect scrapped background scripts exist.

And I thought the script we actually got for that song was one of the best "new" scripts in MAX NA...

To answer your question: pretty much, yeah. It depends on what video clips are available in a given game. Heck, just about every game from this period has at least a few unused clips lying around on-disk. If it is possible to add more, I wouldn't yet know how to go about it. Presumably you'd need to update the game's .ELF (or arcade version equivalent), adding to both the movie name list and the table of contents, but I'm not keen to dig further into that rabbit hole just yet. At any rate, I've just been updating my movie list spreadsheet, adding the clips included in the EU PS2 Dancing Stage games and even the arcade versions.

---

I actually call jrtryt "sun4", jmfilm (the blue one) "film7" (my HD remake pack supports this), "jmglow" colors5, and jmline "line5".

jruzua and jruzub (whatever the clip where the body is glowing white) is "lady21", jruzuc (the clip where the body is green and has a spray-painty effect) is "lady22", and finally jrwira is "light12".

An expansion pack I made for beware's videos all the way back in 2015 support all of these names.

Times like this where I really think we need something for DDR reverse engineering like what the old Sonic Stuff Research Group was for Sonic the Hedgehog game reverse engineering. (God, just saying that makes me feel OLD AF 😂😂😂)

Welp, I have good news and bad news to share on my DDRMAX BGA modding progress. The good news is that my LZ77 compression script is pretty much done (in Python, no less), and I've compiled all my new scripts back into the game. The bad news is that they aren't working, instead showing the song's static background image. I even managed to locate the memory address where the uncompressed step and BGA data for the current song are stored (0xF9F000), plus the address which points to the address of the current step (0x517860), and everything looks correct on that end. There must be something else I'm missing...

---

Have you tried modding the BGAs on DDR Extreme US/Dancing Stage Fusion/DDR Festival? For some reason, the MAX-based games don't load scripts if I even switch one of the clip names around, but on the Extreme US-based games, they still work.