Logo

The DDR Research Master Thread - UPDATED 5-21-2019: Some Arcade MAX-era Engine Notes Added

Register Log In Back To Forums

Post #141 · Posted at 2018-08-05 10:10:32pm 1 year ago

Offline root670
root670 Avatar Member
7 Posts
United States
Reg. 2011-06-02
Log in to send this user a private message

Last updated: 2018-08-13 01:30am
Glad to see so many discoveries have been made in this thread! I just released the source code for my TCB extractor and converter if anyone would like to use it:

https://github.com/root670/ddr-tools

It would be interesting to figure out the "unknown" parts of the TCB header that are currently just copied over when injecting a PNG. I don't remember if it properly converts all TCB files, but it currently supports 2 types (16 and 256 color).

Enjoy!

Post #142 · Posted at 2018-08-05 10:55:20pm 1 year ago

Offline pm41224
pm41224 Avatar Member
219 Posts
United States
Reg. 2012-05-19
Log in to send this user a private messageYouTube
PlayStation Network: pm41224Nintendo Network ID: pm412243DS Friend Code: 4210-4460-8178
"DanceDanceRevolution!"
This is pretty useful for extracting TCB stuff from the DDR PS2 ISOs. You'll never know what you'll find in those TCB files!

Post #143 · Posted at 2018-08-10 08:01:26pm 1 year ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer

Last updated: 2018-08-10 08:05pm
Quote: root670
Glad to see so many discoveries have been made in this thread!

Indeed, progress is (albeit very slowly) being made.

*sad sigh* I just wish I could make use of that DWARF debugging data at the end of the PS2 executable(s) - since not just having function and variable names, but knowing where they go, would make everything a hell of a lot easier. Been trying to make sense of the documentation I can find on the DWARF format, and find myself getting lost quickly in the soup.

Quote: root670

I just released the source code for my TCB extractor and converter if anyone would like to use it:

https://github.com/root670/tcb-tools

It would be interesting to figure out the "unknown" parts of the TCB header that are currently just copied over when injecting a PNG. I don't remember if it properly converts all TCB files, but it currently supports 2 types (16 and 256 color).

Enjoy!

Hey, if I credit you, would it be cool if I integrated this into a program I am trying to design and make to easily facilitate hacking DDR games?

No idea if / when it'd ever be done knowing my busy schedule, but I thought I'd ask now so it'd be ahead of that ball.

Post #144 · Posted at 2018-08-11 03:44:26pm 1 year ago

Offline root670
root670 Avatar Member
7 Posts
United States
Reg. 2011-06-02
Log in to send this user a private message
Quote: travelsonic
Quote: root670
Glad to see so many discoveries have been made in this thread!

Indeed, progress is (albeit very slowly) being made.

*sad sigh* I just wish I could make use of that DWARF debugging data at the end of the PS2 executable(s) - since not just having function and variable names, but knowing where they go, would make everything a hell of a lot easier. Been trying to make sense of the documentation I can find on the DWARF format, and find myself getting lost quickly in the soup.
Try looking into documentation for CodeWarrior, since that seems to be the compiler suite they use. It might have information about how the debugging information is stored since it doesn't seem to follow DWARF 1 or 2 as the standards documents describe them. I remember it seemed like a hybrid of the two standards was used for the debugging information in DDR.
Quote: travelsonic
Quote: root670

I just released the source code for my TCB extractor and converter if anyone would like to use it:

https://github.com/root670/tcb-tools

It would be interesting to figure out the "unknown" parts of the TCB header that are currently just copied over when injecting a PNG. I don't remember if it properly converts all TCB files, but it currently supports 2 types (16 and 256 color).

Enjoy!

Hey, if I credit you, would it be cool if I integrated this into a program I am trying to design and make to easily facilitate hacking DDR games?

No idea if / when it'd ever be done knowing my busy schedule, but I thought I'd ask now so it'd be ahead of that ball.
Yes, absolutely! I had visions of a similar utility in the past to facilitate editing assets and adding/replacing songs from the FILEDATA.BIN file. Never got around to creating it but it's totally possible with knowledge of the formats used in the main data file.

Post #145 · Posted at 2018-08-12 03:09:06pm 1 year ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer

Last updated: 2018-08-12 03:11pm
Quote: root670
Quote: travelsonic
Quote: root670
Glad to see so many discoveries have been made in this thread!

Indeed, progress is (albeit very slowly) being made.

*sad sigh* I just wish I could make use of that DWARF debugging data at the end of the PS2 executable(s) - since not just having function and variable names, but knowing where they go, would make everything a hell of a lot easier. Been trying to make sense of the documentation I can find on the DWARF format, and find myself getting lost quickly in the soup.
Try looking into documentation for CodeWarrior, since that seems to be the compiler suite they use. It might have information about how the debugging information is stored since it doesn't seem to follow DWARF 1 or 2 as the standards documents describe them. I remember it seemed like a hybrid of the two standards was used for the debugging information in DDR.

Hmm, I have a VM with the CodeWarrior compiler suite for Playstation2 development set up on it - documentation and all, I'll take a look, and see what I can find, if anything.

Post #146 · Posted at 2018-08-13 01:36:44am 1 year ago

Offline root670
root670 Avatar Member
7 Posts
United States
Reg. 2011-06-02
Log in to send this user a private message
I've created a new script called filedata-tool.py in https://github.com/root670/ddr-tools. This combines the existing dump and create scripts in addition to adding support for more games, accounting for "hidden" data missing from the file table in the game's ELF, and adding rudimentary filetype identification! I've tested dumping filedata.bin from Extreme, MAX JP, and MAX US and verified a newly created filedata.bin was identical to the original.

Post #147 · Posted at 2018-09-22 03:06:09pm 11 months ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer

Last updated: 2018-09-23 05:05pm
I'm feeling a little disappointed that Saxxonpike hasn't seemed to incorporate into his work on the ssq format info from a fork I did a while back, as I did note a number of things I found important - and it would be nice if it were unified, rather than having the data all over the place.

For example:
1a. In Playstation 2 mixes, at least, the fact that the stepdata is all in one chunk.
1b. At the beginning of said chunk, there is an array of offsets (relative to the start of the chunk) for each piece of stepdata.
1c. Those offsets are for all the stepdata for a particular song - all difficulties, as well as singles AND doubles charts

2a. In Playstation 2 mixes, at least, background movie scripts are stored in another chunk of data immediately following step data.
2b. The script data itself isn't stored in the stepdata file itself, and it doesn't SEEM like, at least, a script is even referenced in the stepdata file.
2c. There is a number of scripts equal to the number of songs in the game.
2d. Like with step data, at the beginning of the chunk, there is an array of offsets (relative to the start of the chunk) for each script.
2e. (speculation) Rather than storing the script in the stepdata, or even referencing the script in the data, the game just uses the same array element number to simultaneously access both the stepdata from the offset in the stepdata chunk, and the script data from the offset in the stepdata chunk. This would make sense, IMO, since upon selecting a song, the game is passed a value that when I altered it, changed what data was loaded. This made it abundantly clear that the value represents an array element number, and it'd be trivial to use that number to access the addresses in both arrays, seeing the that the scripts are in the same song order as the step data - which means that the same array element number would retrieve the correct/desired data from the correct element in both arrays.

Plus there were some terminology cleanups that I did with regard to bits, bytes, nibbles, etc.

Post #148 · Posted at 2018-09-23 04:13:09pm 11 months ago

Offline Wan
Wan Avatar Member
310 Posts
Chile
Reg. 2008-01-13
Log in to send this user a private messageTwitter: WannyTiggah
Steam: WannyTiggah
"I want to change my username =("

Last updated: 2018-09-23 04:49pm
Whoa, things are getting REALLY interesting. When ripping stepcharts from PS2 becomes a reality it'll be the key to a mystery box Surprised

EDIT: How do you guys rip the frames of the MJPEG videos?

Post #149 · Posted at 2018-09-23 07:15:55pm 11 months ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer

Last updated: 2018-09-23 07:19pm
So I've been looking at the order in which the song data for the struct_info metadata, the stepdata, and the actual song files are stored in filedata.bin, and I am perplexed.

This is the MAX2 JP CS song list, where I listed the songs in the same order as the song data is stored in filedata.bin:
Quote


Long Train Runnin'
Maximum Overdrive
Waka Laka
Baby Love Me
D2R
Destiny
Living In America
Sweet Sweet ♥ Magic
Ever Snow
AM-3P (303 BASS MIX)
The Reflex
So Fabulous So Fierce (Freak Out!)
Drifting Away
Stay
Secret Rendez-vous
Little Boy
Rain of Sorrow
Maxx Unlimited
Dive To The Night
Tsugaru
BRE∀K DOWN!
Burning Heat! (3 Option Mix)
Nothing Gonna Stop
Fantasy
i feel ...
CANDY ♥
Spin the Disc
Trance De Janeiro
Look At Us
The Whistle Song (Blow My Whistle Bitch)
革命 (Kakumei)
AFRONOVA (From Nonstop Megamix)
AM-3P (AM EAST MIX)
Brillant 2U (K.O.G G3 Mix)
B4U (B4 ZA BEAT MIX)
DROP OUT (From Nonstop Megamix)
Dynamite Rave (B4 ZA BEAT MIX)
Hysteria 2001
祭 JAPAN (From Nonstop Megamix)
SEXY PLANET (From Nonstop Megamix)
Super Star (From Nonstop Megamix)
Still In My Heart (MOMO MIX)
Wild Rush (From Nonstop Megamix)
Burnin' The Floor (BLUE Fire mix)
Tsugaru (Apple Mix)
Ecstacy (Mifnight Blue Mix)
Silent Hill (3rd Christmas Mix)
Celebrate Nite (Euro Trance Style)
Higher (Next Morning Mix)
My Summer Love (Tommy's Smile Mix)
Look to the Sky (True Color Mix)
Cutie Chaser (Morning Mix)
Do It Right (Harmonized 25step Mix)
Drop the Bomb -System S.F Mix-
Dynamite Rave (Down Bird SOTA Mix)
I'm For Real
Jam & Marmalade
Kind Lady
Logical Dash
Overblast!!
Peace-Out
So In Love
The Shining Polaris
Look To The Sky (Trance Mix)
Do It Right (80s electro Mix)
Kind Lady (Interlude)
B4U
Crash!
Dead End
Drop Out
Dynamite Rave
LOVE ♥ SHINE
PARANOiA Survivor
PARANOiA Rebirth


My question is: when they went with storing the data on disk, how the hell did they end up choosing this particular order for the song data? It's not alphabetical, it's not in ascending or descending BPM, it's not by type (Konami Original, beatmania crossover, license, etc), or at least it doesn't appear to be that way, so ... what the hell is this order based on? How does this make any bloody sense at all?
0_0

Post #150 · Posted at 2018-09-23 08:56:03pm 11 months ago

Offline Wan
Wan Avatar Member
310 Posts
Chile
Reg. 2008-01-13
Log in to send this user a private messageTwitter: WannyTiggah
Steam: WannyTiggah
"I want to change my username =("

Last updated: 2018-09-23 09:10pm
No clue. maybe it's their preferred order?

I extracted the graphics&SSQs from DDRMAX, DDRMAX2 and Extreme... and in all cases, the order is the same, starting with:
Midnite Blaze
ORION.78
Share My love
Groove
Groove 2001
Let the beat hit em (CLASSIC R&B STYLE)
Do It Right
Look To The Sky
ON THE JAZZ
Healing Vision (Angelic Mix)
DIVE (more deep & deeper style)
NORI NORI NORI
...etc

And that order doesn't make any sense either.


EDIT: It's the SAME order in AC.

https://i.imgur.com/UoO2RY0.png
Compare with your list.

Post #151 · Posted at 2018-09-23 09:59:16pm 11 months ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer
Seems really weird that they'd prefer an order that has absolutely no rhyme or reason, and might add complexity in searching for, retrieving data... barring predefined lists taking care of that shit, or something I'm missing/not thinking of. ~___~

Post #152 · Posted at 2018-09-27 05:49:27pm 10.9 months ago

Offline Wan
Wan Avatar Member
310 Posts
Chile
Reg. 2008-01-13
Log in to send this user a private messageTwitter: WannyTiggah
Steam: WannyTiggah
"I want to change my username =("
I just got informed that the stepcharts in CS games are stored in a totally different (and therefore undocumented) format: .CSQ
Ouch.

Post #153 · Posted at 2018-09-27 08:06:15pm 10.9 months ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer
Quote: Wan
I just got informed that the stepcharts in CS games are stored in a totally different (and therefore undocumented) format: .CSQ
Ouch.

Hmmm, are you sure - at least with regards to them being totally different? I mean, going through Saxxonpike's guide did lead me to seeing the same things in the stepdata I was looking at so far as the data structure, et all.

Post #154 · Posted at 2018-09-28 06:30:50am 10.8 months ago

Offline Wan
Wan Avatar Member
310 Posts
Chile
Reg. 2008-01-13
Log in to send this user a private messageTwitter: WannyTiggah
Steam: WannyTiggah
"I want to change my username =("
At least that was what I was told D;
If you can figure the format it'll be awesome, then!

Post #155 · Posted at 2018-10-02 02:52:00pm 10.7 months ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer
Seems I've got some investigating to do then. Big Grin

Post #156 · Posted at 2018-10-02 04:09:18pm 10.7 months ago

Offline CentiJR
CentiJR Avatar Member+
74 Posts
United States
Reg. 2011-10-17
Log in to send this user a private message
The order of the list follows the "masterlist". It's probably in the order when they added/worked on the songs in general to DDR.
http://i109.photobucket.com/albums/n48/shahllen/SNSDGaga20Sig.png

Post #157 · Posted at 2019-01-03 05:24:44am 7.6 months ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer

Last updated: 2019-01-03 05:26am
Man, been trying to see if I can find a section of code that could handle how arrow data is interpreted. If I can find this code, I would be so happy, as I think we could finally properly enable 3-arrow, and quad steps. My thought process is that there is a switch statement for the possible arrow values (non-freeze & freeze?) - and modifying how the cases are handled (and how cases outside the current choices) would allow me to enable this. Taking advantage of IDA Pro's proximity mode has helped me a great deal in finding more things overall, but god damn, this snake of functions is a mess to go through. 0_o

I realize that if this is correct, that I might have to add cases, but in the MAX-era games, I could just use the space currently spent by the old debugging information (which can be as small as 208KB, or as large as 2,983KB (~2.983MB) depending on the game) - make a copy of the function there, modify it as needed, and then change the references to the old function so they point to this new function.

Guess what:
https://3.bp.blogspot.com/-99-g-ahW6ig/XC2bzQWPwJI/AAAAAAAAAzc/ATcQsfrfMGoMDrk58fk-lw_Djouv_X_AACLcBGAs/s1600/1-2-2018%2B-%2BEnabled%2B3-arrow%2BPlacement%2BIn%2BEdit%2BMode%2B2%2B-%2Bshowing%2Bcombinations.png

Just made it so you could put down 3 arrows at a beat, in any combination of steps and freezes. The game doesn't interpret the data correctly in-song yet, but I am working on finding out where the stepdata is read and interpreted during a song, and seeing what changes are needed to make them appear properly in game.

Enabling 3 steps on a beat was rather easy - there are 2 places in edit mode where it checks for attempting to put > 2 arrows on a beat - one for non-freeze steps, and one for freeze steps, both use a SLTI (Set on Less Than Or Equal) instruction to determine if the game should continue to place a step in edit mode or not on a particular beat - with 2 being the value checked against... changing this to a 3 or higher allows you to place 3 steps on a beat. Big Grin

Post #158 · Posted at 2019-01-03 10:47:24am 7.6 months ago

Offline Dancefreak
Dancefreak Avatar Member+
271 Posts
United Kingdom
Reg. 2006-08-12
Log in to send this user a private message
I don’t think I have it on a memory card anymore but there was an era where making edits with 12ths on one side and not 12ths on the other then playing the chart on one player caused the game to get confused and attempt to merge the charts and sometimes resulted in triples and quads. Is that something you might have been able to get an answer for with your work in edit mode ? I’ve always wanted to know what caused it. I wish I knew specifically what I used to do to make it work but I definitely remember doing it at one point and being very confused by the result.

Post #159 · Posted at 2019-02-20 10:04:53pm 6 months ago

Offline travelsonic
travelsonic Avatar Member
135 Posts
Not Set
Reg. 2006-10-18
Log in to send this user a private message
Steam: terminalcancer

Last updated: 2019-02-20 10:11pm
Quote: Dancefreak
I don’t think I have it on a memory card anymore but there was an era where making edits with 12ths on one side and not 12ths on the other then playing the chart on one player caused the game to get confused and attempt to merge the charts and sometimes resulted in triples and quads. Is that something you might have been able to get an answer for with your work in edit mode ? I’ve always wanted to know what caused it. I wish I knew specifically what I used to do to make it work but I definitely remember doing it at one point and being very confused by the result.

I remember reading about that, and I think once when messing around with the game on consoles and having this happen too. That's something else I really wanna look into as well.

Interestingly, used a program called ELF Parser for fun with DDR PS2 executables (ELF format, of course), and I saw that it recognized the section of the ELF executable that contains all that debugging data I had seen before - recognizes the game executable's .debug section, and has an offset that sends you to the point in the file with that debugging data that I can't get IDA Pro, or anything else to recognize. ~_~ Now I'm even more annoyed. 0_o

____________________________________________________________________________________________________________________________________________

FIRST POST HAS BEEN UPDATED with some information on the song selection screen, and how the data that gets loaded is picked.

____________________________________________________________________________________________________________________________________________


So, upon further research, it seems like there is more to the reading of stepdata than I had previously thought.

By this, I mean that it seems like you CAN, upon enabling placing of 3 arrows per beat, have certain combinations work - BUT, you can only have 2 of each kind of step (non-freeze and freeze) on a beat in order for it to show up. That means 2 non-freeze + 1 freeze steps, 1 non-freeze and 2 freeze steps, as well as 2 freeze and 2 non-freeze steps, will all show up correctly, while 3 non-freeze steps, 3 freeze steps, 3 non-freeze + 1 freeze steps, and 1 non-freeze and 3 non-freeze steps, will NOT show up correctly.

But this is not certain, and I need to keep experimenting with things.

____________________________________________________________________________________________________________________________________________


So here's a mystery that has been baffling me about edit mode:
There are two memory addresses. One has a value that starts at 0, the other begins at a predefined value which changes depending on the song.

For example:
Long Train Runnin' = 0x2C000
Burning Heat! (3 Option Mix) = 0x3F000
Waka Laka = 0x40000
Matsuri Japan (From Nonstop Megamix) = 0x43000
AFRONOVA (From Nonstop Megamix) = 0x4C000
MAXX Unlimited = 0x73000

The first memory address has a value added to it. When the value in the first address becomes equal to the value in the second value PLUS 0x762, the song playback is ended.

But what the hell is this value?

It can't be the total seconds the song is played, as each value is too large for that. The size issue also seems to rule out the value being in frames - as even at 60 FPS, the values come nowhere near the values above. For example, Burning Heat is approx. 1 min 35 seconds, or 95 seconds long * 60 fps = 5700 in decimal, which is 1644 in hex, far below any of these values.

____________________________________________________________________________________________________________________________________________

Quote: root670

I wonder - when you have time, would you pursue making a compressor for the LZ compression Konami uses - every Konami related LZ tool I could find couldn't get it quite right.

I MAY have found the decompression routine in the DDRMAX2 JP executable, and I'd imagine the compression routine is there as well, so I will be working on this as well.

Post #160 · Posted at 2019-02-21 02:27:31am 6 months ago

Offline MarsPhoenix
MarsPhoenix Avatar Member
632 Posts
Not Set
Reg. 2007-12-08
Log in to send this user a private message
Could be counting beats and/or measures.
Register Log In Back To Forums

0 User(s) Viewing This Thread (Past 15 Minutes)

©2006-2019 Zenius -I- vanisher.com -5th style- IIPrivacy Policy
Web Server: 6% · Database: 8% · Server Time: 2019-08-21 00:27:37
This page took 0.015 seconds to execute.
Theme: starlight · Language: englishus
Reset Theme & Language