Logo

The DDR Research Master Thread - UPDATED 2023-04-15 - Minor Addition - SOME DDR GB1 Memory Addresses

Register Log In Back To Forums

Post #361 · Posted at 2022-12-26 08:31:02pm 1.3 years ago

Offline travelsonic
travelsonic Avatar Member
190 Posts
Not Set
Reg. 2006-10-18


Last updated: 2022-12-27 09:49pm
Silly question, how different IS Konami's LZ77 decompression algorithms from others, would you say? (Haven't had time to look at it lately and try to make my own implementation and was always curious - breaking down or figuring out algorithms out of disassemblies is also something I absolutely admit I (at least currently) suck at despite considering myself an analytical son of a bitch haha)

Post #362 · Posted at 2022-12-27 02:12:40am 1.3 years ago

Offline SpyHunter29
SpyHunter29 Avatar Member
331 Posts
United States
Reg. 2008-06-09

I don't have examples of other forms of binary LZ compression to compare with, so I don't know if this answers your question, but I can share some of the notes I took for myself when I was working on my own compression/decompression scripts.

For those who know the basics of LZ compression, you'll know that the encrypted text consists of "coordinate" pairs in addition to actual characters. These coordinates instruct how to copy previously-written text into the output string, defining how far back into the written text to start copying from and how many characters to copy. In Konami's LZ77 format, periodic "control bytes" are inserted every eight bytes or so, with each bit indicating whether their corresponding bytes represent actual text (0) or instruction coordinates (1). Copy instruction bytes look like this (in little-endian binary format):

10bb aaaa

where aaaa is the distance (1 to 16) and bb is the length (1 to 4). Note that these bytes always start with "10", or in other words, range from 0x80 to 0xBF (128 to 191).

If we need to copy bytes over a longer distance, two bytes are used:

0bbb bbaa
aaaa aaaa


where the maximum distance is now 1,023 and the maximum length is 33. Note that the aforementioned control bit applies to both of these bytes, and the first byte of these pairs ranges from 0x00 to 0x7F (0 to 127). So far, I believe that the algorithm looks for matches using the maximum distance and length possible, but starting with short-distance searches before trying again with a long-distance search. Finally, unless it is written verbatim, 0xFF marks the end of the file.

To close out with an example, here's the first "line" of AM-3P's script from MAX2/MAX NA.

Encrypted:
54 E0 00 80 05 93 17 97 00

Breakdown:
54: Control flags (00101010)
E0: (0,0) E0
00: (0,0) 00
80: (1,2) 00 00
05: (0,0) 05
93: (4,3) 00 00 00
17: (0,0) 17
97: (8,3) 00 00 00
00: (0,0) 00


Decrypted:
E0 00 00 00
08 00 00 00
17 00 00 00
00


Hope that helps.
98% of teenagers do or have tried caving into peer pressure. If you're one of the 2% who hasn't, DO NOT copy & paste this in your signature.

Post #363 · Posted at 2022-12-27 10:07:06pm 1.3 years ago

Offline travelsonic
travelsonic Avatar Member
190 Posts
Not Set
Reg. 2006-10-18


Last updated: 2022-12-27 11:23pm
That is definitely a huge help! Thanks a lot! Big Grin
This part specifically:
Quote
n Konami's LZ77 format, periodic "control bytes" are inserted every eight bytes or so
Is what was tripping me up a bit (specifically, not knowing about that part). Your explanation of that helped me loads, again thanks a lot!

Post #364 · Posted at 2022-12-28 03:18:37am 1.3 years ago

Offline SpyHunter29
SpyHunter29 Avatar Member
331 Posts
United States
Reg. 2008-06-09


Last updated: 2022-12-28 03:18am
You're quite welcome, travelsonic! Thanks, in turn, to root670 on github, and to the comments they added to their version of the decompression code (https://github.com/root670/ddr-tools).

As for CuzcoBlocko, it took some doing, but I did manage to play some custom BGA scripts in DDR Festival. I'm curious what those games do differently from their predecessors so as not to block those changes... Maybe there's some sort of checksum thing going on to prevent changes?
98% of teenagers do or have tried caving into peer pressure. If you're one of the 2% who hasn't, DO NOT copy & paste this in your signature.

Post #365 · Posted at 2022-12-28 07:05:24pm 1.3 years ago

Offline travelsonic
travelsonic Avatar Member
190 Posts
Not Set
Reg. 2006-10-18


Last updated: 2022-12-28 07:07pm
Quote: SpyHunter29
You're quite welcome, travelsonic! Thanks, in turn, to root670 on github, and to the comments they added to their version of the decompression code (https://github.com/root670/ddr-tools).

I admit it's been a while since I looked at Root670's stuff, but that's neat - gotta take a look at that. IMO he's definitely been a boon, a big help to reverse engineering efforts (even down to figuring out the version of DWARF the leftover debugging data leftover in the various PS2 games used haha)

Post #366 · Posted at 2022-12-28 08:21:55pm 1.3 years ago

Offline CuzcoBlocko
CuzcoBlocko Avatar Member
2,947 Posts
United States
Reg. 2013-10-26

"[Art by LilyBreez]"
Quote: SpyHunter29
Not really. I've seen structures like those pretty much only from what I've read in this thread. In fact, I did try compiling and running Dwarf2CPP on some of the games' ELF files, but it kept crashing when trying to write the second file (which always happens to be les_sel_lv.c). I also gave Ghidra a try to decompile the ELFs and/or arcade binaries, but no such luck at the moment.

On the plus side, I have finally figured out how the video filenames are encoded at the end of the scripts. First, since the system reads 32-bit values like these in little-endian, we will need to reverse the byte order before converting the video's hex code to binary (using jrlova/MAX1.avi as an example):

Quote
292E5701 -> 01572E29 -> 00000001 01010111 00101110 00101001

Once we have our binary string, split it up into groups of 5 bits each. There will be two spare bits left over; these can be ignored. It is important to remember that bits are read from right to left, so start from the right end, like so:

Quote
00000001 01010111 00101110 00101001 -> 00 00000 10101 01110 01011 10001 01001

Now that we have our binary string cut up like so, we can translate it into letters. Simply convert each binary slice into a decimal number -- again, starting from the right -- and convert that number into a letter, starting with 0=a, 1=b, and so on through 25=z.

Quote
01001 10001 01011 01110 10101 00000 -> 9, 17, 11, 14, 21, 0 -> jrlova

And there you have it!

Does this mean jsedca's SSQ value is actually known as 49922100?

Post #367 · Posted at 2022-12-30 12:57:18am 1.2 years ago

Offline SpyHunter29
SpyHunter29 Avatar Member
331 Posts
United States
Reg. 2008-06-09

Yup, that's what I got when I worked it out again for myself.

jsedca... Oh yeah, that's one of those clips that was never used outside of... idk, Ultramix or something, if anything.
98% of teenagers do or have tried caving into peer pressure. If you're one of the 2% who hasn't, DO NOT copy & paste this in your signature.

Post #368 · Posted at 2023-01-03 10:00:37am 1.2 years ago

Offline 幸福一家王小龙
幸福一家王小龙 Avatar Member
65 Posts
China
Reg. 2022-03-13

What is the extractor software?

Post #369 · Posted at 2023-01-06 01:25:21pm 1.2 years ago

Offline SpyHunter29
SpyHunter29 Avatar Member
331 Posts
United States
Reg. 2008-06-09

If you're asking about just extracted the data itself from the PS2 DDR discs, I used the filedata-tool.py Python script from https://github.com/root670/ddr-tools.

I should probably make a fork of that repository sometime, and add in the scripts I created myself. I'll let y'all know once I get to that.
98% of teenagers do or have tried caving into peer pressure. If you're one of the 2% who hasn't, DO NOT copy & paste this in your signature.

Post #370 · Posted at 2023-01-06 06:22:57pm 1.2 years ago

Offline travelsonic
travelsonic Avatar Member
190 Posts
Not Set
Reg. 2006-10-18

Quote: SpyHunter29
If you're asking about just extracted the data itself from the PS2 DDR discs, I used the filedata-tool.py Python script from https://github.com/root670/ddr-tools.

I should probably make a fork of that repository sometime, and add in the scripts I created myself. I'll let y'all know once I get to that.

I've also been tempted to put up on GitHub a version of the data extraction tools that I've been working on (albeit in C) that will allow for people the choice of loading a pair of executable + filedata.bin, OR an ISO image. IRL has been hell lately so I've yet to get it finished though. ~_~

Post #371 · Posted at 2023-01-08 02:30:04pm 1.2 years ago

Offline SpyHunter29
SpyHunter29 Avatar Member
331 Posts
United States
Reg. 2008-06-09


Last updated: 2023-01-08 02:30pm
Jumping back to the topic of debugging and the DWARF headers, I was able to compile that DWARF-to-CPP program (once I downloaded a newer fork of that repository) and generate all the DWARF C files from some games. Having done so, does anyone know how to use them, or the debug symbol files posted by root676 a while back (https://zenius-i-vanisher.com/v5.2/thread?threadid=7564&page=16#p436618), in Ghidra? And when I import the games' ELF files, what language should I read them as? I'm pretty sure it's one of the MIPS 32-bit languages, but I'm still fuzzy about little/big-endian and other options. Pardon my questions; I'm still kind of a total noob when it comes to Ghidra.
98% of teenagers do or have tried caving into peer pressure. If you're one of the 2% who hasn't, DO NOT copy & paste this in your signature.

Post #372 · Posted at 2023-01-09 01:58:56pm 1.2 years ago

Offline travelsonic
travelsonic Avatar Member
190 Posts
Not Set
Reg. 2006-10-18


Last updated: 2023-01-09 03:47pm
Quote: SpyHunter29
Jumping back to the topic of debugging and the DWARF headers, I was able to compile that DWARF-to-CPP program (once I downloaded a newer fork of that repository) and generate all the DWARF C files from some games. Having done so, does anyone know how to use them, or the debug symbol files posted by root676 a while back (https://zenius-i-vanisher.com/v5.2/thread?threadid=7564&page=16#p436618), in Ghidra? And when I import the games' ELF files, what language should I read them as? I'm pretty sure it's one of the MIPS 32-bit languages, but I'm still fuzzy about little/big-endian and other options. Pardon my questions; I'm still kind of a total noob when it comes to Ghidra.

Little endian. Big Grin

There is a plugin (or two? My mind's been a bit scattered lately) for Ghidra that adds support for the Emote Engine + R5900 - but, IIRC, only really go up as far up as 10.0.3 10.0.4 in terms of versions supported, short of recompiling the plugin to make it work with more recent versions.

When it comes to Ghidra, and its decompiler, be aware one annoying as hell thing (that, if not addressed already I hope will be in the future) - which is the tendency for Ghidra to shortcut decompilation output by lumping statements together by use of the comma (',') operator - which isn't terribly hard to figure out how to unravel at times, except for when it is - it definitely hurts readability of decompiler output when it uses this a lot.


After using dwarf2cpp I just used Parse C Source - which I don't think added the function names automatically (it having no awareness of the address-to-function-name relations), but it did mostly successfully import the data structures correctly.
(Unless I, like an idjit, messed something up of course).

Actually, that gave me a thought - the guy who made Dwarf2cpp said he was working on a new version of the program that would preserve in the comments, at least, the addresses of the functions AND memory addresses of global + (where applicable) local variables, maybe if that comes to fruition, someone can made a plugin that adds the ability to use that preserved address information to automatically apply the function and parameter names to the appropriate function(s) in the decompilation/disassemblies (and variable names / types to memory addresses for local and global functions)!

Post #373 · Posted at 2023-01-19 06:11:27am 1.2 years ago

Offline CuzcoBlocko
CuzcoBlocko Avatar Member
2,947 Posts
United States
Reg. 2013-10-26

"[Art by LilyBreez]"

Last updated: 2023-01-19 06:39am
Quote: SpyHunter29
The first byte contains a bunch of flags which affect timing.
- If the lower half of the byte is 3, (i.e., the byte ends in a 3), pause the clip.
- If the lower half is 5, play the clip normally.
- If the lower half is 6, play the clip normally, then in reverse, and so on.
- If the lower half is 9, play the clip in reverse.
- If the lower half is A, play the clip in reverse, then normally, and so on (opposite of 0x_6).

- If the upper half is 1 (i.e., the byte starts with a 2), play the clip at normal speed.
- If the upper half is 2, play the clip at 0.75x speed.
- If the upper half is 3, play the clip at half speed.
- If the upper half is 4 and the third byte is 0x00, play the clip at 1.5x speed.
- If the upper half is 4 and the third byte is not 0x00, play the clip from start to finish for a given length of beats specified in the third byte, adjusting the speed as necessary. In this example, a value of 0x04 means the clip will play start-to-finish within 4 beats at the song's current tempo, while 0x08 will make the clip last for 8 beats, or at half the speed of 0x04.
- If the upper half is 5, play the clip upon finishing the previous one. Used when the accompanying beat entry is a duplicate, for seamless off-beat transitions. If the previous entry has specific timing instructions, they are carried over to this entry.
- If the upper half is 6, start the clip at the frame specified in the third byte.
- If the upper half is 9, display the song's BG image. I don't know what effect, if they, that the other bytes have here.
- If the upper half is A, play the clip at 1.5x speed.

The second byte points to which video clip will be played, as described in the section below. However, there are some exceptions:
- 0x14 represents the song's background image, typically shown at the end of the song.
- 0x15 through 0x1D represent shared generic videos. Most of these were replaced from MAX2 onwards.
- 0x66 and up represent console-exclusive videos, such as those used in Kind Lady or So In Love. Like the common videos, these are not included in the section below.

The third byte controls the video clip's speed. All clips are 80 frames long and, at least for NTSC-region games, play at 30 frames per second by default. This works out to running for four beats at 90 BPM. However, they can also be sped up to match the BPM of the song and play for a specific number of beats. In this example, a value of 0x04 means the clip will play start-to-finish within 4 beats at the song's current temp (about 130.1 BPM).

The fourth byte is not used, at least as far as I have seen so far.

There has to be a way to get the clip to start at a specified frame and also play at a specified rate. One example is teapot1, or the second half of jcteap. In Odoru Ponpokorin in DDR Festival, it looks to be going at 140 BPM just fine without also having drawer1 attached to it.



This is also true for JANEJANA in DDR Extreme, arcade, console JP, and console US.

Post #374 · Posted at 2023-02-13 02:23:54pm 1.1 years ago

Offline SpyHunter29
SpyHunter29 Avatar Member
331 Posts
United States
Reg. 2008-06-09

Quote: CuzcoBlocko
There has to be a way to get the clip to start at a specified frame and also play at a specified rate. One example is teapot1, or the second half of jcteap. In Odoru Ponpokorin in DDR Festival, it looks to be going at 140 BPM just fine without also having drawer1 attached to it.

In the case you asked about, the script is actually playing the whole video file in reverse, but rewinding to the beginning (or end, looking at the clip in forwards order) just before reaching the other half of the clip. The same script in StepMania's format would look like this:

55.750=jcteapr.avi=1.553=0=1=1,
57.750=jcteapr.avi=1.553=0=1=1,


In fact, most of the clips from Beware's pack which came from these combined videos are the reversed versions, since that's how they were used most often in-game. Similarly, the first half of the video can be looped at any given speed by doing the same thing, just with the clip running forwards instead of backwards.
98% of teenagers do or have tried caving into peer pressure. If you're one of the 2% who hasn't, DO NOT copy & paste this in your signature.

Post #375 · Posted at 2023-02-22 03:04:37am 1.1 years ago

Offline CuzcoBlocko
CuzcoBlocko Avatar Member
2,947 Posts
United States
Reg. 2013-10-26

"[Art by LilyBreez]"

Last updated: 2023-02-22 03:04am
Oh what the fuck are you serious. I thought the teapots moved left in the original clip. Dammit.

...shit. Does that mean there is no way to have both of those features at the same time in the original game?

Maybe check LONG TRAIN RUNNIN's script for any byte features you may have missed?


JANEJANA plays teapot1 with the teapots going to the left at a speed that isn't 0.75x, 1x, or 0.5x.

Post #376 · Posted at 2023-04-02 08:48:50am 1 year ago

Offline travelsonic
travelsonic Avatar Member
190 Posts
Not Set
Reg. 2006-10-18


Last updated: 2023-04-02 09:04am
Here's something I am (perhaps foolishly?) confused about: Why in EXTREME 2 for instance, is some of the game's code loaded into memory dynamically - such as Dance Master Mode's functions?

I noticed this after pondering why my attempted disassembly in Ghidra of the 2005-05-10 EXTREME 2 prototype was hampered by memory addresses that should reference functions pointing to empty bytes - and observing the load behavior (and the filling of those memory addresses w/ executable code) whilst running the game in PCSX2Dis... I am struggling to think of a reason to do this - as opposed to keeping all the game function code together in one executable. My first thought was file size constraints, but that doesn't seem to make sense to me... perhaps (again) I'm overlooking something glaringly obvious.

Quote: Wan
Well, now that we're talking about CS games, I'm still trying to figure out where the files are in the PS2 structure.

- In the Arcade games, the data is in GAME.DAT, and its file table is at a specific address in that file (reading sys573tools, it's 0xFE4000). The actual executable, aout.exe is in it.

This definitely seems to be the case for digital mixes, but not necessarily for analog mixes? When I go to that offset in analog mix GAME.DAT file, I think, I'm not finding it. Any luck in finding where the offset for the table for analog mixes? (Am I being dumb, or missing something heh)

Post #377 · Posted at 2023-05-04 11:19:43am 11.6 months ago

Offline Arcorann
Arcorann Avatar Member
59 Posts
Not Set
Reg. 2015-01-29

Is there any info on where the code for determining judgements and the corresponding timing windows are in the various mixes? I've been wondering whether the algorithms have changed over time (both on console and arcade versions).

Post #378 · Posted at 2023-05-04 01:41:53pm 11.6 months ago

Offline travelsonic
travelsonic Avatar Member
190 Posts
Not Set
Reg. 2006-10-18


Last updated: 2023-05-04 01:42pm
Quote: Arcorann
Is there any info on where the code for determining judgements and the corresponding timing windows are in the various mixes? I've been wondering whether the algorithms have changed over time (both on console and arcade versions).

This - the specifics of how the timing windows are utilized - is an area I definitely want to tackle - have been meaning to for some time... #adhdProblems Laughing Out Loud

(strictly speaking from C/S mixes, though I think AC mixes will handle things similarly) I will say that the locations will physically differ from mix to mix - but the layout of the window data should remain consistent in terms of there being 2 arrays of signed 32-bit values - one being the upper bounds of each timing window, the other being the lower bounds of each timing window. Perhaps a bit later (got stuff to do despite being off from work today) I should compile a list of physical and logical addresses for each array in the various CS mixes (retail, prototype, AND demo disc versions to boot) I've been looking into (logical addresses, ofc, being for easy location in tools like IDA, Ghidra, etc - physical addresses for if you want to muck with them in a hex editor)

Post #379 · Posted at 2023-05-14 01:52:46am 11.2 months ago

Offline Arcorann
Arcorann Avatar Member
59 Posts
Not Set
Reg. 2015-01-29


Last updated: 2023-05-16 08:55am
So, about DDR Extreme AC's timing windows. I was rereading Sesse's post from about 7 pages back and 3 years ago, and decided that we could put more work into checking this.

Started off by wanting to verify the numbers listed. Conveniently, this Aaron in Japan forum thread from 2004 has the same numbers, but Taren was unable to work out the correct interpretation.

I've located them now in GAME.DAT: offset 0x085DB8 for the early windows and 0x085DD4 for the late windows.

If Sesse's analysis is correct, we should observe some jitter in judgement evaluations in game, since the windows are not an integer number of frames. By aligning 150 Hz ticks with 60 Hz frames, we are able to enumerate all the possibilities of frame alignment:

https://cdn.discordapp.com/attachments/739814736421715988/1107117490229760101/image.png

These are theoretical but fairly easy to verify with MAME; I've only verified column 5 so far, but it shouldn't be hard to get the rest.

EDIT: Verified columns 2 and 5.

--------

Also, here are the correct conversions into milliseconds:

Marvelous: -16.667 ~ +16.667
Perfect: -30.000 ~ +30.000
Great: -90.000 ~ +76.667
Good: -130.000 ~ +116.667
Boo: -156.667 ~ +183.333

--------

Figured I might as well search for timing windows in other arcade mixes, starting with 5thMIX. In GAME.DAT offset 0x0660E0: FC FF FF FF F3 FF FF FF ED FF FF FF E9 FF FF FF (-4, -13, -19, -23); offest 0x0660F8: 04 00 00 00 0B 00 00 00 11 00 00 00 1B 00 00 00 (4, 11, 17, 27). These are the same as Extreme except without the Marvelous window.

Post #380 · Posted at 2023-05-15 01:36:19pm 11.2 months ago

Offline travelsonic
travelsonic Avatar Member
190 Posts
Not Set
Reg. 2006-10-18

Quote: Arcorann

If Sesse's analysis is correct, we should observe some jitter in judgement evaluations in game, since the windows are not an integer number of frames. By aligning 150 Hz ticks with 60 Hz frames, we are able to enumerate all the possibilities of frame alignment:

POardon me if this is a silly question, or if I am misunderstanding, but basically, its based off of ticks and refresh rate, rather than straight frames? (again, perhaps a suilly question, my brain isn't working right this morning haha)
Register Log In Back To Forums

0 User(s) Viewing This Thread (Past 15 Minutes)

©2006-2024 Zenius -I- vanisher.com -5th style- IIPrivacy Policy
Web Server: 3% · Database: 4% · Server Time: 2024-04-16 17:54:42
This page took 0.023 seconds to execute.
Theme: starlight · Language: englishuk
Reset Theme & Language