decompile songs to MIDI files #137

laqieer · 2022-01-04T14:47:34Z

All Midis Match Completed!🎉

Status: 🆗

ℹ️Information on build

It is a huge change so run make clean before pull from remote to your local machine, or re-clone a clean git repo to build it. Remember to rebuild tools/mid2agb too.

Practical experience for other disasm/decomp projects

It is a nightmare to match MIDI files. Common problems and solutions are collected here. Hope this doc will make life easier for others.

Steps summarized by freshollie

Extract sound data with Normatt's script, ensure still creates a matching build
Extract midis using VG Music Studio, preferably the hacked version as this ensures all the data is written to the midis. Ensure that the reverseVolume argument is enabled, and use the outputted debug info to add the base volume argument to the existing songs.mk
Dump midis into repo and use a script to match the ones which don't need work, I hacked something together to do this
Categories the remaining midis on what needs changing. Diff the compiled midi against the existing data to see if a tempo change will correctly line up the whole note marks (---). Use the edit_time_signature script to change this and recompile.
Some songs may have volume and KEYSH switched around, for sa2 this was fixed by removing the first volume command if it was the base volume
For midis which have a wait at the start, change the saveAsMidi args in VG Music Studio to add the number of ticks before the time signature starts. If W96 (96, (4, 4)) for example.
Match remaining midis by changing time signatures at the places where the notes stop lining up, this is the hardest step. The args in VG Music Studio can contain multiple time signatures and where they start.
Other weird cases, check the doc on all the possible work arounds

…ripper.

… doesn't match.

FireEmblemUniverse#143

It is an old gcc behaviour to pass information to gdb. It causes duplicated symbols at the same address which is confusing. pret/agbcc#50

laqieer · 2022-04-28T07:27:39Z

tools/mid2agb/mid2agb sound/songs/midi/song001.mid sound/songs/midi/song001.s -E  -V046 -G000 -R020 -P010
error: invalid event
make: *** [sound/songs/midi/song001.s] Error 1

sha1sum -c checksum.sha1
fireemblem8.gba: FAILED
sha1sum: WARNING: 1 computed checksum did NOT match
make: *** [Makefile:81: compare] Error 1

laqieer/VGMusicStudio@997a4f7

laqieer · 2022-04-28T15:50:34Z

It doesn't match yet.

The size of song converted from midi is often bigger than vanilla, which causes address offset.
For example:
Vanilla song002:

 sound/songs/song002.o(.rodata)
 .rodata        0x00000000085462b0     0x17ec sound/songs/song002.o

Converted song002:

 sound/songs/midi/song002.o(.rodata)
 .rodata        0x00000000085462b0     0x1830 sound/songs/midi/song002.o

sound/songs/song002.s: vanilla
sound/songs/midi/song002.mid: dumped by hacked VGMusicStudio with note doesn't end fix
sound/songs/midi/song002.s: converted from dumped midi by unmodified mid2agb

For example, Wxx(wait command) may be different:
Vanilla song002:

	.byte	W96
	.byte	W96
	.byte	W48
	.byte		EOT	
	.byte		N48	, Gn4
	.byte	W48
	.byte		N72	, Gs4
	.byte	W96
	.byte	GOTO	
		.word	song002_1_1
	.byte	FINE

	@********************** Track  2 **********************@

	.byte		v035
	.byte	W03
	.byte		v035

	.byte	W03
	.byte		v037
	.byte	W01
	.byte	W03

Converted song002:

	.byte	W72
@ 045   ----------------------------------------
	.byte	W96
@ 046   ----------------------------------------
	.byte	W72
	.byte		EOT   
	.byte		N48   , Gn4 
	.byte	W24
@ 047   ----------------------------------------
	.byte	W24
	.byte		N72   , Gs4 
	.byte	W72
@ 048   ----------------------------------------
	.byte	W24
	.byte	GOTO
	 .word	song002_1_B1
song002_1_B2:
	.byte	FINE

@**************** Track 2 (Midi-Chn.2) ****************@

	.byte		        35*song002_mvl/mxv
	.byte	W01
@ 001   ----------------------------------------
	.byte	W02
	.byte		        35*song002_mvl/mxv

	.byte	W03
	.byte		        37*song002_mvl/mxv
	.byte	W04

They have same duration but different bytes:
96 + 96 + 48 = 72 + 96 + 72
96 = 72 + 24
3 = 1 + 2
1 + 3 = 4

Note: It produces the same problem to dump midi files by unmodified VGMusicStudio and convert them by hacked mid2agb with note doesn't end fix, so it is not related to the note doesn't end fix.

It is caused by incorrect Time Signature, because it is not always 4/4. We need to fix it to match one by one.

PikammunityALT — 2019/10/11
how do i force mid2agb to compile W07 W24 instead of W30 W01?
relevant midi bytes:
07 82 2C 00 18 FF 2F 00
offset 16D in mus_fanfa4.mid 
Deleted User — 2019/10/11
probably time signature related
which is the reason for manual dumping of these
you'd have to insert a time signature at the spot between w07 and w24 which would then shift the rest of the midi depending on the denominator

I didn't fix them because I won't use it anymore.

laqieer · 2022-04-28T23:41:44Z

The songs with different size after conversion are splitted to non_matching folder.

Some songs with same size are still different in several bytes.

Most of them have unordered commands. I added @freshollie's workarounds to mid2agb to match some more cases where some of the original compiled midi tracks had a different order to the VOL and KEYSH commands, but the order of those commands is still incorrect sometimes.

Guess: some songs especially for sound effect in Fire Emblem series are converted from MML by MML2AGB, not from MID by MID2AGB. Its order of commands are decided by the order written in .mml files directly.

Some songs need to align all tracks while others not. Add -A option to mid2agb to enable track alignment. https://github.com/FireEmblemUniverse/fireemblem8u/blob/94d53ef4d521f3f305412d5244d84cf68f411aa4/scripts/dump_sound/Program.cs#L306

laqieer · 2022-04-29T08:11:17Z

song027 doesn't match due to missing patterns. It affects compression of velocity in Nxx commands too.

Vanilla:

song027_2_4:
	.byte		N21	, Dn2, v127
	.byte	W24
	.byte		N21	
	.byte	W24
	.byte		N21	
	.byte	W24
	.byte	PEND
	.byte	PATT	
		.word	song027_2_3
	.byte	PATT	
		.word	song027_2_4
	.byte	PATT	
		.word	song027_2_5
song027_2_6:
	.byte		N44	, Dn2, v127
	.byte	W48
	.byte		N44	
	.byte	W48
	.byte	PEND
	.byte	PATT	
		.word	song027_2_6

Midi
Converted:

	.byte		        Dn2 
	.byte	W24
	.byte		N21   
	.byte	W24
	.byte		N21   
	.byte	W24
@ 011   ----------------------------------------
	.byte	PATT
	 .word	song027_2_003
@ 012   ----------------------------------------
song027_2_012:
	.byte		N44   , Dn2 , v127
	.byte	W48
	.byte		N44   
	.byte	W48
	.byte	PEND

laqieer · 2022-04-29T10:01:58Z

song067 doesn't match due to only a command order difference: its KEYSH appears after VOL at the beginning of track 7, while other tracks has a first KEYSH. Why?

fireemblem8u/sound/songs/song067.s

Line 2276 in 94d53ef

.byte KEYSH , 0

Guess: it seems that the behaviour of pret's mid2agb doesn't match the official one.

Official version:
https://github.com/Touched/mid2agb/blob/27c03fe8a7f482d80ce792f31dee80b0b441f947/sfile.c#L287
https://github.com/pret/pokeruby/blob/1380ad46772737fd9a21a4ff4be9f61d81b59c4b/tools/mid2agb/midi.h#L47
https://github.com/Touched/mid2agb/blob/27c03fe8a7f482d80ce792f31dee80b0b441f947/unused.c#L1034
https://github.com/Touched/mid2agb/blob/27c03fe8a7f482d80ce792f31dee80b0b441f947/unused.c#L1051
KEYSH command comes from midi event: 0x31 (49), which is added in print_sfile_body() -> mystery1() -> sub_4049D0()

PRET's version:
pret/pokeemerald@f613176
KEYSH command is auto written to the head of track anyway

Why?

laqieer · 2022-04-29T18:40:15Z

List of all songs with names

"" = an empty slot with a dummy song
s??? = a song converted from .mid file by MID2AGB
m??? = a song converted from .mml file by MML2AGB

laqieer · 2022-04-30T13:31:13Z

How to fix KEYSH order

Take song067 as example: 65ad237

Reason

If the volume of a track is the same as the max volume of the song, then it can be omitted, and mid2agb will add it back automatically before KEYSH. A dumper such as VGMusicStudio doesn't know the max volume of the song, so it doesn't do the omit for a track, and mid2agb will add KEYSH before the volume at the head of a track.

Note: it can be omited doesn't mean it must be omitted. It is totally alternative. Both are valid:

omit volume in .mid <=> VOL KEYSH order in .s
keep volume in .mid <=>KEYSH VOL order in .s

Fix it only when the order doesn't match, otherwise they will match at first obviously.

Solution

Set the max volume of the song correctly, adjust all volumes in the song, and delete the volume at that track.

How to decide the max volume of the song?

There are 2 ways:

Volume can only be omitted at the track whose volume is equal to the max volume of the song, so volume of the track with reversed order of KEYSH and VOL at the head is the max volume of the song.
For example:

fireemblem8u/sound/songs/song067.s

Line 2275 in 94d53ef

.byte VOL , v050
mks4agb reads the max volume of a song from mks4agb.ini and passes it to mid2agb with -V option, so the max volume of all songs are written here: decompile songs to MIDI files #137 (comment)
For example:
s067 = .\agbfe3_new\bgm\bgm_haru\agbfe3_bgm_op_theme, 0, 02, 50, 10, 20 ;ＦＥテーマ（主題部）

How to fix volumes in midi?

We need to calculate the new volume:
volume_in_midi = volume_in_s * 127 / max_volume
It may not be divisible, so re-check the volume:
volume_in_s = volume_in_midi * max_volume / 127
Adjust volume_in_midi until it the result matches volume_in_s
You can edit volume in .mid file using a midi editor such as MidiQuickFix.

That is a tedious work for all volume commands in all midi files with the same problem, so we can use a dumper such as VGMusicStudio to do the volume fix described above. For example, set ReverseVolume option when export Midi: Kermalis/VGMusicStudio@41323d6, and it will calculate volume automatically: Kermalis/VGMusicStudio@6dd3d17.

Reference

mid2agb's behaviour on the omitted volume before the first note in the track

…ollie/sa2." This reverts commit 1105344.

laqieer · 2022-04-30T17:03:35Z

KEYSH workaround from sa2 was reverted because it causes mismatch of the position of KEYSH in some cases such as song075. They get matched after the revert.

laqieer · 2022-05-03T20:05:36Z

How to fix missing patterns

mid2agb has a complex compression algorithm to find patterns. It is calculated according to time splits, so you can add extra time signatures before patterns to manipulate it. You can use a Midi dumper like VGMusicStudio or a Midi editor like MidiQuickFix to add it. Here are some examples: 8d06134, a0bb889, 3dc0b67, 0f343ee.

laqieer · 2022-05-03T20:16:32Z

How to fix length

Some redundant waits after GOTO at the end of all tracks.

Reason

It is caused by Kermalis/VGMusicStudio@997a4f7 hack for Kermalis/VGMusicStudio#36 which has already been removed.

Solution

It is really easy to fix it with any midi editor or re-export it.

Example

8d06134

laqieer · 2022-05-03T20:25:21Z

How to split wait command

Problem

Solution

Matched this with a load of dummy note events 🤦‍♂️

This project doesn't have this case. Provided by @freshollie. Refer to SAT-R/sa2@9133045 if interested. For memo purpose.

freshollie · 2022-05-04T09:46:12Z

How to fix missing patterns, alternative way

Problem

mid2agb will not compress a pattern if its contents doesn't meet a given threshold of complexity. However, after running the compression algorithm, mid2agb will potentially strip some of the midi instructions in the pattern, which it doesn't handle, out of the final assembly.

For example:
(Left - original, Right - compiled after extracting from ROM)

The pattern extracted from the ROM is not complex enough (3 instructions) for the compression algorithm in mid2agb to consider compressing, but the original did have enough complexity to compress. We have to assume there was more in this pattern (as it's too small to be fixed with time signature changes)

Solution

Add a dummy instruction to the pattern (all places where we expect it to be used) which mid2agb will consider in the compression.

To fix the above example, a foot control command was inserted into the midi in both places where the pattern should be compressed:

The instruction doesn't appear in the final assembly because mid2agb doesn't handle this instruction

laqieer added 9 commits January 4, 2022 22:31

Dump songs to MIDI files.

49c1d2c

Dump MIDI files with camthesaxman/pokeruby_mus_riper fork of gba-mus-…

6af0963

…ripper.

Add MPlayDef.s.

8b6fe81

Fix dummy song.

7355db6

Make MIDI files to build.

fd9ed69

Replace song001.mid with the one dumped by VGMusicStudio but it still…

c0c04b5

… doesn't match.

Remove useless line endings conversion for .pal files in Makefile.

f015818

FireEmblemUniverse#143

Remove label gcc2_compiled.

2963c4c

It is an old gcc behaviour to pass information to gdb. It causes duplicated symbols at the same address which is confusing. pret/agbcc#50

Merge branch 'FireEmblemUniverse:master' into master

b4c3111

laqieer added 2 commits April 28, 2022 23:03

Fix make clean.

56ec986

Fix: note doesn't end.

0ec9639

laqieer/VGMusicStudio@997a4f7

laqieer added 3 commits April 29, 2022 06:15

Add some workarounds to mid2agb from https://github.com/freshollie/sa2.

1105344

Comment some bugs in sound dump script to generate songs.mk.

b211702

I didn't fix them because I won't use it anymore.

Split songs with different size to non_matching folder.

fb1f6ba

Add workaround to fix track alignment.

5ebc938

Some songs need to align all tracks while others not. Add -A option to mid2agb to enable track alignment. https://github.com/FireEmblemUniverse/fireemblem8u/blob/94d53ef4d521f3f305412d5244d84cf68f411aa4/scripts/dump_sound/Program.cs#L306

laqieer referenced this pull request in laqieer/VGMusicStudio Apr 29, 2022

add hacks to assist in gba midi extraction

997a4f7

Fix Makefile.

74c58f4

Match song067.

65ad237

Revert "Add some workarounds to mid2agb from https://github.com/fresh…

ef859ec

…ollie/sa2." This reverts commit 1105344.

laqieer added 3 commits May 1, 2022 03:11

Re-export all midis with VGMusicStudio's reverse volume feature on.

13a8c4d

Fix volumes set in songs.mk.

bdbe480

Fix song067's KEYSH order.

1854281

laqieer added 17 commits May 3, 2022 13:47

Change song014's time signature to 3/4.

1e87958

Change song018's time signature to 3/4.

84da87d

Change song023's time signature to 3/4.

b20e003

Change song029's time signature to 3/4.

8313e75

Change song031's time signature to 3/4 +24.

f8f663b

Change song032's time signature to 3/4.

637cf1b

Shift song023's time signature by 24.

55df13f

Change song044's time signature to 3/4.

a69742e

Change song045's time signature to 3/4.

482c031

Change song046's time signature to 3/4 +24.

0bd2a06

Shift song066's time signature by 24.

9141bbd

Change song068's time signature to 3/4.

645411b

Shift song069's time signature by 24.

6f78c2d

Match song027 by adding time signatures and fixing its length.

8d06134

Match song028 by adding time signatures.

a0bb889

Match song038 by adding time signatures.

3dc0b67

Match song057 by adding time signatures.

0f343ee

laqieer added 5 commits May 4, 2022 04:46

Merge branch 'FireEmblemUniverse:master' into master

4be270d

Merge branch 'master' into midi

4ac4ae6

Move song814, song815, song816, song962 to mml folder.

5464368

Rename all songs to meaningful names.

a5e5dba

Remove songs.ini.

4dd7fa0

laqieer changed the title ~~Dump songs to MIDI files (not ready)~~ decompile songs to MIDI files May 3, 2022

laqieer marked this pull request as ready for review May 3, 2022 22:24

RevoSucks merged commit 15a2e5d into FireEmblemUniverse:master May 3, 2022

laqieer deleted the midi branch May 4, 2022 14:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

decompile songs to MIDI files #137

decompile songs to MIDI files #137

laqieer commented Jan 4, 2022 •

edited

Loading

laqieer commented Apr 28, 2022 •

edited

Loading

laqieer commented Apr 28, 2022 •

edited

Loading

laqieer commented Apr 28, 2022 •

edited

Loading

laqieer commented Apr 29, 2022

laqieer commented Apr 29, 2022 •

edited

Loading

laqieer commented Apr 29, 2022

laqieer commented Apr 30, 2022 •

edited

Loading

laqieer commented Apr 30, 2022

laqieer commented May 3, 2022

laqieer commented May 3, 2022 •

edited

Loading

laqieer commented May 3, 2022 •

edited

Loading

freshollie commented May 4, 2022 •

edited

Loading

decompile songs to MIDI files #137

decompile songs to MIDI files #137

Conversation

laqieer commented Jan 4, 2022 • edited Loading

ℹ️Information on build

Practical experience for other disasm/decomp projects

Steps summarized by freshollie

laqieer commented Apr 28, 2022 • edited Loading

laqieer commented Apr 28, 2022 • edited Loading

laqieer commented Apr 28, 2022 • edited Loading

laqieer commented Apr 29, 2022

laqieer commented Apr 29, 2022 • edited Loading

laqieer commented Apr 29, 2022

laqieer commented Apr 30, 2022 • edited Loading

How to fix KEYSH order

Reason

Solution

How to decide the max volume of the song?

How to fix volumes in midi?

Reference

laqieer commented Apr 30, 2022

laqieer commented May 3, 2022

How to fix missing patterns

laqieer commented May 3, 2022 • edited Loading

How to fix length

Reason

Solution

Example

laqieer commented May 3, 2022 • edited Loading

How to split wait command

Problem

Solution

freshollie commented May 4, 2022 • edited Loading

How to fix missing patterns, alternative way

Problem

Solution

laqieer commented Jan 4, 2022 •

edited

Loading

laqieer commented Apr 28, 2022 •

edited

Loading

laqieer commented Apr 28, 2022 •

edited

Loading

laqieer commented Apr 28, 2022 •

edited

Loading

laqieer commented Apr 29, 2022 •

edited

Loading

laqieer commented Apr 30, 2022 •

edited

Loading

laqieer commented May 3, 2022 •

edited

Loading

laqieer commented May 3, 2022 •

edited

Loading

freshollie commented May 4, 2022 •

edited

Loading