Improve Reko's handling of compiler emitted boiler-plate entry points

[ptomin]

WinMain is not entry point of PE .exe binary.
Let's see RussianText.exe. Its entry point is at 0x401018. Its main function (int main(int argc, char* argv[]), not WinMain) is at 0x401168 (it's not discovered by Reko but it is another issue)

[ptomin]

The sequence of instructions seems to be like
<entry point>:
..........some code...........
<call of main/WinMain, may be indirect>
..........some code...........

[ptomin]

There is call of __startup at the end of RussianText.exe entry function.
__startup is Borland function. MODULE_DATA structure is passed to __startup

typedef struct module_data
{
    INIT *init_start;           /* start of a module's _INIT_ segment */
    INIT *init_end;             /* end of a module's _INIT_ segment */
    INIT *exit_start;           /* start of a module's _EXIT_ segment */
    INIT *exit_end;             /* end of a module's _EXIT_ segment */
    int  flags;                 /* flags */
    int  hmod;                  /* module handle */
    int  (*main)();             /* main/WinMain/_dllmain function */
    int  (*matherr)(void *);    /* (EXE only) _matherr function */
    int  (*matherrl)(void *);   /* (EXE only) _matherrl function */
    long stackbase;             /* (EXE only) base of stack */
    int  *fmode;                /* (EXE only) address of _fmode variable */
} MODULE_DATA;

[ptomin]

This is right only for Borland (not sure about differences between versions of Borland linker), of course. Visual C and other compilers are used other way to start *.exe file

[uxmal]

And this works only for x86 on Win32. A more general solution is required to dig out the WinMain / main program from the C startup code (which typically isn't interesting). Other decompilers do pattern matching of various sorts to figure this out. We may want to look at their solutions to see if any could be applied to Reko.

[ptomin]

But PeImageLoader changes for .exe (not sure about dlls) introduced in 7c49dcc should be undone
It marks entry point as WinMain. it is incorrect

[uxmal]

It's tricky. The entry point for a Win32 PE executable does have that signature, but it isn't the "real" WinMain that most programmers are working with. It probably should be called Win32CrtStartup or something instead. And, if reko can identify it as CRT "fluff", reko should be able to locate the (possibly indirect) call to the "real" WinMain, and just decompile that instead.

[uxmal]

So it seems we have a few issues to resolve here

• Rename the entry points generated by 7c49dcc to Win32ExeStartup and Win32DllStartup as appropriate
• Implement a mechanism to detect the CRT startup stubs that different compilers inject into a binary
• Adapt reko so that if it detects this startup stub, it will locate the "real" main function, and start decompilation there.

[uxmal]

And finally, the big question: do think this is worth postponing the 0.6.0.0 release to implement this? There are obviously workarounds (manually selecting the "real" WinMain and marking it), but it is nice to have Reko perform this task automatically.

My personal opinion is to finish 0.6.0.0, which has a lot of GUI work that I want to release, and then focus on more code-generation stuff in the next release.

[ptomin]

It probably should be called Win32CrtStartup

I agree

Win32 PE executable does have that signature

Really? Are you sure that crt startup has

int <func-name>(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, INT CmdShow)

signature? I'm not sure but it looks like crt startup has not arguments. It gets hInstance, lpCmdLine and other arguments from Win API functions and passes them to WinMain

[uxmal]

You're right, Pavel, I was thinking of Win16 (yes, I'm that old). According to Raymond Chen (https://blogs.msdn.microsoft.com/oldnewthing/20110525-00/?p=10573/) the signature is

DWORD CALLBACK RawEntryPoint(void).

It seems to me that fixing all these entrypoint items will have to wait until after 0.6.0.0 is released. Agree?

BTW: reko already has some mechanisms in place to detect signatures for the purpose of detecting unpackers. I suggest we use those mechanisms to identify and process CRT startup code, to save a lot of time.

[ptomin]

do think this is worth postponing the 0.6.0.0 release to implement this?

All I want is the correct name and signature for PE *.exe entry point. I think it is a little fix and should be done in 0.6.0.0
As for discovering WinMain/main and other improvements it could be done in the next releases.

[uxmal]

I suggest that you make the correction in the master branch and submit a PR, then. I will then merge those changes to the nested-textModel branch as well.

[ptomin]

correction in the master branch

Do you mean replacing INT WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, INT CmdShow) with DWORD Win32CrtStartup(void)?

[uxmal]

Yes, let fix both name and signature.

[ptomin]

OK, I'll do it when I'll have spare time.
And the same change should be done for win32 DLLs, is not it?

[uxmal]

Yes let's do both at the same time.

[ptomin]

... but DllMain is correct. See https://msdn.microsoft.com/en-us/library/windows/desktop/ms682583%28v=vs.85%29.aspx and PySample

[uxmal]

Indeed. We want correct signatures for both and correct names for both.

[ptomin]

Done in 2bccebe

[ptomin]

Now it could be closed, is not it, John?

[uxmal]

Actually, I'll leave this open. We want to track the other items in the checklist above.

[uxmal]

Each implementation of IPlatform that wants to find the "real" main program will need to implement the method IPlatform.FindMainAddress. I'm doing so for MS-DOS on the issue-211 branch.

[uxmal]

Starting on Win32Platform.FindMainAddress. Will probably steal the one from Boomerang unless better ideas appear.

[uxmal]

An implementation of Win32Platform.FindMainAddress was committed in d7da25d. It's now a question of finding more patterns that match different compilers' CRT entry points. I leave the finding of such patterns as an "exercise for the reader".

Each ImageLoader that wants to support detection of "real" entry points must do the following:

• In the implementation of ImageLoader.Relocate, call IPlatform.FindMainProcedure
• Implement IPlatform.FindMainProcedure. You may need to provide multiple implementations for the different architectures the platform supports.

Can we consider this a reasonable resolution for the issue?

[ptomin]

Good! Now Reko discovers main procedure for RussianText.exe

Can we consider this a reasonable resolution for the issue?

Yes, we can