Link.exe chokes on debug info generated by UASM

[vid512]

In some cases, debug info generated by UASM, when linked with standard .lib, causes MS link.exe to throw "LINK : fatal error LNK1102: out of memory". This is in any case a bug in MS linker, but I strongly suspect it gets triggered by invalid debug info generated by UASM.

I managed to strip the problem down to following minimal example. Hopefully it should be simple enough to check the resulting debug info manually. Unfortunately I know nothing about whatever debug format is used.

util.asm

p typedef ptr byte

x STRUCT
  y p ?
x ENDS

ExitProcess PROTO :DWORD
z EQU ExitProcess

end

reproduce with:

uasm64 -c -Zi -win64 util.asm
link /DEBUG util.obj kernel32.lib

[john-terraspace]

Which version of link are you using, 32/64bit and also have you tried using one of the newer CodeView formats like -Zi8 (for CV8) ? From: vid512 ***@***.***> Sent: Sunday, December 3, 2023 3:30 PM To: Terraspace/UASM ***@***.***> Cc: Subscribed ***@***.***> Subject: [Terraspace/UASM] Link.exe chokes on debug info generated by UASM (Issue #194) In some cases, debug info generated by UASM, when linked with standard .lib, causes MS link.exe to throw "LINK : fatal error LNK1102: out of memory". This is in any case a bug in MS linker, but I strongly suspect it gets triggered by invalid debug info generated by UASM. I managed to strip the problem down to following minimal example. Hopefully it should be simple enough to check the resulting debug info manually. Unfortunately I know nothing about whatever debug format is used. util.asm p typedef ptr byte x STRUCT y p ? x ENDS ExitProcess PROTO :DWORD z EQU ExitProcess end reproduce with: uasm64 -c -Zi -win64 util.asm link /DEBUG util.obj kernel32.lib — Reply to this email directly, view it on GitHub <#194> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEAZAVFRYUDMXALCYNQEHWLYHSLGRAVCNFSM6AAAAABAE3RGZ2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZDENJYHAYTENQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[vid512]

I use 64 bit linker:

Microsoft (R) Incremental Linker Version 14.38.33130.0
Copyright (C) Microsoft Corporation.  All rights reserved.

With /Zi8, the problem disappears on the particular minimal example I posted, but persists with my original larger codebase.

I can try to reduce failing code with /Zi8 to minimal example again, if that helps, but that will take some effort. Let me know if there's any purpose in doing so.

[john-terraspace]

No I don’t think so, might as well try and identify it in the original CV format, a lot of the symbol generation logic is the same. From: vid512 ***@***.***> Sent: Sunday, December 3, 2023 3:44 PM To: Terraspace/UASM ***@***.***> Cc: John Hankinson ***@***.***>; Comment ***@***.***> Subject: Re: [Terraspace/UASM] Link.exe chokes on debug info generated by UASM (Issue #194) I use 64 bit linker: Microsoft (R) Incremental Linker Version 14.38.33130.0 Copyright (C) Microsoft Corporation. All rights reserved. With /Zi8, the problem disappears on the particular minimal example I posted, but persists with my original larger codebase. I can try to reduce failing code with /Zi8 to minimal example again, if that helps, but that will take some effort. Let me know if there's any purpose in doing so. — Reply to this email directly, view it on GitHub <#194 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEAZAVBESKRFPY5S4TPJUULYHSM4ZAVCNFSM6AAAAABAE3RGZ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXGUZDANJRHA> . You are receiving this because you commented.Message ID: ***@***.***>

[john-terraspace]

I've managed to fix the minimal example in 2.57 for -Zi, but as you said larger codebase still exhibits issue even with -Zi8 - so we'll need another minimal test case to try and narrow that one down.

[vid512]

I don't have the example with larger codebase anymore, nor do I remember how I tested it. So, I think you can close this issue for the moment.