Improve Objective-C support

[LunaTheFoxgirl]

This pull-request aims to improve the Objective-C support of LDC. This PR additionally changes the Objective-C generation system to only support the Objective-C 2.0 ABI, as very little documentation of older ABIs have survived.

A good chunk of this code is more or less taken from the OpenD implementation, but I'm already working on cleaning parts up as I go. I am looking into how feasible it'll be to make extern(Objective-C) classes export neccesary ABI for Objective-C code to be able to call it as well, which mainly involves metadata generation.

TODO List:

• Classes
  • ClassRO Tables
  • Class List Tables
  • Swift Stub Class Tables (@swift keyword)
  • Dynamic Class Casting
• Methods
  • Method List Tables
  • Selector Reference Tables
• Protocols
  • Protocol List Table
  • Dynamic Protocol Casting
• Instance Variables
  • Generate instance variables and offsets

[LunaTheFoxgirl]

Question: in this implementation I wish to add another UDA to ldc.attributes called @swift.

Idea there is to implement support for binding Swift interop class stubs as well, which have unique behaviour that would require this new attribute.

Essentially Swift may declare Objective-C classes in the __objc_stubs section with an isa field pointing to address 0x1. Followed by a function pointer that would need to be invoked to instantiate the class definition, via the objc_loadClassRef helper function.

This would allow interop with Swift code using bridging headers and wouldn't be too much extra work.

Is it okay if I add that to this PR?

[thewilsonator]

Is it okay if I add that to this PR?

Yes.

[adamdruppe]

I am looking into how feasible it'll be to make extern(Objective-C) classes export neccesary ABI for Objective-C code to be able to call it as well, which mainly involves metadata generation.

this already works in opend so if you copy the impl it should work here too; that's how the final iphone demo from my blog worked, as well as simpledisplay's cocoa implementation, and to triple check i just did one more test:

@interface Thing : NSObject
   - (void) test
@end 

int main() {
  [[[Thing alloc] init] test];
}

and then the D

extern(Objective-C)
class Thing : NSObject {
  void test() @selector("test") {
        printf("here\n");
   }
}

and all worked.

of course, possible i missed some details that won't emerge until you get into more features, but still a lot already works in the opend world.

[adamdruppe]

BTW something i think would be kinda cool is a kind of import objective-c

dstep can do a good amount of that already, and the objective c parts look easier to parse and interface with than regular import c

so im p sure it doable with a week or two of work

[LunaTheFoxgirl]

BTW something i think would be kinda cool is a kind of import objective-c

dstep can do a good amount of that already, and the objective c parts look easier to parse and interface with than regular import c

so im p sure it doable with a week or two of work

Yeah, I'm thinking of making an Objective-C code generator once I get the compiler support up and running.
Got protocols working now, might implement categories as well. Next up is codegen for swift stub classes.

[thewilsonator]

(once this is good to go) Could you please open a PR to DMD with all of the frontend/druntime changes? (Just stub out anything needs anything extra to work)

[LunaTheFoxgirl]

(once this is good to go) Could you please open a PR to DMD with all of the frontend/druntime changes? (Just stub out anything needs anything extra to work)

Sure, will do. However I think I'll reduce the scope of this PR slightly to just be the core objective-c and Swift stub class support. Given that some existing Apple platforms don't align neatly with the code I was referencing I'm doing a pretty big refactor of the objective-c data struct generation to be compatible with all of the Apple sub architectures supported.

[LunaTheFoxgirl]

cc @adamdruppe once I finish this PR you may want to back integrate it into OpenD's LDC fork. Your current implementation is broken on quite a few platforms and in some cases may crash the Xcode linker

[adamdruppe]

plz write up some notes as you go, as it might also apply back to dmd's separate implementation (and the abi spec on the website) as well. depends on if it was just me bugging it up or if the original was wrong too.

much of it came from reverse engineering the output on a x86 Mac so no surprise they'd be some differences (one we all found early is the arm objc_msgSend is slightly different on arm), then i just sucked at ldc code so prolly a lot of impl bugs too lol.

[LunaTheFoxgirl]

plz write up some notes as you go, as it might also apply back to dmd's separate implementation (and the abi spec on the website) as well. depends on if it was just me bugging it up or if the original was wrong too.

much of it came from reverse engineering the output on a x86 Mac so no surprise they'd be some differences (one we all found early is the arm objc_msgSend is slightly different on arm), then i just sucked at ldc code so prolly a lot of impl bugs too lol.

Yeah my recommendation there is to look at the actual objective-c runtime source. The data structures relevant are listed there https://github.com/apple-oss-distributions/objc4 as well as the Swift interop documentation.

I'm adding the finishing touches to this PR tonight, and will mark it as ready to merge then.

[adamdruppe]

I'll reread the whole thing when I get another chunk of time free, idk if it'll be today or tomorrow, but I'll let you know when I do.

My biggest concern is just if it works and that last test I did with it passing was a good sign!

[LunaTheFoxgirl]

@kinke anything left to solve? All the tests succeed (minus the ones that are either broken for unrelated reasons and the ones where you've run out of compute credits)

[LunaTheFoxgirl]

Also it seems my refactor somehow fixed all the random asserts that was happening.

[LunaTheFoxgirl]

I think that's everything?
Sorry if I'm a bit pushy, I've been working unholy hours on this the past few days. I'm very exhausted ^^;

[kinke]

Yeah that's it from my side at least, thanks. :)

[p0nce]

This is super valuable work! Thanks

[LunaTheFoxgirl]

Is it possible that this could be merged for the next release or is there further things I need to sort out? I'm slightly burnt out after staring at aarch64 disassembly for 50+ hours the past few days @thewilsonator @kinke @jacob-carlborg

[kinke]

I think we can have this in v1.40 final, which I wanna release in December, regardless whether DMD v2.110 is finalized or not by then.

In that regard, feel free to add a changelog entry in CHANGELOG.md yourself, to let users know about this.

[LunaTheFoxgirl]

I think we can have this in v1.40 final, which I wanna release in December, regardless whether DMD v2.110 is finalized or not by then.

In that regard, feel free to add a changelog entry in CHANGELOG.md yourself, to let users know about this.

Okay, done!

[jacob-carlborg]

Isn't this exactly the same as declaredAsOptionalCount, it just takes a different kind of declaration? Can't that be reused?

[LunaTheFoxgirl]

Possibly but I don't know my way around this codebase very well. Takes a long time to figure out how anything works. Also the Swift stub UDA only applies to classes, as it affects how the isa pointer/classref is dereferenced.

[jacob-carlborg]

Well, isn't declaredAsOptionalCount what you copy-pasted? Seems very likely, especially since the documentation is now wrong, it says: as optional.

[LunaTheFoxgirl]

Yeah I copied it and modified it as needed since I didn't see an obvious way of checking for UDAs otherwise.

I'll look into making it a reusable function after I get some rest.

[jacob-carlborg]

Looks like you can replace ClassDeclaration with Dsymbol for the cd parameter, given it a more generic name, like "countUdas" and pass in the UDA identifier as an additional argument.

[LunaTheFoxgirl]

Yeah, something like that

[jacob-carlborg]

This is different behavior compared to DMD.

[jacob-carlborg]

I don't see a test for this. This is not implemented in upstream.

[LunaTheFoxgirl]

I'm avoiding writing my own tests, would prefer is someone else picks that up. I simply don't know this testing system very well.

[jacob-carlborg]

It's pretty straightforward. Just add a new file here https://github.com/ldc-developers/ldc/tree/master/tests/dmd/runnable and it will be picked up automatically. You can take a look at one of the existing Objective-C tests, like this one: https://github.com/ldc-developers/ldc/blob/master/tests/dmd/runnable/objc_call.d.

[jacob-carlborg]

I'll let someone else decide if the tests are necessary or not.

[kinke]

Landing this eagerly for upcoming beta5. Thx all!

[s-ludwig]

Very nice, thanks @LunaTheFoxgirl! That will lower the bar for macOS/iOS development considerably. It's a shame that the money from the bounty has been lost by now.

[LunaTheFoxgirl]

Very nice, thanks @LunaTheFoxgirl! That will lower the bar for macOS/iOS development considerably. It's a shame that the money from the bounty has been lost by now.

It is what it is. I've already begun working on dub packages to cover API bindings to Apple APIs, which should lower the barrier further.

That being said, I'm doing it by hand to make sure all of the API has appropriate ddoc documentation as well.