Add Kotlin multiplatform support (google#7969)

* [Kotlin] Introduction to Kotlin Multiplaform

The first implementation of the Kotlin code generation was made years
ago at the time Kotlin Multiplaform was not stable and Kotlin is mostly
used on JVM-based targets. For this reason the generated code uses java
based runtime.

That design decision comes with many drawbacks, leaving the code
generated more java-like and making it impossible to use more advanced
features of the Kotlin language.

In this change we are adding two parts: A pure, multi-plaform, Kotlin
runtime and a new code generator to accompany it.

* [Kotlin] Remove scalar sign cast from code generation

Now that we have a new runtime the accepts unsigned types, we don't
need to code generate casting back and from signed scalars. This
MR removes this from both code generations and adds the necessary
API to the runtime.

* [Kotlin] Use offset on public API to represent buffer position

Currently, kotlin was following Java's approach of representing objects,
vectors, tables as "Int" (the position of it in the buffer). This change
replaces naked Int with Offset<T>, offering a type-safe API. So,
instead of

fun Table.createTable(b: FlatBufferBuilder, subTable: Int)

We will have

fun Table.createTable(b: FlatBufferBuilder, subTable: Offset<SubTable>)

Making impossible to accidentally switch parameters.

The performance should be similar to use Int as we are using value
class for Offset and ArrayOffset, which most of the time translate to
Int in the bytecode.

* [Kotlin] Add builder for tables

Add builder constructor to make create of table more ergonomic.
For example the movie sample for the test set could be written as:

Movie.createMovie(fbb,
    mainCharacterType = Character_.MuLan,
    mainCharacter = att) {
    charactersType = charsType
    this.characters = characters
}

instead of:

Movie.startMovie(fbb)
Movie.addMainCharacterType(fbb, Character_.MuLan)
Movie.addMainCharacter(fbb, att as Offset<Any>)
Movie.addCharactersType(fbb, charsType)
Movie.addCharacters(fbb, charsVec)
Movie.endMovie(fbb)

* [Kotlin] Move enum types to value class

Moving to flatbuffer enums to value class adds type safety for parameters
with minimum to no performance impact.

* [Kotlin] Simplify Union parameters to avoid naked casting

Just a small change on the APIs that receive union as parameters,
creating a typealias UnionOffset to avoid using Offset<Any>. To "convert"
an table offset to an union, one just call Offset.toUnion().

* [Kotlin] Apply clang-format on kotlin code generators

* [Kotlin] Update kotlin generator to follow official naming conventions

Updating directory, package and enum naming to follow Kotlin official
convention.

https://kotlinlang.org/docs/coding-conventions.html#naming-rules

* [Kotlin] Add fixes to improve performance

1 - Add benchmark comparing serialization between Java & Kotlin
2 - ReadWriteBuffer does not auto-grow (thus avoid check size in every op)
3 - Add specialized add functions on FlatBufferBuilder to avoid boxing
offsets.
4 - Remove a few Kotlin syntax sugar that generated performance penalties.

* [Kotlin] Remove builder from Kotlin KMP and add some optimizations
to avoid boxing of Offset classes

---------

Co-authored-by: Derek Bailey <[email protected]>

.github/labeler.yml

@@ -48,6 +48,7 @@ java:
 kotlin:
   - '**/*.kt'
   - src/idl_gen_kotlin.cpp
+  - src/idl_gen_kotlin_kmp.cpp
 
 lua:
   - '**/*.lua'

.github/workflows/build.yml

@@ -422,9 +422,14 @@ jobs:
       with:
         distribution: 'temurin'
         java-version: '11'
+    - name: Build flatc
+      run: |
+       cmake -DFLATBUFFERS_BUILD_TESTS=OFF -DFLATBUFFERS_BUILD_FLATLIB=OFF -DFLATBUFFERS_BUILD_FLATHASH=OFF .
+       make -j
+       echo "${PWD}" >> $GITHUB_PATH
     - name: Build
       working-directory: kotlin
-      run: ./gradlew clean iosX64Test macosX64Test
+      run: ./gradlew clean iosSimulatorArm64Test macosX64Test macosArm64Test
 
   build-kotlin-linux:
     name: Build Kotlin Linux
@@ -437,6 +442,11 @@ jobs:
         distribution: 'temurin'
         java-version: '11'
     - uses: gradle/[email protected]
+    - name: Build flatc
+      run: |
+       cmake -DFLATBUFFERS_BUILD_TESTS=OFF -DFLATBUFFERS_BUILD_FLATLIB=OFF -DFLATBUFFERS_BUILD_FLATHASH=OFF .
+       make -j
+       echo "${PWD}" >> $GITHUB_PATH
     - name: Build
       working-directory: kotlin
       # we are using docker's version of gradle

.gitignore

@@ -151,3 +151,5 @@ flatbuffers.pc
 # https://cmake.org/cmake/help/latest/module/FetchContent.html#variable:FETCHCONTENT_BASE_DIR
 cmake-build-debug/
 _deps/
+**/.gradle/**
+kotlin/**/generated

CMakeLists.txt

@@ -160,6 +160,7 @@ set(FlatBuffers_Compiler_SRCS
   src/idl_gen_csharp.cpp
   src/idl_gen_dart.cpp
   src/idl_gen_kotlin.cpp
+  src/idl_gen_kotlin_kmp.cpp
   src/idl_gen_go.cpp
   src/idl_gen_java.cpp
   src/idl_gen_ts.cpp

android/.project

@@ -10,7 +10,7 @@
 	</natures>
 	<filteredResources>
 		<filter>
-			<id>1677235311958</id>
+			<id>1672434305228</id>
 			<name></name>
 			<type>30</type>
 			<matcher>

include/flatbuffers/idl.h

@@ -342,7 +342,10 @@ struct FieldDef : public Definition {
   bool Deserialize(Parser &parser, const reflection::Field *field);
 
   bool IsScalarOptional() const {
-    return IsScalar(value.type.base_type) && IsOptional();
+    return IsScalar() && IsOptional();
+  }
+  bool IsScalar() const {
+      return ::flatbuffers::IsScalar(value.type.base_type);
   }
   bool IsOptional() const { return presence == kOptional; }
   bool IsRequired() const { return presence == kRequired; }
@@ -725,6 +728,7 @@ struct IDLOptions {
     kSwift = 1 << 16,
     kNim = 1 << 17,
     kProto = 1 << 18,
+    kKotlinKmp = 1 << 19,
     kMAX
   };
 

kotlin/benchmark/build.gradle.kts

@@ -1,5 +1,3 @@
-import org.jetbrains.kotlin.ir.backend.js.compile
-
 plugins {
   kotlin("multiplatform")
   id("org.jetbrains.kotlinx.benchmark")
@@ -27,7 +25,7 @@ benchmark {
       iterationTime = 300
       iterationTimeUnit = "ms"
       // uncomment for benchmarking JSON op only
-      include(".*JsonBenchmark.*")
+       include(".*FlatbufferBenchmark.*")
     }
   }
   targets {
@@ -36,24 +34,34 @@ benchmark {
 }
 
 kotlin {
-  jvm()
-
-  sourceSets {
-
-    all {
-      languageSettings.enableLanguageFeature("InlineClasses")
+  jvm {
+    compilations {
+      val main by getting { }
+      // custom benchmark compilation
+      val benchmarks by compilations.creating {
+        defaultSourceSet {
+          dependencies {
+            // Compile against the main compilation's compile classpath and outputs:
+            implementation(main.compileDependencyFiles + main.output.classesDirs)
+          }
+        }
+      }
     }
+  }
 
+  sourceSets {
     val jvmMain by getting {
       dependencies {
         implementation(kotlin("stdlib-common"))
         implementation(project(":flatbuffers-kotlin"))
         implementation(libs.kotlinx.benchmark.runtime)
-        implementation("com.google.flatbuffers:flatbuffers-java:2.0.3")
+        implementation("com.google.flatbuffers:flatbuffers-java:23.5.9")
         // json serializers
         implementation(libs.moshi.kotlin)
         implementation(libs.gson)
       }
+      kotlin.srcDir("src/jvmMain/generated/kotlin/")
+      kotlin.srcDir("src/jvmMain/generated/java/")
     }
   }
 }
@@ -67,3 +75,64 @@ tasks.register<de.undercouch.gradle.tasks.download.Download>("downloadMultipleFi
   dest(File("${project.projectDir.absolutePath}/src/jvmMain/resources"))
   overwrite(false)
 }
+
+abstract class GenerateFBTestClasses : DefaultTask() {
+  @get:InputFiles
+  abstract val inputFiles: ConfigurableFileCollection
+
+  @get:Input
+  abstract val includeFolder: Property<String>
+
+  @get:Input
+  abstract val outputFolder: Property<String>
+
+  @get:Input
+  abstract val variants: ListProperty<String>
+
+  @Inject
+  protected open fun getExecActionFactory(): org.gradle.process.internal.ExecActionFactory? {
+    throw UnsupportedOperationException()
+  }
+
+  init {
+    includeFolder.set("")
+  }
+
+  @TaskAction
+  fun compile() {
+    val execAction = getExecActionFactory()!!.newExecAction()
+    val sources = inputFiles.asPath.split(":")
+    val langs = variants.get().map { "--$it" }
+    val args = mutableListOf("flatc","-o", outputFolder.get(), *langs.toTypedArray())
+    if (includeFolder.get().isNotEmpty()) {
+      args.add("-I")
+      args.add(includeFolder.get())
+    }
+    args.addAll(sources)
+    println(args)
+    execAction.commandLine = args
+    print(execAction.execute())
+  }
+}
+
+// Use the default greeting
+tasks.register<GenerateFBTestClasses>("generateFBTestClassesKt") {
+  inputFiles.setFrom("$projectDir/monster_test_kotlin.fbs")
+  includeFolder.set("$rootDir/../tests/include_test")
+  outputFolder.set("${projectDir}/src/jvmMain/generated/kotlin/")
+  variants.addAll("kotlin-kmp")
+}
+
+tasks.register<GenerateFBTestClasses>("generateFBTestClassesJava") {
+  inputFiles.setFrom("$projectDir/monster_test_java.fbs")
+  includeFolder.set("$rootDir/../tests/include_test")
+  outputFolder.set("${projectDir}/src/jvmMain/generated/java/")
+  variants.addAll("kotlin")
+}
+
+project.tasks.forEach {
+  if (it.name.contains("compileKotlin")) {
+    it.dependsOn("generateFBTestClassesKt")
+    it.dependsOn("generateFBTestClassesJava")
+  }
+}

kotlin/benchmark/monster_test_java.fbs

@@ -0,0 +1,37 @@
+// Example IDL file for our monster's schema.
+
+namespace jmonster;
+
+enum JColor:byte { Red = 0, Green, Blue = 2 }
+
+union JEquipment { JWeapon } // Optionally add more tables.
+
+struct JVec3 {
+  x:float;
+  y:float;
+  z:float;
+}
+
+table JMonster {
+  pos:JVec3;
+  mana:short = 150;
+  hp:short = 100;
+  name:string;
+  friendly:bool = false (deprecated);
+  inventory:[ubyte];
+  color:JColor = Blue;
+  weapons:[JWeapon];
+  equipped:JEquipment;
+  path:[JVec3];
+}
+
+table JWeapon {
+  name:string;
+  damage:short;
+}
+
+table JAllMonsters {
+   monsters: [JMonster];
+}
+
+root_type JAllMonsters;

kotlin/benchmark/monster_test_kotlin.fbs

@@ -0,0 +1,37 @@
+// Example IDL file for our monster's schema.
+
+namespace monster;
+
+enum Color:byte { Red = 0, Green, Blue = 2 }
+
+union Equipment { Weapon } // Optionally add more tables.
+
+struct Vec3 {
+  x:float;
+  y:float;
+  z:float;
+}
+
+table Monster {
+  pos:Vec3;
+  mana:short = 150;
+  hp:short = 100;
+  name:string;
+  friendly:bool = false (deprecated);
+  inventory:[ubyte];
+  color:Color = Blue;
+  weapons:[Weapon];
+  equipped:Equipment;
+  path:[Vec3];
+}
+
+table Weapon {
+  name:string;
+  damage:short;
+}
+
+table AllMonsters {
+   monsters: [Monster];
+}
+
+root_type AllMonsters;

kotlin/benchmark/src/jvmMain/kotlin/com/google/flatbuffers/kotlin/benchmark/FlatbufferBenchmark.kt

@@ -0,0 +1,68 @@
+@file:OptIn(ExperimentalUnsignedTypes::class)
+
+package com.google.flatbuffers.kotlin.benchmark
+
+
+import com.google.flatbuffers.kotlin.FlatBufferBuilder
+import jmonster.JAllMonsters
+import jmonster.JMonster
+import jmonster.JVec3
+import monster.AllMonsters.Companion.createAllMonsters
+import monster.AllMonsters.Companion.createMonstersVector
+import monster.Monster
+import monster.Monster.Companion.createInventoryVector
+import monster.MonsterOffsetArray
+import monster.Vec3
+import org.openjdk.jmh.annotations.*
+import java.util.concurrent.TimeUnit
+
+@State(Scope.Benchmark)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.NANOSECONDS)
+@Measurement(iterations = 20, time = 1, timeUnit = TimeUnit.NANOSECONDS)
+open class FlatbufferBenchmark {
+
+  val repetition = 1000000
+  val fbKotlin = FlatBufferBuilder(1024 * repetition)
+  val fbJava = com.google.flatbuffers.FlatBufferBuilder(1024 * repetition)
+
+  @OptIn(ExperimentalUnsignedTypes::class)
+  @Benchmark
+  fun monstersKotlin() {
+    fbKotlin.clear()
+    val monsterName = fbKotlin.createString("MonsterName");
+    val items = ubyteArrayOf(0u, 1u, 2u, 3u, 4u)
+    val inv = createInventoryVector(fbKotlin, items)
+    val monsterOffsets: MonsterOffsetArray = MonsterOffsetArray(repetition) {
+      Monster.startMonster(fbKotlin)
+      Monster.addName(fbKotlin, monsterName)
+      Monster.addPos(fbKotlin, Vec3.createVec3(fbKotlin, 1.0f, 2.0f, 3.0f))
+      Monster.addHp(fbKotlin, 80)
+      Monster.addMana(fbKotlin, 150)
+      Monster.addInventory(fbKotlin, inv)
+      Monster.endMonster(fbKotlin)
+    }
+    val monsters = createMonstersVector(fbKotlin, monsterOffsets)
+    val allMonsters = createAllMonsters(fbKotlin, monsters)
+    fbKotlin.finish(allMonsters)
+  }
+
+  @Benchmark
+  fun monstersjava() {
+    fbJava.clear()
+    val monsterName = fbJava.createString("MonsterName");
+    val inv = JMonster.createInventoryVector(fbJava, byteArrayOf(0, 1, 2, 3, 4).asUByteArray())
+    val monsters = JAllMonsters.createMonstersVector(fbJava, IntArray(repetition) {
+      JMonster.startJMonster(fbJava)
+      JMonster.addName(fbJava, monsterName)
+      JMonster.addPos(fbJava, JVec3.createJVec3(fbJava, 1.0f, 2.0f, 3.0f))
+      JMonster.addHp(fbJava, 80)
+      JMonster.addMana(fbJava, 150)
+      JMonster.addInventory(fbJava, inv)
+      JMonster.endJMonster(fbJava)
+    })
+    val allMonsters = JAllMonsters.createJAllMonsters(fbJava, monsters)
+    fbJava.finish(allMonsters)
+  }
+
+}

kotlin/benchmark/src/jvmMain/kotlin/com/google/flatbuffers/kotlin/benchmark/FlexBuffersBenchmark.kt

@@ -13,6 +13,8 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+@file:OptIn(ExperimentalUnsignedTypes::class)
+
 package com.google.flatbuffers.kotlin.benchmark
 import com.google.flatbuffers.ArrayReadWriteBuf
 import com.google.flatbuffers.FlexBuffers

kotlin/benchmark/src/jvmMain/kotlin/com/google/flatbuffers/kotlin/benchmark/JsonBenchmark.kt

@@ -53,9 +53,10 @@ open class JsonBenchmark {
 
   val fbParser = JSONParser()
 
-  final val twitterData = this.javaClass.classLoader.getResourceAsStream("twitter.json")!!.readBytes()
-  final val canadaData = this.javaClass.classLoader.getResourceAsStream("canada.json")!!.readBytes()
-  final val citmData = this.javaClass.classLoader.getResourceAsStream("citm_catalog.json")!!.readBytes()
+  final val classLoader = this.javaClass.classLoader
+  final val twitterData = classLoader.getResourceAsStream("twitter.json")!!.readBytes()
+  final val canadaData = classLoader.getResourceAsStream("canada.json")!!.readBytes()
+  final val citmData = classLoader.getResourceAsStream("citm_catalog.json")!!.readBytes()
 
   val fbCitmRef = JSONParser().parse(ArrayReadBuffer(citmData))
   val moshiCitmRef = moshi.adapter(Map::class.java).fromJson(citmData.decodeToString())

kotlin/benchmark/src/jvmMain/kotlin/com/google/flatbuffers/kotlin/benchmark/UTF8Benchmark.kt

@@ -35,14 +35,14 @@ import kotlin.random.Random
 @BenchmarkMode(Mode.AverageTime)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 @Measurement(iterations = 100, time = 1, timeUnit = TimeUnit.MICROSECONDS)
-class UTF8Benchmark {
-
-  private final val sampleSize = 5000
-  private final val stringSize = 25
-  final var sampleSmallUtf8 = (0..sampleSize).map { populateUTF8(stringSize) }.toList()
-  final var sampleSmallUtf8Decoded = sampleSmallUtf8.map { it.encodeToByteArray() }.toList()
-  final var sampleSmallAscii = (0..sampleSize).map { populateAscii(stringSize) }.toList()
-  final var sampleSmallAsciiDecoded = sampleSmallAscii.map { it.encodeToByteArray() }.toList()
+open class UTF8Benchmark {
+
+  private val sampleSize = 5000
+  private val stringSize = 25
+  private var sampleSmallUtf8 = (0..sampleSize).map { populateUTF8(stringSize) }.toList()
+  private var sampleSmallUtf8Decoded = sampleSmallUtf8.map { it.encodeToByteArray() }.toList()
+  private var sampleSmallAscii = (0..sampleSize).map { populateAscii(stringSize) }.toList()
+  private var sampleSmallAsciiDecoded = sampleSmallAscii.map { it.encodeToByteArray() }.toList()
 
   @Setup
   fun setUp() {