Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile-time RTTI bindings generation #76

Draft
wants to merge 101 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
20f4c97
Compile-time RTTI
ShadelessFox Sep 12, 2024
29ccb1f
Enum generation correctness
ShadelessFox Sep 13, 2024
31cd86f
Category annotations
ShadelessFox Sep 13, 2024
b2e7ade
Type generation improvements
ShadelessFox Sep 17, 2024
0e64600
Generate runtime POJO types for simple compounds without categories
ShadelessFox Sep 17, 2024
d9a1da0
Condition fix
ShadelessFox Sep 17, 2024
edaee1e
Remove caching
ShadelessFox Sep 17, 2024
978074f
Generate categories
ShadelessFox Sep 17, 2024
4066b6a
Enum constant name generation fix
ShadelessFox Sep 17, 2024
80b7f3c
Get/set attribute value via `AttributeInfo#get/set`
ShadelessFox Sep 17, 2024
23af7bb
Reorganization; represent type name as a sum type
ShadelessFox Sep 17, 2024
597bd1c
Move generation code to an annotation processor
ShadelessFox Sep 20, 2024
ba90e8a
Compile-time `MsgReadBinary` callbacks
ShadelessFox Sep 21, 2024
956eab6
Default handler for `MsgReadBinary`; attributes sorting fix
ShadelessFox Sep 22, 2024
3ea0112
Fix attribute sorting
ShadelessFox Sep 23, 2024
02c9d8e
Fix attribute ordering for good
ShadelessFox Sep 24, 2024
72d28ca
Add support for enum flags
ShadelessFox Sep 24, 2024
375284a
Disable enum name normalization for now
ShadelessFox Sep 24, 2024
4bb2782
Generate `valueOf` for enums
ShadelessFox Sep 24, 2024
46845a8
Fix `RTTI::getCategories` returning overridden categories
ShadelessFox Sep 24, 2024
d671df3
Remove unused repository
ShadelessFox Sep 24, 2024
a30ab3c
Use `TypeName` in the generator, too
ShadelessFox Sep 24, 2024
fc0c17f
Experimental `.core` file reader for Until Dawn (PS4)
ShadelessFox Oct 4, 2024
a1e64db
Fix RTTI attribute sorting
ShadelessFox Oct 7, 2024
b2c12f0
Optimize and cleanup reading code
ShadelessFox Oct 7, 2024
d0c6a33
Refactor enum representation
ShadelessFox Oct 8, 2024
8df6d9f
Update runtime type representation generator
ShadelessFox Oct 9, 2024
db4d56c
Add tests and proper implementation of `equals` and `hashCode`
ShadelessFox Oct 9, 2024
77ea2c7
Add a test for ensuring correctness of attribute sorting
ShadelessFox Oct 9, 2024
daf1cdb
Include only RTTI classes in the by-name lookup
ShadelessFox Oct 9, 2024
97db540
Optimize code generation for `equals` and `hashCode`
ShadelessFox Oct 9, 2024
9c201b6
New representation for enums allowing lossless reading of unknown values
ShadelessFox Oct 10, 2024
198451e
Move bindings generator to a separate module; cleanup dependencies
ShadelessFox Oct 12, 2024
c859fc4
Move builtin type registration to the `@GenerateBindings`
ShadelessFox Oct 13, 2024
97b6723
Add bindings generation for Horizon Zero Dawn and Death Stranding
ShadelessFox Oct 25, 2024
70d43bc
Mark JSON files as binary
ShadelessFox Oct 26, 2024
fd27857
Mark JSON files as generated so they won't be accounted in stats
ShadelessFox Oct 26, 2024
021bc71
Category fields are synthetic, no need to exclude them in tests anymore
ShadelessFox Oct 26, 2024
7a29755
Replace MathUtils with modern APIs
ShadelessFox Nov 1, 2024
c288109
Add bindings generation for Horizon Zero Dawn Remastered
ShadelessFox Nov 2, 2024
8e16d90
Introduce BinaryReader API for reading binary data
ShadelessFox Nov 9, 2024
15e32ba
Add common implementation for chunked binary readers
ShadelessFox Nov 9, 2024
bc0c1fe
Initial work on HRZ Remastered
ShadelessFox Nov 9, 2024
7cdc710
Replace static callback field with an interface
ShadelessFox Nov 9, 2024
c6bb2c1
Extract runtime POD generation to a separate class
ShadelessFox Nov 11, 2024
0b48e99
Type representation at runtime
ShadelessFox Nov 13, 2024
d20e65f
Remove unused code
ShadelessFox Nov 13, 2024
e747368
Remove experimental and unused code
ShadelessFox Nov 13, 2024
2b3ba80
Use an isolated class loader per factory
ShadelessFox Nov 13, 2024
7ffb63a
Remove the `RTTI` class and minor cleanup
ShadelessFox Nov 13, 2024
f07f27a
Implement type factory for HRZR
ShadelessFox Nov 13, 2024
5b5cbea
Use a proper platform constants instead of a hardcoded string
ShadelessFox Nov 17, 2024
2ae7bde
Fix enum set type generation
ShadelessFox Nov 23, 2024
9fb5111
Small reader improvements
ShadelessFox Nov 23, 2024
685d0de
Some progress on implementing HRZR MsgReadBinary callbacks
ShadelessFox Nov 23, 2024
41391ff
Update HRZR RTTI to `1.3.51.0`
ShadelessFox Nov 23, 2024
dd6d984
Implement missing MsgReadBinary callbacks for full coverage of HRZR
ShadelessFox Nov 24, 2024
422de62
Misc cleanups
ShadelessFox Nov 24, 2024
f4b9187
Add modules and refactoring
ShadelessFox Nov 24, 2024
fb5a707
Improve enum name generation
ShadelessFox Nov 24, 2024
205d96d
Move array-related helper methods into a separate class
ShadelessFox Nov 25, 2024
94ec8c1
Work in progress migrating to modules; migrate Until Dawn to modules
ShadelessFox Nov 25, 2024
221425d
Migrate HRZR to modules
ShadelessFox Nov 26, 2024
b5fa450
Introduce new API for hashing inspired by Guava's Hashing class
ShadelessFox Nov 26, 2024
5c6250c
Upgrade Gradle to the latest version and disable incremental build
ShadelessFox Nov 29, 2024
9cd7c3f
Enable auto provisioning of JDK and remove redundant workflow step
ShadelessFox Nov 29, 2024
6a5463d
Small refactoring; extract asset stuff to a separate module
ShadelessFox Dec 7, 2024
36ad93f
Code cleanup; migrate to a maintained fork of JavaPoet
ShadelessFox Dec 7, 2024
c1c4342
Initial work on Forbidden West; read streaming graph
ShadelessFox Dec 7, 2024
4bd2673
Move DirectStorage reader into common module
ShadelessFox Dec 11, 2024
14d29f1
More work on Forbidden West
ShadelessFox Dec 11, 2024
00091d0
Extract shared logic of type readers
ShadelessFox Dec 11, 2024
334c8f0
Remove unchecked cast
ShadelessFox Dec 11, 2024
396e6ec
More little cleanups
ShadelessFox Dec 13, 2024
f37bce9
Update HRZR RTTI to `1.4.63.0`
ShadelessFox Dec 13, 2024
db337ec
Implement handlers for TextureXXX types; fix group span read
ShadelessFox Dec 14, 2024
777417c
Implement some more handlers for HFW types
ShadelessFox Dec 16, 2024
ad1ca30
Ability to extend generated RTTI types
ShadelessFox Dec 17, 2024
3b2385e
Use interface type for `TypeInfo#type`, instance type for instantiating
ShadelessFox Dec 17, 2024
8c75e74
Implement even more handlers for HFW types and misc cleanups
ShadelessFox Dec 17, 2024
b66d2e0
Add ability to change byte order of a reader using `BinaryReader#order`
ShadelessFox Dec 17, 2024
90ad21a
Implement missing handlers for a complete coverage of all groups
ShadelessFox Dec 17, 2024
f46f034
Small code style changes
ShadelessFox Dec 18, 2024
ef1dd86
Fix `BinaryReader#position` not accepting position at the very end
ShadelessFox Dec 20, 2024
e385556
Decode HFW's texture components
ShadelessFox Dec 21, 2024
7ff3bc0
Update HFW RTTI to `1.5.80.0`
ShadelessFox Dec 21, 2024
2a48dfe
Enhance `Value`'s representation of a "set of values"
ShadelessFox Dec 25, 2024
74a945b
Add `getType` method for RTTI classes that returns its representation
ShadelessFox Jan 2, 2025
116a089
Consider attributes defined in MsgReadBinary's handler as "declared"
ShadelessFox Jan 2, 2025
8e12e83
Slightly optimize type generator; add explanation comments
ShadelessFox Jan 2, 2025
c8466cd
Clean up redundant dependencies
ShadelessFox Jan 12, 2025
a372abf
Small RTTI touches
ShadelessFox Jan 12, 2025
1766025
Log the channel of a package file
ShadelessFox Jan 12, 2025
62fdf75
Fix default implementation of `BinaryReader#readXXX` methods
ShadelessFox Jan 29, 2025
c92d3e8
A bit of experimentation with APIs
ShadelessFox Feb 8, 2025
1b4ebf0
HFW streaming graph visualizer
ShadelessFox Feb 17, 2025
9c6874e
Remove unused code
ShadelessFox Feb 18, 2025
509a5e9
Fix paginated tree model showing incorrect number of items left
ShadelessFox Feb 20, 2025
5c4669a
Fix sorting of numbers with leading zeroes in alphanumeric comparator
ShadelessFox Feb 20, 2025
59a52da
Remove unused code
ShadelessFox Feb 20, 2025
e6cfbec
Figure out more details about string encoding in KZ4/UD
ShadelessFox Feb 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[*]
charset = utf-8
end_of_line = crlf
indent_size = 4
indent_style = space
insert_final_newline = true
tab_width = 4
ij_continuation_indent_size = 4

[*.java]
ij_java_do_while_brace_force = always
ij_java_for_brace_force = always
ij_java_if_brace_force = always
ij_java_while_brace_force = always
ij_java_call_parameters_new_line_after_left_paren = true
ij_java_call_parameters_right_paren_on_new_line = true
ij_java_call_parameters_wrap = off
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.json linguist-vendored linguist-generated
*.json binary
6 changes: 0 additions & 6 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@ jobs:
steps:
- uses: actions/checkout@v4

- name: Set up JDK 17
uses: actions/setup-java@v4
with:
java-version: '17'
distribution: 'temurin'

- name: Build with Gradle Wrapper
run: ./gradlew build

Expand Down
13 changes: 6 additions & 7 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,13 @@ application {
}

dependencies {
implementation project(':bundle-lwjgl')
implementation project(':platform-model')
implementation project(':platform-ui')
implementation project(':decima-model')
implementation project(':decima-ui')
implementation project(':decima-ext-model-exporter')
implementation project(':decima-ext-model-viewer')
implementation project(':decima-ext-shader-viewer')
implementation project(':decima-ext-texture-viewer')

implementation project(':decima-app')
}

allprojects {
Expand All @@ -37,7 +35,8 @@ allprojects {

java {
toolchain {
languageVersion = JavaLanguageVersion.of(17)
languageVersion = JavaLanguageVersion.of(21)
vendor = JvmVendorSpec.ADOPTIUM
}
}

Expand All @@ -47,11 +46,11 @@ allprojects {

dependencies {
// https://mvnrepository.com/artifact/ch.qos.logback/logback-classic
implementation group: 'ch.qos.logback', name: 'logback-classic', version: '1.4.4'
implementation group: 'ch.qos.logback', name: 'logback-classic', version: '1.4.12'

// https://mvnrepository.com/artifact/org.junit.jupiter
testImplementation group: 'org.junit.jupiter', name: 'junit-jupiter-api', version: '5.9.1'
testImplementation group: 'org.junit.jupiter', name: 'junit-jupiter-params', version: '5.9.1'
testRuntimeOnly group: 'org.junit.jupiter', name: 'junit-jupiter-engine', version: '5.9.1'
}
}
}
Binary file modified gradle/wrapper/gradle-wrapper.jar
Binary file not shown.
2 changes: 1 addition & 1 deletion gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-7.5-bin.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-8.11.1-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
38 changes: 25 additions & 13 deletions gradlew
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
#

##############################################################################
#
Expand Down Expand Up @@ -55,7 +57,7 @@
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
Expand All @@ -80,13 +82,12 @@ do
esac
done

APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit

APP_NAME="Gradle"
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}

# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s
' "$PWD" ) || exit

# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD=maximum
Expand Down Expand Up @@ -133,22 +134,29 @@ location of your Java installation."
fi
else
JAVACMD=java
which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
if ! command -v java >/dev/null 2>&1
then
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.

Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
fi

# Increase the maximum file descriptors if we can.
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
Expand Down Expand Up @@ -193,11 +201,15 @@ if "$cygwin" || "$msys" ; then
done
fi

# Collect all arguments for the java command;
# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of
# shell script including quotes and variable substitutions, so put them in
# double quotes to make sure that they get re-expanded; and
# * put everything else in single quotes, so that it's not re-expanded.

# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'

# Collect all arguments for the java command:
# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
# and any embedded shellness will be escaped.
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
# treated as '${Hostname}' itself on the command line.

set -- \
"-Dorg.gradle.appname=$APP_BASE_NAME" \
Expand Down
23 changes: 13 additions & 10 deletions gradlew.bat
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@rem SPDX-License-Identifier: Apache-2.0
@rem

@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
Expand All @@ -26,6 +28,7 @@ if "%OS%"=="Windows_NT" setlocal

set DIRNAME=%~dp0
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%

Expand All @@ -42,11 +45,11 @@ set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if %ERRORLEVEL% equ 0 goto execute

echo.
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
echo.
echo Please set the JAVA_HOME variable in your environment to match the
echo location of your Java installation.
echo. 1>&2
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2

goto fail

Expand All @@ -56,11 +59,11 @@ set JAVA_EXE=%JAVA_HOME%/bin/java.exe

if exist "%JAVA_EXE%" goto execute

echo.
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
echo.
echo Please set the JAVA_HOME variable in your environment to match the
echo location of your Java installation.
echo. 1>&2
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2

goto fail

Expand Down
13 changes: 13 additions & 0 deletions modules/decima-app/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
plugins {
id 'java-library'
}

dependencies {
implementation project(':platform-util')

// implementation project(':decima-game-death-stranding')
// implementation project(':decima-game-horizon-zero-dawn')
implementation project(':decima-game-horizon-forbidden-west')
implementation project(':decima-game-horizon-zero-dawn-remastered')
implementation project(':decima-game-until-dawn')
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package com.shade.decima.app;

import com.shade.decima.app.ui.GraphInspector;
import com.shade.decima.game.FileSystem;
import com.shade.decima.game.hfw.rtti.HFWTypeFactory;
import com.shade.decima.game.hfw.rtti.HFWTypeReader;
import com.shade.decima.game.hfw.rtti.HorizonForbiddenWest;
import com.shade.decima.game.hfw.rtti.HorizonForbiddenWest.EPlatform;
import com.shade.decima.game.hfw.storage.ObjectStreamingSystem;
import com.shade.decima.game.hfw.storage.StorageReadDevice;
import com.shade.decima.game.hfw.storage.StreamingGraphResource;
import com.shade.decima.game.hfw.storage.StreamingObjectReader;
import com.shade.util.NotNull;
import com.shade.util.io.BinaryReader;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.swing.*;
import java.io.IOException;
import java.nio.file.Path;

public class Launcher {
private static final Logger log = LoggerFactory.getLogger(Launcher.class);

public static void main(String[] args) throws Exception {
var path = Path.of("E:/SteamLibrary/steamapps/common/Horizon Forbidden West Complete Edition");

log.info("Loading type factory");
var fileSystem = new HorizonFileSystem(path);
var factory = new HFWTypeFactory();

log.info("Loading streaming graph");
var graph = readStreamingGraph(fileSystem, factory);

log.info("Loading storage files");
try (StorageReadDevice device = new StorageReadDevice(fileSystem)) {
for (String file : graph.files()) {
device.mount(file);
}

ObjectStreamingSystem system = new ObjectStreamingSystem(device, graph);
StreamingObjectReader reader = new StreamingObjectReader(system, factory);

log.info("Opening UI");
ui(graph, reader);
}
}

private static void ui(StreamingGraphResource graph, StreamingObjectReader reader) throws Exception {
UIManager.getDefaults().addResourceBundle("com.shade.decima.app.ui.util.Bundle");
UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());

SwingUtilities.invokeAndWait(() -> GraphInspector.show(graph, reader));
}

@NotNull
private static StreamingGraphResource readStreamingGraph(@NotNull FileSystem fileSystem, @NotNull HFWTypeFactory factory) throws IOException {
try (var reader = BinaryReader.open(fileSystem.resolve("cache:package/streaming_graph.core"))) {
var result = new HFWTypeReader().readObject(reader, factory);
return new StreamingGraphResource((HorizonForbiddenWest.StreamingGraphResource) result.object(), factory);
}
}

private record HorizonFileSystem(@NotNull Path source) implements FileSystem {
@NotNull
@Override
public Path resolve(@NotNull String path) {
String[] parts = path.split(":", 2);
return switch (parts[0]) {
case "source" -> source.resolve(parts[1]);
case "cache" -> resolve("source:LocalCache" + EPlatform.WinGame).resolve(parts[1]);
default -> throw new IllegalArgumentException("Unknown device path: " + path);
};
}
}
}
Loading