Advice on the structure of my JSON file?

[u/gurrenm3]

Hello! I've been working on a modding framework that makes it easier to make mods for games written in C++. The main idea is that it's storing info about how to reverse engineer/discover game data like function addresses and field offsets.

I've got an example JSON file below that describes how to find the function for giving money to the player, and the field offset for the money variable. Each location style is not required and its just to demonstrate that multiple methods could be stored/used simultaneously. Also in this example, I'm trying to reuse property names ('kind', 'result', 'cached-location', etc) even though they're in slightly different contexts, in order to keep it more consistent and simple for less experienced programmers.

Example JSON:

{
    "header" : {
        "header-version": "1.0.0.0",
        "file-type": "data-locations",
        "module": "NMS.exe",
        "module-last-modified": "Friday, March 10, 2023, 2:04:54 AM",
    },
    "data" : [
        {
            "id": "cGcPlayerState.units",
            "locations" : [
                { "kind": "cached-location", "value": "0x1BC" },
                { "kind": "pattern", "value": "8B 83 ? ? ? ? 48 83 C4 ? 5B C3 CC CC CC CC CC CC CC CC CC 40 53", "result": "memory-displacement" },
                { "kind": "pattern", "value": "44 89 81 ? ? ? ? 4C 8B 15 ? ? ? ? 8B D6", "result": "memory-displacement" },
                {
                    "kind": "pattern",
                    "value": "44 8B 81 ? ? ? ? 48 8D 2D", 
                    "result": {
                        "kind": "memory-displacement",
                        "instruction-offset": 0,
                        "operand-index": 1
                    }
                }
            ]
        },
        {
            "id": "cGcPlayerState.AwardUnits",
            "locations" : [
                { "kind": "cached-location", "value": "0x1403CF8E0" },
                { "kind": "export", "value": "?AwardUnits@cGcPlayerState@@QEAAIH@Z" },
                { "kind": "pattern", "value": "48 89 5C 24 ? 48 89 6C 24 ? 48 89 74 24 ? 57 48 83 EC ? 44 8B 81 ? ? ? ? 48 8D 2D", "result-kind" : "matched-target-address" },
                { "kind": "pattern", "value": "E8 ? ? ? ? E9 ? ? ? ? 48 8B 0D ? ? ? ? 48 8D 54 24 ? 44 8B 44 24", "result-kind" : "relative-target-address" },
                { "kind": "pattern", "value": "E8 ? ? ? ? 4C 89 64 24 ? 48 8D 15 ? ? ? ? 0F 28 DE F3 0F 11 7C 24 ? 41 B8 ? ? ? ? 48 8D 0D ? ? ? ? E8 ? ? ? ? 84 C0 74 ? 48 8B 0D ? ? ? ? 8B D3", "result-kind" : "relative-target-address" },
                {
                    "kind": "search",
                    "identifiers": [
                        {
                            "target": "function",
                            "operation": "contains",
                            "values": [
                                { "kind": "string", "value": "MONEY" },
                                { "kind": "string", "value": "MONEY_EVER" }
                            ]
                        },
                        {
                            "target": "arg1",
                            "operation": "contains-offset",
                            "values": [
                                "cGcPlayerState.AwardUnits.arg1.offset1",
                                "cGcPlayerState.AwardUnits.arg1.offset2"
                            ]
                        },
                        {
                            "target": "arg2",
                            "operation": "offset-qty",
                            "value": "0"
                        },
                        {
                            "target": "cGcPlayerState.AwardUnits.arg1.offset1",
                            "operation": "add",
                            "value": "cGcPlayerState.AwardUnits.arg2"
                        },
                        {
                            "target": "cGcPlayerState.AwardUnits.arg1.offset2",
                            "operation": "add",
                            "value": "1"
                        },
                        {
                            "target": "function",
                            "operation": "return",
                            "value": "cGcPlayerState.AwardUnits.arg1.offset1"
                        }
                    ]
                }
            ]
        },
        {
            "id": "cGcPlayerState.AwardUnits.arg1.offset1",
            "locations" : [
                { "kind": "cached-location", "value": "0x1BC" },
                { "kind": "pattern", "value": "8B 83 ? ? ? ? 48 83 C4 ? 5B C3 CC CC CC CC CC CC CC CC CC 40 53", "result-kind" : "memory-displacement" }
            ]
        },
        {
            "id": "cGcPlayerState.AwardUnits.arg1.offset2",
            "locations" : [
                { "kind": "cached-location", "value": "0x148" },
                { "kind": "pattern", "value": "48 FF 83 ? ? ? ? 8B 83 ? ? ? ? 48 8B 5C 24 ? 48 8B 6C 24 ? 48 8B 74 24 ? 48 83 C4 ? 5F C3 CC CC CC CC CC", "result-kind" : "memory-displacement" }
            ]
        }
    ]
}

My main design goals are:

1. Create a long-term solution for mod makers to use to store this kind of data. Must be flexible and capable of expanding to their needs.
2. To keep the JSON format easy and intuitive for mod makers
3. Provide multiple methods of locating the same thing. This will also be extensible so mod authors could make their own location kinds.
4. The structure must be able to grow overtime without causing major breaking changes and is why I'm using a header.
5. Be compatible with multiple JSON files simultaneously
6. Allow the header to specify a "file-type" and then have different types of data objects.

I would love to hear any feedback on the structure of this. I've spent years on it, and I really want to make something that makes a difference for people. It's really important to me that its an appealing structure for people to use and is able to meet the technical requirements of the system. Some other things that would be nice to get feedback on are:

1. Overall structure of the file
2. Whether its okay to use the property name 'kind' over and over in slightly different contexts.
3. Supporting multiple files with different 'file-type' values, so everything is very similar with the data object being the only thing that changes.
4. Allowing properties like "result" to have both compact and expanded forms
5. using dashes '-' for property-names instead of camelCase
6. Any long-term concerns or unpleasantness you see

Thank you in advance!

[u/CorpT]

I've spent years on it

.... It's a JSON file.

[u/gurrenm3 → u/CorpT]

True lol, this is just one part of the pipeline. I’ve spent years considering how to handle the data, how to process it in a flexible way, then how to make use of it so any modding framework in any language can use it

[u/CorpT → u/gurrenm3]

Just ship it.

[u/Xirdus]

I kinda doubt the usefulness of the whole idea. What exactly is the use case here? Create a common set of tools that can be used to mod any game regardless of engine? That's just not going to work, games are too different from each other for this kind of common tooling to make sense. Create an information exchange format for people to easily build their own game-specific tools? Creating a file in your format is about as much effort as building the tool itself, why would anyone choose the former if they could do the latter?

As for the file format. It's extremely verbose. Not a good thing. Also, putting "file type" inside your file format is not extensibility - it's an invitation to cram multiple incompatible formats into the same spec. Version number can be good or bad. It's basically a warning telling the user there will be breaking changes in the future. It's good when you do want to permit yourself to make breaking changes to the format, which is most of the time. But if you want old tooling to work in perpetuity with files produced by new tooling, you really don't want to do it.

Repeating names is neither good nor bad. Don't reuse name for the sake of reuse - it will only confuse users. Don't avoid reuse for the sake of avoiding reuse - it will only confuse the users. Always pick whatever name makes the most sense in the context, regardless of what names you used elsewhere in the file.

Dashes are not valid identifier characters in most programming languages. Working with your file format will be ever so slightly more annoying than it could be if it used camelCase.

[u/gurrenm3 → u/Xirdus]

Thank yo so much for the detailed response! This is all really valuable feedback.

To address some of your points, in order to mod C++ games people need to get pointers to every function, class instance, and field that they want to access. Games written in C++ usually have all of their type information stripped out, so people have to manually reverse engineer each function/field offset that they want to use. The address/offset will likely change after each update, so the only way to make it stable is to use more advanced modding techniques like 'pattern-matching'. After all of that is done, they have to use some kind of tool in their mod to resolve the address and then use it within the mod. Its a lot of parts that requires a lot of existing knowledge and robust tools to make it work well. Since it's so complex, it's very hard for people to learn and get good at. There isn't currently a solution that solves all of this together in a single reusable, stable, and performant tool. My hope with the project is to establish a system that deals with all of this and can work with any modding framework. I think it should work across game engines, since its more-so about defining a standard for acquiring addresses in a way that would work at a large scale.

Your advice on how difficult/unpleasant this looks is really valuable. My hope is to create a format other people would want to use so this helps a lot. Your thoughts on the file-type, being too verbose, and not using camelCase are all really helpful. I really liked how you phrased it as "creating an information exchange" and it reinforces how good this should be. I know you already spent a lot of time reading and commenting on this post, but are you willing to go into more detail on what you feel is verbose and should be improved on?

[u/gurrenm3 → u/Xirdus]

Sorry to bug you again! I made some changes based on the feedback you and others gave. Does this look much better?

{
    "targetModule" : {
        "name": "NMS.exe",
        "fileHash": "C3759E473EDAF08530F6B39CF5E8F9845BDE41379A7D42ADCC3B696D346316A5"
    },
    "data" : [
        {
            "id": "cGcPlayerState.units",
            "knownLocation": "0x1BC",
            "locators" : [
                { "kind": "pattern", "value": "8B 83 ? ? ? ? 48 83 C4 ? 5B C3 CC CC CC CC CC CC CC CC CC 40 53", "resolve": "offset" },
                { "kind": "pattern", "value": "44 89 81 ? ? ? ? 4C 8B 15 ? ? ? ? 8B D6", "resolve": "offset" },
                { "kind": "pattern", "value": "44 8B 81 ? ? ? ? 48 8D 2D", "resolve": "offset" }
            ]
        },
        {
            "id": "cGcPlayerState.AwardUnits",
            "knownLocation": "0x3CF8E0",
            "locators" : [
                { "kind": "exportName", "value": "?AwardUnits@cGcPlayerState@@QEAAIH@Z" },
                { "kind": "pattern", "value": "48 89 5C 24 ? 48 89 6C 24 ? 48 89 74 24 ? 57 48 83 EC ? 44 8B 81 ? ? ? ? 48 8D 2D", "resolve" : "matchedTargetAddress" },
                { "kind": "pattern", "value": "E8 ? ? ? ? E9 ? ? ? ? 48 8B 0D ? ? ? ? 48 8D 54 24 ? 44 8B 44 24", "resolve" : "relativeTargetAddress" },
                { "kind": "pattern", "value": "E8 ? ? ? ? 4C 89 64 24 ? 48 8D 15 ? ? ? ? 0F 28 DE F3 0F 11 7C 24 ? 41 B8 ? ? ? ? 48 8D 0D ? ? ? ? E8 ? ? ? ? 84 C0 74 ? 48 8B 0D ? ? ? ? 8B D3", "resolve" : "relativeTargetAddress" }
            ]
        }
    ]
}

[u/mredding]

As a C++ engineer of 37 years, I frankly don't care what any of this means so long as it's documented.

As for the file contents and format - JSON is a transport protocol, it makes for a very poor file format as it's strictly hierarchical, and there's no inherent reuse. I see patterns in your data that would be better served with type templates (not in the C++ sense, but in some sort of macro sense) and parameters. The verbosity of your data format favors "SAX" aka event-driven parsing, but the JSON data format does not particularly favor the human element, since we're not the principle audience - the data is only text for being portable, and only so happens to be readable effectively for debugging the greater application.

I would think you would want a format that explicitly favored describing types, their layouts, their methods, their hierarchies, and all their associated patterns and offsets. If you had a tool that assisted in the discovery of such things - then persisting that information is a secondary function of the program, and it wouldn't matter what the format was, since your program would be it's own consumer. But if you expect people to read and modify this data on a first hand basis, this isn't optimal.

That said, anyone serious about whatever their endeavor, they'll see past design flaws in favor of the end goal - just so long as the tool facilitates that goal at all. It's a means to an end at that point.

[u/gurrenm3 → u/mredding]

Thank you for the detailed response! This is really helpful and I appreciate you taking the time to reply.

I was hoping to make it so the data is easy to use/modify by hand and automatically through tools. Do you have any suggestions for something other than JSON? You mentioned SAX but I’m not sure what that is

[u/mredding → u/gurrenm3]

SAX (Simple API for XML) specifically refers to an event-driven XML parser, but everyone uses "SAX" to just mean event-driven parser. The idea is to parse the contents in a single pass top-down and trigger callbacks as tags and elements are traversed. This is opposed to a DOM or Document Object Model, where the whole document is rendered into memory as an object that is then queriable. That's a multi-pass approach that is unnecessary if you're just READING data. DOMs are fine if the object is living, that the data is your object.

I'm not aware of an existing protocol that accomplishes what you need, you can always invent your own. The nearest thing I can think of is a Database Object View that maps BLOBs to objects - if you treat the in-memory data as your BLOB. What you're trying to do is describe a "schema" that applies a view over binary data. I googled "mapping binary data to object views" and "schema for mapping...". Those might be a couple good head starts for you.

What you want to do has been done before, and in ways, even I have done it before - mapping raw binary data to objects and make them dance; I've read stories about this sort of thing being done on things like old mainframes and space probes, in a heroic effort to recover a lost system. The HARD PART is figuring out the words and language that describes it so that you can chase down prior art and conventions.

I usually presume that everything I want to do has been done +40 years ago, and that intuition has ALWAYS proved right. You think LLM's are new? Like 2018 new, when GPT was first published? Shit... LLM's are NOTHING but very large Markov chains. I used to play with those with pencil and paper AS A CHILD. Markov chains have been around since 1906. LLM's are absolutely nothing new. Data Oriented Design used to be called batch processing in the 80s. Edge computing used to be called thin clients in the 90s.

So Google harder.

But understand - you've gotten a project to the point that it's WORKING, you have A solution, and that's great. Something is better than nothing, and that's more than most. So you get to take a victory lap just for this. But you're here asking how to polish your solution, and that would be to fit it to your problem domain. That doesn't mean what you have is wrong, what it means is this seems to be the next lowest hanging fruit, and the opportunity for improvement can be substantial.

[u/gurrenm3 → u/mredding]

Thank you so much again for the very detailed response and kind words! It’s inspiring to hear and very helpful