Modding: not supported ...but is allowed!

Talk about Corven
Post Reply
User avatar
Farrier
Posts: 11
Joined: Sun Jul 05, 2020 6:10 pm

Modding: not supported ...but is allowed!

Post by Farrier »

Over in Discord, we have confirmation that there will be no official modding capability for Corven.
In #questions-for-the-team, on 06/09/2020 wrote:Bowen Bloodgood: Ok I've got one.. are you considering MOD support?
Corv: Not at the moment, no. That is a LOT of work to make such a system work and it's out of the scope of our possibilities at the moment. I could see it if we reach insane pledge levels. But more realistic would be in Corven 2
But it's a Unity game, and the mono assemblies are unstripped, which means we can just do it ourselves, hackishly. Asked about this, Corv replied:
In #questions-for-the-team, on 07/05/2020 wrote:Corv: I have no problem with anyone modding Corven
In #questions-for-the-team, on 07/06/2020 wrote:Corv: "Modability" won't change from the demo - will be the same at release.
I confirmed that the latter should not be taken as a promise. Things always change in game development, and compromises need to be made in order to get closest to the game they want to make. Ease of moddability was never a pledge goal, it's not something we've paid for, but if we're lucky, they won't need to change their build pipeline, and we'll be able to mod it :)

Since he's confirmed he doesn't plan to change that, we can start working on modding it. I'll post my progress, if any, in this thread.

User avatar
Farrier
Posts: 11
Joined: Sun Jul 05, 2020 6:10 pm

Re: Modding: not supported ...but is allowed!

Post by Farrier »

In this post, I'll keep a list of people's suggestions for mods. What would you like to see?

Stuff we need for modding to work:
  • Proof of Concept system: reverse compiling. This is the obvious necessary prerequisite to anything else.
  • Proof of Concept system: asset extraction.
  • Proof of Concept system: recompiling.
  • Proof of Concept system: debugging.
  • Proof of Concept mod: hook existing public methods.
  • Proof of Concept mod: logging.
  • Proof of Concept mod: UI (buttons, menus etc).
  • Proof of Concept mod: user input: map new inputs, and intercept existing controller events.
  • Proof of Concept mod: import a new asset.
  • Proof of Concept mod: spawn a new instance of an asset.
  • Proof of Concept mod: shared libraries.
  • Proof of Concept mod: physics? Not sure Corven even uses physics at all: I see no need for it.
  • Debugging mod: Runtime inspection of the GameObject hierarchy. This will make modding vastly easier.
  • Library to simplify persistence for mod data (ie savegames and settings for mods).
  • Proof of Concept mod: reskinning.
  • Proof of Concept mod: re-meshing.
  • Proof of Concept mod: modifying loot lists.
  • Proof of Concept mod: modifying level data.
  • Proof of Concept mod: adding new level.
  • Proof of Concept mod: new dialog, and modify existing dialog.
  • Proof of Concept mod: new characters, mobs.
  • Proof of Concept mod: new quests, and modify existing quests.
  • Library for dependency management (eg what happens if two mods modify a single person's dialog?)
  • Documentation for mod users.
  • Documentation for modders.
Until we've proven we can do all the above, we can't really claim we're able to "fully mod" the game. This is why Corv as, wisely, opted not to include modding support in the game. It's a Big Lift.

So far, people have suggested:
  • Nude mod (of course!)
  • Minimap
  • Cheat console
  • Lua hooks/console
  • "Naturalistic" dialog backgrounds (wood, leather, etc)
  • Linux port
  • Camera-change/first-person-view (need very close fog layer. Will still look bad with "see inside houses", stuff you're not meant to see behind, or up close, and windows, water, doorways, etc).
  • VR mod?
  • Unofficial fan bugfix mod (only needed if/when Corv ever stops supporting the game)
We can be fairly confident these are possible, since we got about halfway through this list with Underworld Ascendant, until they switched to il2cpp pipeline, making modding next to impossible.
Still, I tend to flake out on projects, so based on my previous performance, chances of me achieving even just item one are minimal.
But I intend to at least try.
Last edited by Farrier on Sat Jul 25, 2020 4:36 pm, edited 8 times in total.

User avatar
Farrier
Posts: 11
Joined: Sun Jul 05, 2020 6:10 pm

Re: Modding: not supported ...but is allowed!

Post by Farrier »

First looks:

So... what do we know?

Taking a peek into Assembly-CSharp.dll and similar files, I see that Corven uses (or at least, the demo includes) a few libraries.

For now, I just have the vaguest possible guesses as to their use, based on their names and method and class-names within them. This is an incomplete list which I'm sure will change during development of the game! We shouldn't assume any of these libraries will remain in, but it gives us an idea of the kinds of "power tools" we can call on within the game, without rolling our own solution.
  • Nicholas Hoellermeier's ORK framework. Probably the most significant one, this will either make things a lot easier for modding, or a lot harder: not sure yet. It appears to have a whole BUNCH of nice features for dialogs and HUDs, NPC management, and more. The giant potential downside is... it seems to be all clicky-clicky, UI-based editing. So modding it may require digging into how the ORK framework stores all of that stuff. Depends if there's a programmatic API as well. More research needed.
  • PixelCrushers.DialogueSystem.ORKFrameworkSupport - conversation trees?
  • Animmal.YezugaRPG - clothing and equipment management?
  • GAIA: https://assetstore.unity.com/packages/t ... ator-42618 - probably manages most of the gorgeousness, hooks in SpeedTree
  • AwesomeTechnologies - VegetationStudioPro, Infinite terrain, etc.
  • Triangle.NET and MeshCombineStudio - some geometry stuff?
  • RealSoftGames' AdvancedLoadingScreen - the loading screen?
  • DuoloGames.UI - for UI stuff and tweening?
  • cCharkes - Screen resolutions?
  • EntriPo - weather/clouds?
  • FoxieGames - serializing? Savegames?
  • JBooth.MicroSplat and LightingBox.Effects - lighting effects?
  • ShaderControl - shaders?
  • TMPro - text display/font management?
One nice sign is that over in Assembly-CSharp-firstpass.dll, it seems to contain a whole LUA interpreter! :) Might make modding the UI a whooooole lot easier if we can find a way to just use Lua plugins! Definitely a worthy avenue to explore.

First, though, I've downloaded the ORK framework, subscribed to their forums, and I'm gonna try to figure out how it stores data by following one of their turorials to create a simple ORK game, then tweaking parameters and see which files get changed.

User avatar
Corv
Site Admin
Posts: 34
Joined: Sun Feb 02, 2020 2:08 am

Re: Modding: not supported ...but is allowed!

Post by Corv »

For the record here as well: modding is allowed and there will be no active effort to prevent it by us.

(though it is a bit weird to see reverse engineering going on before the game is even out ;) )
Image

User avatar
Farrier
Posts: 11
Joined: Sun Jul 05, 2020 6:10 pm

Re: Modding: not supported ...but is allowed!

Post by Farrier »

Modding tools:

Reading the source with dnSpy

For this, you want dnSpy: https://github.com/0xd4d/dnSpy, a project built on various other projects (Roslyn, ILSpy, dnLib, etc), which integrates them into a single smooth interface which reverse compiles, debugs, and recompiles .NET assemblies. Which is to say: Unity games.

Installing dnSpy:
Go to https://github.com/0xd4d/dnSpy/releases, download dnSpy-net472.zip and unzip it wherever you want to run the program from: no installer, sorry! Extracting to your desktop is fine, so is to C:\Program Files. You do you.
  • Run dnspy.exe from the folder you installed it into (well, I put a shortcut on my start bar).
  • File -> Open -> <game install folder>\<GameName>_Data\Managed\Assembly_CSharp.dll
  • That’ll load the C# code into dnSpy. You can navigate the source’s namespace tree on the left. Generally speaking, the “-” namespace is where all the interesting code for a game is, and everything else is libraries, which are cool because you can call them.
This is all I did for the post above, talking about the libraries.

Debugging with dnSpy

Runtime debugging with breakpoints and inspection makes any kind of development a million times easier. Even better if you can make changes, and recompile to see the effect of those changes. Getting dnSpy able to do this takes a bit of work, but it's well worth it. This is covered in more depth at: https://github.com/0xd4d/dnSpy/wiki/Deb ... nity-Games, but in short, having installed dnSpy above, you then need to replace Corven’s .NET library with one that has debugging hooks.
  • Find out what Unity version your copy of the game was built with. This is important. Your extension will break in various unpredictable ways if you make it for the wrong version of Unity! At the moment (Demo v1.2), it’s "2019.2.15.46892". But there’s a good chance this'll change in future versions, so for each version, browse to where you have the game installed, rightclick Corven.exe, and click the “details” tab. You should see something like "File Version: 2019.2.15.46892", which is the Unity version the exe was compiled with. If it’s different, then change the following instructions appropriately.
  • While you’re in the game's install folder, maybe slap a shortcut to it on your desktop, since you’re going to be going to that folder a lot. Or at least keep the file explorer window open.
  • Also while you’re in there, make a backup of the file MonoBleedingEdge/EmbedRuntime/mono-2.0-bdwgc.dll within there. (The location of the mono file will depend on which game you are debugging - this is currently was the location for Corven.)
  • Download Unity-debugging-2019.x.zip from https://github.com/0xd4d/dnSpy/releases/tag/unity (if this link changes, you can navigate here from the dnSpy installer page). Extract it somewhere, too. Within it, you’ll find a whole bunch of folders for the Unity versions released that year.
  • Overwrite the above MonoBleedingEdge\EmbedRuntime\mono-2.0-bdwgc.dll from your game’s install folder with the file Unity-debugging\unity-2019.2.15\win64\mono-2.0-bdwgc.dll - this replaces the mono DLL with one that has debugging hooks in it! Note: at the moment, this file doesn't exist in the downloaded archive! The compiled version only goes up to 2019.1.8, which means debugging this is going to be PAINFUL, as we're going to either have to ask the dnSpy devs to compile a new version, or compile one ourselves for this version.
You're now ready to debug!
  • Launch Corven the normal way. I’d recommend to run it in a window, it’s just way easier to switch to the debugger that way.
  • Run dnspy.exe from the folder you installed it into (well, I put a shortcut on my start bar).
  • Click Debug -> Attach to Process… (Note: NOT “Attach to Process (Unity)”!)
  • If a Windows Firewall warning comes up, click “Allow access”.
  • Select the Corven.exe process and click “Attach”
  • If everything went right, the game’s source tree should automagically load in the lefthand panel, as well as those for all the DLLs it has loaded. You can navigate around, set breakpoints (even conditional ones), click things to go to the definition, find usages, edit the code and recompile, all the stuff you expect in a decent IDE.
Extracting assets with uTinyRipper

Download the program https://sourceforge.net/projects/utinyripper/ and run it on one of the .asset or .level files. It will take a long time to run (some games take multiple hours on a beefy gaming machine), and loading the extracted resources back into Unity can take FOREVER (maybe a day) and typically won’t work perfectly. Main static scenery may not load at all, and you will almost certainly have to change the lighting on all the objects that show up purple. But it gives a starting point, and lets you extract character models, etc.

Reverse compiling with dnlib

This uses dnlib (https://github.com/0xd4d/dnlib).

Making a mod plugin for UnityModManager
This relies on using UnityModManager aka UMM (https://www.nexusmods.com/site/mods/21/) which in turn builds on Harmony (https://github.com/pardeike/Harmony) to insert hooks around C# methods.

Inserting assets using Unity asset DLLs
This relies on compiling asset DLLs using Unity, that can then be hooked in using UMM hooks. It definitely remains an open challenge, though, because Unity doesn’t allow you to embed code within these dll packages.

A solution may well involve separate DLLs for assets and code, with the code ones being built using something like Visual Studio.

Runtime debug displays

I've never got this to work, but my plan if I can is to use the Unity Asset Store assets like: They don't, out of the box, "just work", though: I ran into the same problem that unity packages don't allow code to be embedded in dll packages.
Someone over on https://forum.unity.com/threads/runtime ... 220/page-3 claims to have got it working using Harmony, but has sadly not replied to messages asking how the heck they managed to make it work.
Last edited by Farrier on Sat Jul 18, 2020 7:25 pm, edited 2 times in total.

User avatar
Farrier
Posts: 11
Joined: Sun Jul 05, 2020 6:10 pm

Re: Modding: not supported ...but is allowed!

Post by Farrier »

Corv wrote:
Sat Jul 18, 2020 3:35 pm
For the record here as well: modding is allowed and there will be no active effort to prevent it by us.

(though it is a bit weird to see reverse engineering going on before the game is even out ;) )
Heee :D Thank you so much! I really appreciate this kind of support of the modding community.

And yes, we're exceptionally lucky in Corven that so early in the development you've given us a whole freaking working demo, built using the same dev pipeline as the whole game probably will be. That really lets us get up to speed with modding it!

No matter what you say, of course, I'll always assume that everything can change if it has to: legalese or financial or marketing BS could force you to ask us to stop modding, or performance concerns require you to use il2cpp, etc. You're a reasonable person, and support us modding the game, so I've faith you'd explore reasonable options, but I know your priorities in this project are always going to be (have to be):
  • fulfill paid pledges
  • release the game
  • make the game you want to make
... probably in that order of priority. "Allow mods" was never a paid pledge, so I for one plan to support any hard decisions you're forced to make about modding, rather than endanger the project.

But it's super unlikely, so I shan't worry about it, and will have fun modding! Worst case, I still learn a TON of useful skills!

User avatar
Farrier
Posts: 11
Joined: Sun Jul 05, 2020 6:10 pm

Re: Modding: not supported ...but is allowed!

Post by Farrier »

So today I'm trying to get ORK framework working, and find where it stores stuff.

Their getting started tutorial was last updated two years ago, which likely means there'll be glitches with Unity 2020. I tried it with 2019 and the scene wizard didn't appear in the relevant menu after importing the framework: instead there were errors in the console that a quick google-search suggested were Unity-2019-specific.

Unity 2020 works better, though the animation in part 3 of the tutorial does not play: not sure if that's a version thing, or just something I broke somehow.

Yikes, there are 50 tutorials in this chain, I'm only at tutorial 9. Some other day, then. I'll download the finished project, and see what I can learn from it.

Libraries

First, let's compile it, and see what libraries get included, just as part of ORK and Unity.

Wow, OK, there is NO Assembly-CSharp.dll or Assembly-CSharp-managed.dll! That's a shocker! Looks like the ORK framework works entirely without C#!

Equally, the only DLLs I see are core Unity ones, plus the ORK Framework. That suggests that all the libs we see in the Corven demo were either added by Corv, or by some other package he's using.

This blows most of my plans for reverse-compiling C# stuff right out of the water!

Well, now I've compiled it, that means I can find out where ORK stores its stuff...

Comparing

To find where stuff is stored, we need to, tweak some things to see which files change. First, let's change NOTHING.

I'm using this command from within a "Build" folder, to identify differences between the files in there, and the files in a "Build2" folder:

Code: Select all

find . | while read -d $'\n' file; do if [ -f "../Build2/$file" ] ; then diff "$file" "../Build2/$file"; fi ; done
That finds only:

Code: Select all

Binary files ./ORK Demo_Data/globalgamemanagers and ../Build2/./ORK Demo_Data/globalgamemanagers differ
Comparing them, it seems that file contains a per-build unique hash (BuildGUID), which is the only thing that differs. I can apparently set a build option to zero out the GUID, so they wouldn't differ, but I don't know how to set that build option, and can't find it in a quick search. I can likely safely ignore changes in this file, though: ORK is unlikely to store its settings here.

Strings

I think Corven uses a different conversation library, so this may well be different... but strings are strings, so I expect them to be stored at least roughly in the same position.

Changing just a text string in dialog, I get:

Code: Select all

Binary files ./ORK Demo_Data/sharedassets1.assets and ../Build2/./ORK Demo_Data/sharedassets1.assets differ
That's where I expected text to be stored, so that doesn't really tell me anything new, though it does confirm that the "diff" process is working! :)

Events

Most of the complexity in ORK is abstracted away behind a clicky-clicky thing called "events" which are chained-together sequences of boxes in the editing UI, each box having properties.
So next, I'll change a simple boolean flag in an event script: turning off "Block Player Control" in the dialog event script.

Interesting. Still only the sharedassets1.assets file differs. Are event scripts stored as assets, then? Yes they are!

Image

The above simple dialog event gets stored in the asset file as a bunch of Unity asset strings, which I call "UStrings", each comprised of: leading nulls to align to 4 bytes; a 4-byte little-endian int length; then the characters of the string.

Each event object seems to have a header of four strings, all fields that I recognize from the event's .asset file that ORK saves in the project folder when you create it. My comments after the dash, with the fieldname from the ORK file:

Code: Select all

simpleDialogue - the name of the event, "m_Name".
7/19/2020 8:35:54 PM - seems to be the time that I saved the event, "time".
2.29.3 - ORK framework version, not necessarily dotted numbers, I see Corven has "2.29.2 BETA". "time".
gameEvent - event type. "data.name".
Then comes this XML block, "data.xmlData" from the event .asset file, though in that file, the string contains many escape codes like \", \n and \t: in the sharedassets1.assets file, all those codes are replaced by the character they were encoding. The XML contains all the settings for the event, the whole thing again held in a UString. My comments after a dash:

Code: Select all

<gameEvent startIndex="0" > - Event type. In the Corven Demo I see startIndex sometimes being non-zero, not sure what it does.
	<_bool blockingEvent="True" blockMoveAI="False" blockActorMoveAI="False" 
	  blockControlMaps="False" clearFoundObjects="True" closeAllDialogues="False"
	  blockPlayer="True" blockCamera="True" clearBlocks="False" mainCamera="True"
	  cameraTag="False" destroyPrefabs="False" overrideNodeName="False" /> - Checkboxes from event properties, except the last three are new to me.
	<_floatarrays>
		<nodePosition 38 38 />
	</_floatarrays> - Where the box is shown in the editor UI, in pixels.
	<_string>
		<cameraName><![CDATA[]]></cameraName>
		<nodeName><![CDATA[]]></nodeName>
	</_string>
	<_subarrays>
		<actor>
			<0 type="1" memberIndex="0" > - Details of who is talking. In this case, type=1 means the player is talking.
				<_bool isEventObject="False" isFindObject="False" onlyBattle="False"
				  useRoot="False" useSceneName="False" setName="False" />
				<_string>
					<childName><![CDATA[]]></childName>
				</_string>
				<_subarrays>
					<portrait>
					</portrait>
				</_subarrays>
			</0>
		</actor>
		<waypoint>
		</waypoint>
		<prefab>
		</prefab>
		<audioClip>
		</audioClip>
		<step>
			<0 next="-1" > - This is the second box in the event.
				<_bool active="True" overrideNodeName="False" />
				<_floatarrays>
					<nodePosition 38 133 />
				</_floatarrays>
				<_string>
					<nodeName><![CDATA[]]></nodeName>
					<_type><![CDATA[ShowDialogueStep]]></_type>
				</_string>
				<dialogue type="0" guiBoxID="1" notificationQueue="2" actorID="0" 
				  portraitType="0" consoleTypeID="0" clipID="0" cancelNext="-1" >
					<_bool waitClose="True" waitCloseNotControlable="False" lockFocus="False"
					  useSpeaker="True" atActorPosition="False" speakerName="True"
					  speakerPortrait="False" ownPortraitPosition="False" useTitle="False"
					  outputToConsole="False" useAudio="False" stopAudio="True"
					  onSpeaker="False" updateChoiceStates="False" allowCancel="False"
					  blockAccept="False" afterAudio="False" wait="False" />
					<_floatarrays>
						<posOff 0 0 />
					</_floatarrays>
					<_string>
						<posChildName><![CDATA[]]></posChildName>
						<childName><![CDATA[]]></childName>
					</_string>
					<_stringarrays>
						<message>
							<0><![CDATA[Hi, this is a simple dialogue!
It has been edited.]]></0> - Here's the text I modified in the Strings part of this post.
						</message>
					</_stringarrays>
				</dialogue>
			</0>
		</step>
	</_subarrays>
</gameEvent>
Most of this makes sense. XML implies there's a DOM, at least briefly when this data is read in and converted to some other in-memory format, which implies that script events MIGHT be modifiable using DOM manipulation?

The next step for events will be showing that they can be modified, and new ones created... preferably at runtime!

Other Settings
Events are kinda special in that they are saved as external files. Most other stuff in ORK is just "Settings". A lot of these seem to be saved in sharedassets0.assets, with a similar structure to the above: a header (usually just a name identifying the setting, eg "inputKeys", "battleEvent", "statusValues", etc), then a block of XML. So hopefully these can be handled much like events.

Next steps
That's enough for today, I think, but the next step probably has to be to find a way to find, modify and create events and other ORK values *at runtime*.

For this, it turns out there's a C# API at http://api.orkframework.com/namespace_o ... ework.html.

It's a big API, but a guide to "where to find what" exists at http://orkframework.com/tutorial/howto/ ... e-is-what/

But while there's an API, I don't know yet how to call the API on an existing project. Much to research!

Post Reply