Returning to The Sims: 128 errors in a fresh lot

This is part of TSaaS (The Sims as a Service), my series about adding LLM-plays capability to The Sims 1.

Errors

The main reason this project is possible at all is Simitone — an open-source .NET from-scratch implementation of The Sims 1’s engine, running against the original game’s content files. About a year ago, when I tested it as a human, I noticed that entering a new lot caused a bunch of “unhandled sims engine” errors, which was annoying but click-through-able. Some of them had no visible impact on the game world. Some affected a specific object (e.g. a sofa you can’t sit on), which I could then sell and buy again, solving the problem.

(Note the dialog half-visible behind this one. When it rains, it pours.)

Part of the project was moving the engine to an unattended container. This involved removing the exception dialogs (as no one is there to acknowledge them), but also made it harder to know whether an exception indeed indicates something bad or not. I decided to treat them like I would treat exceptions in a log of enterprise software - group by callsite, target the most frequent ones first, do the research to figure out which are harmful and which are benign, and then either fix or suppress.

The big boys

Loading the lot reported 128 exceptions. Most of them belonged to these types:

1. A sim having an 80-slot “values” (motives, skills, flags) array, where the engine wanted 101.
   I added the missing entries with sane values. One might ask “what is the sane value for a new attribute?”, but looking at the code, the new attributes were obvious things like “is a superstar” or “is a ghost”, which I can confidently set to false for newly-loaded sims.
2. Carry slot mismatch. Simitone knows that a sim should have 3 (head, hands, pelvis), but deferred to the savefile’s slot size (1).
   Fixed by setting the slot size to 3, initializing missing ones as empty

Eliminating both, I managed to get startup errors from 128 to 14, which is a nice start.

Here’s an example of #1, a Makin’ Magic script checking whether Mortimer is a ghost, but Mortimer comes from a pre-Makin’ Magic save and is missing the relevant flag’s slot:

Kitchen counter breakdown

After accepting 14 startup errors, I moved on and tried making the actual gameplay work. On occasion, the sims would fail when trying to cook themselves a meal. The food would just disappear, an exception would pop up, and the sim would forget about the matter completely. My only hint was that the meal process involved:

1. Taking food out of the fridge
2. Chopping stuff on the countertop
3. Cooking (this is where fires happen)
4. Taking the meal to eat (if “have a meal”) or on a flat surface (for “serve meal”) for others to utilize later

Since the sim was able to use the fridge and then errored out, I figured it’s related to countertops. The popping exception was obviously involved too, so it was time to do some heavy debugging.

Need to go deeper

Since I was able to modify the engine, I could add more debug printing into the exception handler and the Sims VM in general to get more context on what broke. The stack traces themselves are uninteresting. What IS interesting was the annotation I added:

(Tiled Counter:Tiled Counter)
 > main:3/2 #4096 (IP OUT OF RANGE) [restored]

[restored] marks a frame that came out of a savefile rather than from live execution, which was the smoking gun.
They were all “IP (Instruction Pointer) OUT OF RANGE”, coming straight from restored saves

It turns out each object has a small script it runs, looking a bit like assembly. The counter’s state looks like this:

...
"routine": { "name": "main", "id": 4096,
           "instructionCount": 2, "opcodes": [8224, 280] },
"ip": 3,
...

The interesting part here is that despite the routine having two instructions, we’re supposed to resume at instruction #3, which is… what?

The VM’s reaction was acceptable, but I had two problems with it:

1. The error manifested only when someone wanted to use the object, which was just the time where the object was interesting to the player.
2. The current handling caused the affected sim to drop its action completely, possibly losing items or disrupting whatever grand plan it was following.

We should prevent this error outright, at load time

Quick but incomplete

Thanks to the e2e framework I set up (worth its own post), I was able to create a regression test. I loaded the savefile, and ensured no object has an ip that is outside the bounds of its instruction array.

Getting this test green means eliminating the exceptions, which I did by reviewing every object when loading, and resetting ones where the ip was out of bounds of the routine. This made the exceptions go away and the meal prep complete successfully, so I could leave it there, but I wasn’t quite happy

Nagging feeling

The index being out of bounds happened because the object’s routine (coming from the content directory, in a specific Sims version) was no longer matching the one that was referred by the ip (which came from the save file). The out-of-bounds is an obvious indicator of such a mismatch, but such a mismatch can happen without an oob.
For instance, let’s assume there exists a fishtank, and its script looks something like this:

 1: flip graphic (fish swimming left are now right, etc)
 2: sleep 10s
 3: if fed_since > 1 sim-day:
 4:   goto L7
>5: else:
 6:   goto L1
 7: show dead fish graphic
 8: (stuck here, fish are dead)

And the ip happened to be set on L5, because the fish were alive and well. Now, we load this save into another version of content, where the script looks like this:

 1: flip graphic (fish swimming left are now right, etc)
 2: sleep 10s
 3: if fed_since < 1 sim-day:
 4:   goto L1
>5: show dead fish graphic
 6: (stuck here, fish are dead)

Simpler script, victory for all. But if we load the save that has ip set to L5, we’ll get a dead fishtank. While this is not an exception (a mechanical corruption), it’s definitely wrong (a logical corruption)

My conclusion was that without the original engine available to extract the original scripts from, we have no way of accurately translating ips to the right instruction.

Discarding everything

When faced with dirty data, I see 3 options:

1. Leave it as is
2. Clean it
3. Discard it

#1 was unacceptable here, because these corruptions could affect the game in a way that’s going to be hard to detect.
#2, as discussed previously, was only partially doable. Resetting index-bounds-violating objects was easy and solved the mechanical corruption, but silently-remapped lines would stay, causing an unknown amount of logical corruptions.
#3, the way I see it, is not so bad. I prefer losing object states, since:

1. The first time I load into a lot, I have not initiated any actions that affect object state, so I’m not afraid of losing a state I care about.
2. I get a guarantee that any actions I do initiate only interact with clean objects, so they won’t be ruined by objects with invalid state.

Good deal for me

At the end

The sims savefile has a “version identifier”, and the Simitone engine is using it when reading savefiles.
For instance, to make the savefile smaller, an object’s type is not its fully qualified GUID, but rather a reference to a lookup table in the start of the file that holds a mapping to the GUID. Simitone is built in an era where storage is cheap, so it gets rid of this table completely, dereferencing all objects to point to their direct GUIDs.
The same treatment is not afforded to the objects’ states, as without concrete data from previous versions, it’s impossible to correct

Having no good understanding on “which version change justifies an object reset”, I used a trick relying on the fact that Simitone’s savefile is different than the original Sims one:

def is_old_savefile(blob):
  return blob.simitone_specific_data is Null

def deserialize_savefile(blob, ...):
  if is_old_savefile(blob):
    reset_all_objects(blob)
  ...

This works because:

1. All of my never-touched-before saves are in the original Sims format
2. Every save I produce is in the new Simitone format
3. I will never ever get a new “content” pack that modifies objects’ behavior. I’m only needed to differentiate between “before me” and “after me”

This solution is:

1. Free of both mechanical and logical corruptions
2. Very easy to explain and implement - “when encountering a savefile that didn’t come from our engine, reset all objects”. No heuristics on the ip of every object, no guesswork on whether this specific state is ok.
3. Losing nothing of value gameplay-wise

We now run with 0 exceptions on load, and even better - we eliminated a class of invisible potential bugs.

My takeaway here was that the original tests were only guarding against the visible symptom (the exceptions). This meant that any solution only aimed at “making the tests green” was not guaranteed to solve the real problem.