Broken compatiblity with legacy LVL files in ANSI format

[Flower35]

Hello! I would like to report an issue regarding the loading of legacy episodes made in the "SMBX 1.3" engine.

In short, it is expected that old file formats (".wld", ".lvl") retain the ANSI encoding, while the newer formats (".lvlx", ".lua") use the UTF-8 codepage. The encoding of ".lvl" and ".lvlx" should not be mixed (LVL misintepreted as UTF-8).

---

The handling of WLD files is done via the original Public Sub OpenWorld(FilePath As String) subroutine, thus the file contents are read in ANSI encoding (specific to current Windows CodePage) and translated to UTF16-LE encoding. This potentially affects any referenced LVL and LVLX filenames with diacritics and other Unicode symbols.

LunaLua/LunaDll/Misc/RuntimeHookComponents/RuntimeHookGeneral.cpp

Line 1712 in 29450a6

PATCH(0x8DF5B0).JMP(runtimeHookLoadWorld).NOP_PAD_TO_SIZE<6>().Apply();

LunaLua/LunaDll/Misc/RuntimeHookComponents/RuntimeHookHooks.cpp

Line 2162 in 29450a6

loadWorld_OrigFunc(filename);

This case is partially checked with the GetNonANSICharsFromWStr() function (the SMBX installation path and the WLD file path, but NONE of the world map contents).

LunaLua/LunaDll/GameConfig/GameAutostart.cpp

Lines 63 to 69 in 29450a6

	std::wstring nonAnsiCharsEpisode = GetNonANSICharsFromWStr(wldPath);
	if (!nonAnsiCharsEpisode.empty())
	{
	std::wstring path = L"The episode path has characters which are not compatible with the system default Windows ANSI code page. This is not currently supported. Please rename or move your episode folder.\n\nUnsupported characters: " + nonAnsiCharsEpisode + L"\n\nPath:\n" + wldPath;
	LunaMsgBox::ShowW(0, path.c_str(), L"SMBX does not support episode path", MB_ICONERROR);
	_exit(1);
	}

---

A problem appears with the custom Public Sub OpenLevel (FilePath As String) hook, which was written to handle both the old LVL and the new LVLX formats:

LunaLua/LunaDll/Misc/RuntimeHookComponents/RuntimeHookGeneral.cpp

Line 1702 in 29450a6

PATCH(0x8D8F40).JMP(runtimeHookLoadLevel).NOP_PAD_TO_SIZE<6>().Apply();

LunaLua/LunaDll/Misc/RuntimeHookComponents/RuntimeHookHooks.cpp

Line 1634 in 29450a6

base.ReadFile(static_cast<std::wstring>(*filename), getCurrentLevelData());

LunaLua/LunaDll/FileManager/SMBXFileManager.cpp

Line 119 in 29450a6

if (m_isValid && !FileFormats::OpenLevelFile(utf8_encode(filePath), outData))

---

The "PGE File Library" correctly detects the file format and uses the correct encoding to read level data:
https://github.com/WohlSoft/PGE-File-Library-STL/blob/f2b83a89ce04ad5a40f1249d1c125b53de6f1750/src/file_rwopen.cpp#L85

However, the discrepancy between the ANSI and UTF-8 encoding was not taken into account in the LunaLua_loadLevelFile() function.

LunaLua/LunaDll/FileManager/LoadFile_Level.cpp

Line 55 in 29450a6

void LunaLua_loadLevelFile(LevelData &outData, std::wstring fullPath, bool isValid)

Specifically, the encoding of the following strings could be affected:
https://github.com/WohlSoft/PGE-File-Library-STL/blob/f2b83a89ce04ad5a40f1249d1c125b53de6f1750/src/smbx64/file_rw_lvl.cpp#L179

// "PGE-File-Library-STL/src/smbx64/file_rw_lvl.cpp"

// Internal level name

SMBX64::ReadStr(&FileData.LevelName, line);

// Custom music filepath

SMBX64::ReadStr(&section.music_file, line);

// Various layer names

SMBX64::ReadStr(&blocks.layer, line);
SMBX64::ReadStr(&blocks.event_destroy, line);
SMBX64::ReadStr(&blocks.event_hit, line);
SMBX64::ReadStr(&blocks.event_emptylayer, line);

SMBX64::ReadStr(&bgodata.layer, line);

// NPC message boxes (even though only the ASCII font is supported)

SMBX64::ReadStr(&npcdata.msg, line);

SMBX64::ReadStr(&npcdata.layer, line);
SMBX64::ReadStr(&npcdata.event_activate, line);
SMBX64::ReadStr(&npcdata.event_die, line);
SMBX64::ReadStr(&npcdata.event_talk, line);
SMBX64::ReadStr(&npcdata.event_emptylayer, line);
SMBX64::ReadStr(&npcdata.attach_layer, line);

// The level filename of a warp target (pipe or door)

SMBX64::ReadStr(&doors.lname, line);
SMBX64::ReadStr(&doors.layer, line);

SMBX64::ReadStr(&waters.layer, line);

SMBX64::ReadStr(&layers.name, line);

// Various event names

SMBX64::ReadStr(&events.name, line);
SMBX64::ReadStr(&events.msg, line);

SMBX64::ReadStr(&events_layers.hide, line);
SMBX64::ReadStr(&events_layers.show, line)
SMBX64::ReadStr(&events_layers.toggle, line);

SMBX64::ReadStr(&events.trigger, line);
SMBX64::ReadStr(&events.movelayer, line);

---

When entering a warp to another level (NOT from the world map), this error message appears on the screen:

LunaLua/LunaDll/FileManager/SMBXFileManager.cpp

Lines 95 to 97 in 29450a6

	std::string msg = "Can't load this level, "
	"because the file does not "
	"exist: ";

I solved this issue by manually converting old files to remove any unicode characters from the world map, custom music filenames, and LVL filenames.

However, the LunaLua_loadLevelFile() function could be updated to properly handle cases where StrA2WStr() should be called instead of Str2WStr(), for example:

const LevelDoor& nextDataLevelDoor = outData.doors[i];

if (outData.meta.RecentFormat == LevelData::SMBX64)
{
    // ANSI (legacy format)
    nextDoor->warpToLevelFileName = StrA2WStr(nextDataLevelDoor.lname);
}
else
{
    // UTF-8
    nextDoor->warpToLevelFileName = nextDataLevelDoor.lname;
}

(similar change for all the layer names, event names, and custom music filepaths)

---

[Wohlstand]

The main problem of ANSI levels in genral that they will always fail on non-native computers. For example, file made with German filenames that use diacritics will fail to open at Greek machine. And Chinese filename will fail to open at Russian, etc. That why legacy levels are required to be named with ASCII only characters.

[Flower35]

The main problem of ANSI levels in genral that they will always fail on non-native computers. For example, file made with German filenames that use diacritics will fail to open at Greek machine. And Chinese filename will fail to open at Russian, etc. That why legacy levels are required to be named with ASCII only characters.

Thank you for a quick reply! Yes, I agree. However, when I copy over an old episode from SMBX 1.3 to 2.0, made on the same PC (or made by someone else using the same locale setting), I would still expect it to be playable with zero errors.

The issue is not only with filenames from different codepages (which can be translated to ASCII without much problems using some simple script, outside of the SMBX/PGE), but also making sure any "cross-level" references (on the world map and warps between levels) are also changed in those legacy episodes to ASCII, if possible.

[Wohlstand]

Yea, anyway, I guess, I could try to implement the proper charset converter at the Moondust Maintainer tool that converts episodes between different format. Right now it spits on ANSI and expects ASCII or "local 8bit chatset", but I could add an option to select the source/destination charaet for SMBX64 levels/worlds to properly convert between modern/legacy formats.