This is a small library I wrote to handle UTF-8.
Usage is meant to be as simple as possible - see for example decoding
a UTF-8 string:
const char* str = "asdf";
uint32_t codepoint;
while ((codepoint = UTF8_next(&str)))
{
// you have a codepoint congrats
}
Or encoding a single codepoint to add it to a string:
std::string result;
result.append(UTF8_encode(0x1234).bytes);
There are some other functions (UTF8_total_codepoints() to get the
total number of codepoints in a string, UTF8_backspace() to get the
length of a string after backspacing one character, and
UTF8_peek_next() as a slightly less fancy version of UTF8_next()), but
more functions could always be added if we need them.
This will allow us to replace utfcpp (utf8::unchecked) and also fix
some less-than-ideal code:
- Some places have to resort to ignoring UTF-8 (next_wrap) or using
UCS-4→UTF-8 functions (VFormat had to use PHYSFS ones, and one other
place has four lines of code including a std::back_inserter just for
one character)
- The iterator stuff is kinda confusing and verbose anyway
This is still a work in progress, but the existing font system has been
removed and replaced by a new one, in Font.cpp.
Design goals of the new font system include supporting colored button
glyphs, different fonts for different languages, and larger fonts than
8x8 for Chinese, Japanese and Korean, while being able to support their
30000+ characters without hiccups, slowdowns or high memory usage. And
to have more flexibility with fonts in general. Plus, Graphics.cpp was
long enough as-is, so it's good to have a dedicated file for font
storage.
The old font system worked with a std::vector<SDL_Surface*> to store
8x8 surfaces for each character, and a std::map<int,int> to store
mappings between codepoints and vector indexes.
The new system has a per-font collection of pages for every block of
0x1000 (4096) codepoints, that may be allocated as needed. A glyph on
a page contains the index of the glyph in the image (giving its
coordinates), the advance (how much the cursor should advance, so the
width of that glyph) and some flags which would be at least whether the
glyph exists and whether it is colored.
Most of the *new* features aren't implemented yet; it's currently
hardcoded to the regular 8x8 font.png, but it should be functionally
equivalent to the previous behavior. The only thing that doesn't really
work yet is level-specific font.png, but that'll be supported again
soon enough.
This commit also adds fontmeta (xml) support.
Since the fonts folder is mounted at graphics/, there are two main
options for recognizing non-font.png fonts: the font files have to be
prefixed with font (or font_) or some special file extension is
involved to signal what files are fonts. I always had a font.xml in
mind (so font_cn.xml, font_ja.xml, etc) but if there's ever gonna be
a need for further xml files inside the graphics folder, we have a
problem. So I named them .fontmeta instead.
A .fontmeta file looks somewhat like this:
<?xml version="1.0" encoding="UTF-8"?>
<font_metadata>
<width>12</width>
<height>12</height>
<white_teeth>1</white_teeth>
<chars>
<range start="0x20" end="0x7E"/>
<range start="0x80" end="0x80"/>
<range start="0xA0" end="0xDF"/>
<range start="0x250" end="0x2A8"/>
<range start="0x2AD" end="0x2AD"/>
<range start="0x2C7" end="0x2C7"/>
<range start="0x2C9" end="0x2CB"/>
...
</chars>
<special>
<range start="0x00" end="0x1F" advance="6"/>
<range start="0x61" end="0x66" color="1"/>
<range start="0x63" end="0x63" color="0"/>
</special>
</font_metadata>
The <chars> tag can be used to specify characters instead of in a .txt.
The original idea was to just always use the existing .txt system for
specifying the font charset, and only use the XML for the other stuff
that the .txt doesn't cover. However, it's probably better to keep it
simple if possible - having to only have a .png and a .fontmeta seems
simpler than having the data spread out over three files. And a major
advantage: Chinese fonts can have about 30000 characters! It's more
efficient to be able to have a tag saying "now there's 20902 characters
starting at U+4E00" than to include them all in a text file and having
to UTF-8 decode every single one of them.
If a font.txt exists, it takes priority over the <chars> tag, and in
that case, there's no reason to include the <chars> tag in the XML.
But font.txt has to be in the same directory as font.png, otherwise it
is rejected. Same for font.fontmeta. If neither font.txt nor <chars>
exist, then the font is seen as a 2.2-and-below-style ASCII font.
In <special>: advance is the number of pixels the cursor advances after
drawing the character (so the width of the character, without affecting
the grid in the source image), color is whether the character should
have its original colors retained when printed (for button glyphs).
As for <white_teeth>:
The renderer PR has replaced draw-time whitening of sprites/etc
(using BlitSurfaceColoured) by load-time whitening of entire images
(using LoadImage with TEX_WHITE as an argument).
This means we have a problem: fonts have always had their glyphs
whitened at printing time, and since I'm adding support for colored
button glyphs, I changed it so glyphs would sometimes not be whitened.
But if we can't whiten at print time, then we'd need to whiten at load
time, and if we whiten the entire font, any colored glyphs will get
destroyed too. If you whiten the image selectively, well, we need more
code to target specific squares in the image, and it's kind of a waste
when you need to whiten 30000 12x12 Chinese characters when you're only
going to need a handful, but you don't know which ones.
The solution: Whitening fonts is useless if all the non-colored glyphs
are already white, so we don't need to do it anyway! However, any
existing fonts that have non-white glyphs (and I know of at least one
level like that) will still need to be whitened. So there is now a
font property <white_teeth> that can be specified in the fontmeta,
which indicates that the font is already pre-whitened. If not
specified, traditional whitening behavior will be used, and the font
cannot use colored glyphs.
We get these warnings because of the typedefs for 64-bit integers in
PhysFS's header files. The compiler will treat them as extensions and
will still compile it fine but it does mean we aren't strictly standards
conforming. Which really isn't a problem anyway. Probably.
This commit adds most of the code changes necessary for making the game
translatable, but does not yet "unhardcode" nearly all of the strings
(except in a few cases where it was hard to separate added
loc::gettexts from foundational code changes, or all the localization-
related menus which were also added by this commit.)
This commit is part of rewritten history of the localization branch.
The original (unsquashed) commit history can be found here:
https://github.com/Dav999-v/VVVVVV/tree/localization-orig
A relevant paragraph copied from the original commit history:
The idea is that we store all strings somewhere managed, and then the
hashmap only needs pointers to those strings. For storing strings, I
created a `Textbook` structure, which consists of one or more 50 KB
"pages" (allocated as needed) on which you can simply write strings in
both languages back-to-back with `textbook_store(textbook, text)` and
get pointers to each of them. (I was originally going to just use one
big buffer and realloc to double the size when filled up, but then the
hashmap would be full of dangling pointers...) When needed, like when
switching to a different language, an entire textbook can be freed at
once.
This commit is part of rewritten history of the localization branch.
The original (unsquashed) commit history can be found here:
https://github.com/Dav999-v/VVVVVV/tree/localization-orig
This reworks how the commit hash and date are compiled so that if
they're changed (and they're changed often), only one source file needs
to be recompiled in order to update it everywhere in the game, no matter
how many source files use the hash or date.
The commit hash and date are now declared in InterimVersion.h (and they
need `extern "C"` guards because otherwise it results in a link fail on
MSVC because MSVC is stupid).
To do this, what now happens is that upon every rebuild,
InterimVersion.in.c is processed to create InterimVersion.out.c, then
InterimVersion.out.c is compiled into its own static library that is
then linked with VVVVVV.
(Why didn't I just simply add it to the list of VVVVVV source files?
Well, doing it _now_ does nothing because at that point the horse is
already out of the barn, and the VVVVVV executable has already been
declared, so I can't modify its sources. And I can't do it before
either, because we depend on the VVVVVV executable existing to do the
interim version logic. I could probably work around this by cleverly
moving around lines, but that'd separate related logic from each other.)
And yes, the naming convention has changed. Not only did I rename
Version to InterimVersion (to clearly differentiate it from
ReleaseVersion, which I'll be adding later), I also named the files
InterimVersion.in.c and InterimVersion.out.c instead of
InterimVersion.c.in and InterimVersion.c.out. I needed to put the file
extension on the end because otherwise CMake wouldn't recognize what
kind of language it is, and I mean like yeah duh of course it doesn't,
my text editor doesn't recognize it either.
I thought all of these were removed earlier but apparently not. Anyways,
add_definitions is bad because it pollutes the definitions of every
single target, we should be using target_compile_definitions instead.
This option is enabled by default and will replace absolute paths of all
source directory file paths with relative paths in the compiled binary,
if the compiler supports it. Of course, this isn't needed if you compile
with all paths removed anyways (e.g. in Release mode).
The purpose is to help make builds reproducible and to remove any
potentially sensitive information about the user or the user's system
from the compiled binary.
Both Clang and GCC support -fdebug-prefix-map, -fmacro-prefix-map, and
-ffile-prefix-map. In particular, -ffile-prefix-map is just a flag that
does both -fdebug-prefix-map and -fmacro-prefix-map.
According to https://reproducible-builds.org/docs/build-path/ ,
-fdebug-prefix-map is available in all GCC versions but only available
starting from Clang 3.8, and -fmacro-prefix-map and -ffile-prefix-map
are available since GCC 8 and Clang 10. So we check the compiler version
and use the available flags depending on if the compiler supports it or
not.
This does make debugging a bit more annoying, but there are a couple
ways to rectify this. Either disable it with
-DREMOVE_ABSOLUTE_PATHS=OFF, or add a `.gdbinit` that consists of
set substitute-path . ../..
so that `.` is considered to be `../..`. Of course, if you need to,
replace `../..` with the actual source directory path (in my case it's
`../../..` because I place my build folders in another subdirectory to
have multiple build folders in one directory).
This doesn't need to be a global `.gdbinit`, it can be in a
directory-specific `.gdbinit` (similar to how `.gitignore`s can also be
directory-specific). But then you need to add `add-auto-load-safe-path`
to your `.gdbinit` to load any directory-specific `.gdbinit`s.
The above is for GDB; I don't know what (if anything) needs to be done
for LLDB; I don't use LLDB.
Fixes#889.
Whereas all `SDL_assert`s will go away when compiling with optimization
flags and all plain `assert` calls (used in PhysFS) will go away when
compiling in Release mode, FAudio has a bunch of debug stuff that needs
to be explicitly disabled with its own `FAUDIO`-prefixed flag.
To do this in Release mode, we need to use generator expressions for
dumb CMake reasons. Basically, if checking the CMAKE_BUILD_TYPE variable
will not work for certain generators (Ninja, Visual Studio) because they
only specify the build type at build time, not generation/configuration
time.
This is so flags that apply globally (i.e. to the game and all static
libraries it's compiled with), such as /MT on MSVC, can be put in a
list, and along with putting all static libraries in a list, we remove
the need for each flag to be repeated for each static library and we can
just use a foreach loop instead.
(Global compile flags of course don't apply to us meddling with
CMAKE_C_FLAGS and CMAKE_CXX_FLAGS directly, because we need to do that
in order to make sure the C and C++ standards are set properly.)
This flag makes it so the MSVC runtime libraries are statically linked.
This avoids needing Windows users to have these libraries installed.
Apparently /MT stands for "MultiThreaded", and there's a bit of a
history there where originally by default you could only have a
single-threaded library, and then the multi-threaded flags were added in
later.
First I tried doing target_compile_options on VVVVVV, but then got a
linker error. Then I tried doing add_compile_options because I figured
/MT had to be applied everywhere, and it seemed to work, but it still
linked to the runtime libraries. Apparently it was being overridden.
Then I tried target_compile_options again but this time did it to
everything, and that linked correctly and also removed the runtime
dependency. I would've tried using the MSVC_RUNTIME_LIBRARY property
- along with the CMP0091 policy - but those were only introduced in
CMake 3.15.
You can verify that a binary is built without dependencies by installing
LLVM and running llvm-readobj --needed-libs path/to/binary. This is the
output for a binary with runtime dependencies:
infoteddy@fedorarune ~/d llvm-readobj --needed-libs VVVVVV.exe
File: VVVVVV.exe
Format: COFF-i386
Arch: i386
AddressSize: 32bit
NeededLibraries [
ADVAPI32.dll
KERNEL32.dll
MSVCP140.dll
SDL2.dll
SHELL32.dll
USER32.dll
VCRUNTIME140.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-utility-l1-1-0.dll
]
And this is the output for a binary with those dependencies having been
statically-linked in:
infoteddy@fedorarune ~/d llvm-readobj --needed-libs VVVVVV.exe
File: VVVVVV.exe
Format: COFF-i386
Arch: i386
AddressSize: 32bit
NeededLibraries [
ADVAPI32.dll
KERNEL32.dll
SDL2.dll
SHELL32.dll
USER32.dll
]
This commit adds a new string formatting system to replace uses of
`SDL_snprintf` and string concatenation.
Making our own string formatting system has been briefly discussed
during the review of the localization branch, and on the VVVVVV
Discord. It's inspired by Python's format strings, but simpler.
This is primarily to benefit localization - strings will be easier to
understand (`Now using %s Tileset` → `Now using {area} Tileset`,
`"%s remain"` → `"{n_crewmates|wordy} remain"`), translators can change
the word order for their language's grammar (`%1$s` is a POSIX
extension), and this system is also less error-prone (making the format
string not align with the actual arguments won't result in a crash or
UB).
It also integrates our needs better - particularly the "wordy" numbers
without having to have a `help.number_words(n).c_str()` at the
callsite, translators can opt in and out of wordy numbers per string,
and this should also make it easier to solve #859.
This commit adds the formatting system itself, and changes one
`SDL_snprintf` in the code to use it as a small demo (the rest should
probably be done in the localization branch to avoid more unneeded
work).
The system is described in full detail in VFormat.h and in the pull
request description.
So, it turns out we weren't quite done fighting CMake yet...
To accommodate #869 (and actually also #272), the C standard was raised
from C90 to C99. This turned out to require a bit of a fight with the
CentOS CI's CMake version to get it to set the flags we wanted (and to
not overwrite them later). Eventually the fix was to move the block
that sets the standards to later in the file, which was done in
24353a54bb.
As it apparently turns out, if your CMake is at least 3.1.3 and
`CMAKE_<LANG>_STANDARD` is used instead of the workaround, the standard
setting now has an effect on the third party libraries, but not on
VVVVVV itself. The cause is (probably) the phrase "if it is set when a
target is created" in the CMake documentation - the
`CMAKE_<LANG>_STANDARD` values have to come before the VVVVVV target is
defined. In other words, the compiler's default C/C++ standard will be
used, probably something like C17 and C++17. As I can confirm with
`__cplusplus` and `__STDC_VERSION__` with my recent-enough CMake. If I
force the pre-3.1.3 workaround to be used, everything is compiled with
C99/C++98 as expected; and the `-fno-exceptions` `-fno-rtti` flags
appear everywhere regardless of version.
So my fix is to make the CMakeLists a little less complex by
simplifying away the `CMAKE_<LANG>_STANDARD` and
`CMAKE_<LANG>_EXTENSIONS`, and always using the workaround regardless
of CMake version. There's nothing wrong with the workaround, the same
thing is also done for `-fno-exceptions` `-fno-rtti`, and it's good to
have a less complicated CMakeLists that doesn't do different and
unexpected things for different versions.
The previous commit f6d7a214f8 ended up
breaking CI because the workaround ended up breaking the PhysFS build
too, which was previously relying on extensions to compile.
Since #869 is going to require C99 anyways, I might as well just up the
standard now. That way the PR won't have to fight it too.
Previously, if the user had a CMake version below 3.1.3, we told them to
set `-std` themselves.
However, we are going to go to C99 soon (because of FAudio, see #869),
and CentOS 7's CMake is too old to set `-std=` automatically, defaulting
to C90. This is bad because it fails the CI.
To work around this, we set `-std=` ourselves, but first we have to
clear any existing `-std=` flag in C_FLAGS or CXX_FLAGS. Amusingly
enough, MSVC does not have `/std:` switches for either C90 or C++98, so
we just get to do nothing.
This means we are no longer copy-pasting PhysFS source files directly.
Since the source files reside in a src/ subdirectory, the paths in the
CMakeLists.txt have to be adjusted.
We are no longer copy-pasting LodePNG source files directly.
As we can't rename lodepng.cpp to lodepng.c in the submodule itself, we
need to make a wrapper file, lodepng_wrapper.c, that #includes
lodepng.cpp, but gets compiled as C.
This moves editorrenderfixed(), editorrender(), editorinput(),
editorlogic(), and their associated functions to a new file named
Editor.cpp - which is exactly what it says on the tin; it stores all the
functions related to the actual in-game editor loop. Also, the existing
editor.cpp has been renamed to CustomLevels.cpp.
The RWops stuff isn't a part of any standard PhysFS package (and given
that it explicitly wraps around SDL I'm not sure how you _would_ package
it). So we need to get the physfsrwops.h include in if
BUNDLE_DEPENDENCIES is off, otherwise this results in a compile-time
include-not-found failure.
Additionally, I've placed the PhysFS RWops stuff in their own extras/
folder, so none of the other PhysFS stuff gets included in a
-DBUNDLE_DEPENDENCIES=OFF build.
This is to make it so RNG is deterministic when played back with the
same inputs in a libTAS movie even if screen effects or backgrounds are
disabled.
That way, Gravitron RNG is on its own system (seeded in hardreset()),
separate from the constant fRandom() calls that go to visual systems and
don't do anything of actual consequence.
The seed is based off of SDL_GetTicks(), so RTA runners don't get the
same Gravitron RNG every time. This also paves the way for a future
in-built input-based recording system, which now only has to save the
seed for a given recording in order for it to play back
deterministically.
Previously, turning glitchrunner mode on essentially locked you to
emulating 2.0, and turning it off just meant normal 2.3 behavior. But
what if you wanted 2.2 behavior instead? Well, that's what I had to ask
when a TAS of mine would desync in 2.3 because of the two-frame delay
fix (glitchrunner off), but would also desync because of 2.0 warp lines
(glitchrunner on).
What I've done is made it so there are three states to glitchrunner mode
now: 2.0 (previously just the "on" state), 2.2 (previously a state you
couldn't use), and "off". Furthermore, I made it an enum, so in case
future versions of the game patch out more glitches, we can add them to
the enum (and the only other thing we have to update is a lookup table
in GlitchrunnerMode.c). Also, 2.2 glitches exist in 2.0, so you'll want
to use GlitchrunnerMode_less_than_or_equal() to check glitchrunner
version.
These are two C++ features that we don't need, don't use, and will never
use in the future. Apparently the best way of doing this in CMake is to
fiddle with the CXX_FLAGS using regex.
Now this is one less flag I need to supply myself when I invoke CMake...
This variable is not defined anywhere and never has been since the
source code release (which is when this CMake file was first created).
To make things clearer, I'm cleaning this variable up.
A function like add_definitions() adds definitions to ALL targets, not
just VVVVVV. This kind of namespace pollution is messy, and could result
in bugs if you pollute with the right kind of pollutant.
So instead of using add_definitions(), use target_compile_definitions().
And instead of using include_directories(), use
target_include_directories().
All the C third-party dependencies are C90, and all the C files we have
are also C90 (well, almost, but that's easily sorted). So we have
basically no reason to not go with C90 here.
The only wrinkle is, turning C extensions off for physfs-static results
in linker errors because PhysFS implicitly uses alloca() without
including it properly (on Linux). I am not the only one who has ran into
this - see https://icculus.org/pipermail/physfs/2020-April/001293.html -
and it's a bug with PhysFS. The workaround I've gone with is to enable C
extensions. (There might also be some funkiness with PhysFS's use of the
`inline` keyword, so enabling extensions will paper over that as well.)
It seems like CMake 3.1.3 introduced the C/C++ standard properties,
while the minimum version of this CMake file is 2.8.12. So we do what
FAudio does, which is print a warning if the CMake version is too old
and otherwise use it if we have the feature.
They're the same thing, but using option() better conveys intent.
However this can't be done for anything that isn't a bool, which the
CUSTOM_LEVEL_SUPPORT option is not (it's a tri-state string).
These were introduced in 098fb77611 - did
Leo not know that they were already there at the top of the file? This
does the same thing, except it only sets it for VVVVVV instead of
everything (so this wouldn't set it for the third-party dependencies).
Sometimes, there needs to be code that gets ran at the end of the game
loop, otherwise rendering issues might occur. Currently, we do this by
special-casing each deferred routine (e.g. shouldreturntoeditor), but it
would be better if we could generalize this deference instead.
Deferred callbacks can be added using the DEFER_CALLBACK macro. It takes
in one argument, which is the name of a function, and that function must
be a void function that takes in no arguments. Also, due to annoying C++
quirks, void functions taking no arguments cannot be attributes of
objects (because they have an implicit `this` parameter), so it's
recommended to create each callback separately before using the
DEFER_CALLBACK macro.
If you configure the build with -DBUNDLE_DEPENDENCIES=OFF, then VVVVVV
will dynamically link with TinyXML-2 and PhysicsFS instead of using the
bundled source code in third_party/ and statically linking with them.
Unfortunately, it doesn't seem like distros package LodePNG, and LodePNG
isn't intended to be packaged, so we can't dynamically link with it, nor
can we use some system LodePNG header files somewhere else because those
don't exist.
UTF8-CPP is a special case, because no matter what, it's going to be
statically linked with the binary (it doesn't come as a shared object
file in any way). So with -DBUNDLE_DEPENDENCIES=OFF, we will use the
system UTF8-CPP header files instead of the bundled ones, but it will
still be statically linked in with the binary.
The main motivation for doing this is so if VVVVVV ever gets packaged in
distros, distro maintainers would be more likely to accept it if there
was an option to compile the game without bundled dependencies. Also, it
discourages modifying the third-party dependencies we have, because it's
always possible for someone to compile those dependencies without our
changes, with this CMake option.
Dav999-v
Misa
N00byKing
Ethan Lee
leo60228