STPPC2x v1.0 (Work in Progress Information Only)

It's been a while since I posted any updates here and in that time a lot of important things have happened within the GP2X community.

Most importantly, the GP2X has been discontinued (if you didn't know this by now, it was even on Slashdot's frontpage for a while!). This came as a great shame, but hardly a shock, to myself. However, I will be (and have been) continuing development as normal. I don't have an F-200, I don't have a Pandora (an "unofficial" GP2X successor-in-spirit), I don't have a Wiz (an official succesor). I can't afford any of them anyway. Until that situation changes, I'll be programming for the GP2X F-100 (with F-200 compatibility where I can manage it) and will keep programming. The GP2X rekindled my love of programming and I'll be both playing and coding on it until the day my personal unit dies. In fact, just lately I've been seriously looking at the prospect of buying another cheap F-100 second-hand if one comes up.

If I do suddenly manage to get any of those whizzy new machines, though, then I'll still be programming for the GP2X anyway - I prefer the challenge of programming for the lowest denominator. There is little fun in programming on a machine which just laughs at everything you throw at it (however, *playing* on it is another matter entirely!), and a set of 2D puzzles will never really stretch a powerful machine in practical use. So, I'll probably be leaving any Pandora/Wiz conversion of STPPC2x to someone else (to be honest, people would probably just use the official GTK versions on Pandora anyway, or their new Java ports).

That said, development slowed a little this month and last month but this was for completely unrelated reasons. What with a holiday and some end-of-term work at the school I work at, all programming activities took a back-seat. However, when I did get some programming time, I put aside my GP2X for a moment and attempted to update the Palm port of Simon Tatham's Portable Puzzle Collection, the code for which seems to have stagnated a little. However, this proved frustrating and, in the end, fruitless.

My wife has a Palm TX and we found that although the "official" Palm port of the puzzle collection works fantastically on a Palm Tungsten E2, the TX somehow ran the same executable differently. Most importantly, it often saw right-clicks instead of left-clicks all the time, no matter what options were set. This, obviously, made affected games absolutely unplayable. What was odd was that it only occurred in about 50% of the games and there was no way around it short of actually diving into the code and seeing what happened. Additionally, the Palm port doesn't include several of the games in STPPC2x and it would have been nice to get them working on Palm too, not to mention the various bug-fixes and new options that have been added since.

My attempts to recompile a test version of the Palm port proved much more difficult than I ever imagined. First, finding an appropriate build environment was nothing short of a nightmare - the SDK's are hidden away under obscure names that have changed since the README for the Palm port was written, contained only on websites that require arcane registration. After several hours (and some creative hacking), I was able to get a full suite of headers and compilers working. This is when the second and most damning problem showed up.

For some reason, the Palm has a very, very rigid memory structure and programs are written in segments of code that need to be of a certain size. No problem, the original Palm port took care of that with some ever-so-fancy scripting which tags functions so that they end up in seperate modules. With some creative tagging, all the modules are below the required size and they can all "talk" to each other - with nothing more than adding a function name to a script's datafile. However, because of the problems of getting the exact build environment that the original porter used, a straight compile no longer generates the same size code. Thus, a lot of the programs now need to be split into not just two but three or more segments, which greatly increases the complexity of compiling, linking and the various supporting scripts. In short, it's an absolute nightmare.

What started off as "I'll just compile this and see if I can find out what went wrong for Palm TX" turned out to be hours of frustrated downloading, hacking, patching, compiling of compilers followed by more hours of changing headers and filenames, editing scripts and lots of other fannying about just to get the original source code to compile once more in a useable fashion. After deciding to just speed things along and get the simple things running and work on the complex stuff later, I did get a working executable in the end, but without 40% of the games present in it because of the various code size changes and other problems. In short, it really wasn't worth the effort and was certainly useless for anything but a developer.

The frustration was so intense having fought to get to that point that I'd completely forgotten the original point of doing it all and in the end, never even got to looking at the relevant lines of code or whether a simple updated header could fix the problem. I would have spent 5% of the time fixing my problem and the rest just trying to get the original games back into a working state, let alone what would happen once I started introducing the new updates, games, etc. into the Palm collection. In the end, I left it for another day (month, year) and just kept hold of what I had downloaded/patched to get things working that much.

That meant that I was pretty sick of the Puzzle code for a while, although it was hardly the Puzzles' code that was at fault as much as the design of the Palm and/or the gcc compilers/linkers for Palm. Why all that can't be done automatically is beyond me but I'm sure there's a good reason, even if it's just "nobody got around to doing that". I tried other compilers and I tried searching for help but the answer was always the same "There's nothing we can do, split your code into smaller modules".

So, getting back to the GP2X (once I'd mustered some programming enthusiasm again) was an absolutely joy. Compile for x86. Link. Run on laptop. Compile for ARM. Link. Transfer to GP2X. Run. Rinse and repeat. Programming for the GP2X is a cinch in comparison and, more importantly, is intuitive and standardised. If something goes wrong, it's obvious and never anything to do with the compiler/linker being used. I ballsed up a Makefile once, so that things compiled oddly for ARM but okay for x86, but that's hardly anybody's fault but my own.

Unfortunately, my TODO list for the next (and probably "definitive") version of STPPC2x, with which I'm going to tempt fate and call v1.0, is very long but I am working steadily through it, even if I'm not putting out BETA's every month like I used to. A lot of the new stuff involves major infrastructure changes. The dodgy menu code that was thrown in with the intention of having some way to restart a game, and then changed to an actual menu to have some way to pause the game, and then changed to have some way to display the current options (in some of the most horrible code you'll ever see), and then repeatedly altered, bug-fixed and torn apart to fix some interface issues - well, let's just say that it's not going to be practical to extend it again and add the new options that I want to (such as using difficulty presets, etc.).

I've had a design for a new menu for a while and it'll look and work much better than the current one. We're talking a proper menu on the Pause button. With actual sub-menus. And actual proper changeable options. And an actual list of save-games. And an actual list of presets (including custom ones) that can be selected. And all without having to remember fifty keys or cramming it all into one screen.

It does involve touching some code that even I'm not sure I can remember how it works (I made a point of documenting the configuration-option stuff very heavily at the time, because I knew it would become a problem) so it's a big change to make that'll impact on a lot of things. However, all the "big stuff" is already in the code and it's just a matter of making it work in an intuitive way, in the right order, without bugs.

I need to factor out a lot of things to make the changes to the menu and thus there probably won't be many "preview" versions of the new code until it's finished: e.g. at the moment, you are "in the menu" or not, and the helpfile display "sub-menu" is a massive bodge. I need to change that so that you are "in menu number X" etc. so that the game only plays when you're not in a menu and options on sub-menus inherit from their parents etc. correctly and the key-press changes differ depending on the menu you're in etc.

It's a big job but I'm aiming for the magic 1.0 and I'm hoping to get it all working so that the program has that "professional" air to it, rather than the rather hackish backend hidden behind a nice PNG and some bash scripts that the user gets at the moment.

As a first step, I have integrated the "select a game" menu supplied by juanvvc into the main code. This is now just seen as "another type of menu" by the game, so it works much more closely and was a good way to see what sort of functions I need to have. This meant almost a complete re-write of that code but it now uses all the in-game functions to do its job rather than being a tagged-on executable that runs a script, that runs an executable to actually play the game. You won't notice, except that there are some tiny cosmetic changes such as the menu now showing the version number and being "part of" the main program.

Now one program is distributed and run, and the switch in and out of games is nigh-on instantaneous (However, it was pretty damn impressive before!). The Palm port was always the inspiration for this - it uses a so-called "Combined" executable (as does the MacOS port) that has all the games in it. That's now how the GP2X one works, too (after a massive migration from single-game-per-executable to monolithic-executable-that-can-only-run-one-game-at-a-time to the current "proper" combined monolithic executable). Additionally, it means that the "join" between the menu and the games is seamless - so, for instance, a music track could play across both without needing a jolt in between.

It does add a lot of complexities, though. For instance, it does mean that I have to check carefully for memory bugs because they will now accumulate if a player plays a long session of several games without quitting out of the thing entirely. I've already caught quite a few bugs that wouldn't have bitten people on the old menu but bite the second you change in and out of different the games themselves. There are also games which require a resolution change mid-flight if you run them - no matter how you play with them, some games run better in 320x240 and some in 640x480, so the menu has to know about them and switch between them. Fortunately, that's already done and dusted (although for some reason my PC has problems doing that same thing - I must update my SDL libraries!).

All these changes mean, however, that a lot of extraneous files and code can disappear and also that a lot of in-game facilities can make it back out to the menu. I'm hoping to eventually ship just one executable and the associated PNG's, helpfile TXT's and a single menu datafile. If I can manage it, I'd like to put everything but the executable into a single file (probably a ZIP or something) purely so that I can ship the minimum amount of files necessary.

The other thing that I'm working on as and when my mood changes is to have game "persistence". This was one of the things that working on the Palm port brought home to me - if you quit out of the game, you really should be able to go straight back to where you were just by running the program again. It's a bit like an "auto-save" whenever you leave one game and start another, or quit the game entirely.

It'll only work if you specifically leave a game, though, because otherwise it would involve writing to the SD card all the time which is something I want to avoid. Avoiding writes also means doing things like checking whether you've actually done anything to the game or whether you just hit the wrong game on the menu and then quit out straight away. If you didn't press a button, then there's no need to auto-save, which would save one "write" from hitting the disk. It also means that I have to be careful - what if that auto-save is corrupted, or what if it takes hours to load/redraw because it's a massive game? There has to be a facility for the user to ignore it. That means more menu options... All this adds extra code, simple though it may be, which needs careful positioning, checking and re-writing, so it all adds up. Like any big change, it'll be a option for people to turn off - for instance if they want to play on a write-protected card.

I also have some "bigger" items on the TODO list. One of them is to be able to stop a game halfway through generation. Unfortunately, this is quite a tricky one because the games are inherently designed to be single-threaded and non-interruptible while in generation (all the official ports suffer from this and the Palm port suffers especially badly). However, there are a few tweaks to the code that can be made to make them interruptible (even if it takes a few seconds for the interrupt to be "seen" by the program). It means adding code to each and every game, unfortunately, but I think it's a sensible thing to do because hitting a large Bridges game by accident could leave the user with no option but to kill the whole GP2X (or at very least telnet in and kill the process). That's an ugly way to play a game. And now that we have a nice integrated menu to fall back to, there's no reason it can't be done in a nice way.

I'm also pretty certain that the games just aren't drawing themselves properly, even though it's damn close. Other ports show clean lines on all resolutions but the GP2X has some overlap issues. This has been there since the start but I was always of the "it's playable" mindset before so I left it well alone. Unfortunately, without single-stepping the code or sitting down with some graph paper and manually following along, it's rather difficult to see where the problem lays. I'm guessing that there is an off-by-one error somewhere but there are so many "definitions" of the various graphics routines involved that it's hard to track down - does setting a clip rectangle of 10x10 at 5,8 mean that it extends from 5,8 to 15,18 or from 6,9 to 15,18 or from 5,8 to 14,17 or what? It looks likely that I misread something and made a small error between the translation of the puzzles concept of "clipping", "blitting", or "drawing" and SDL's concept of the same. This would fix such things as Solo's missing lines.

One of the other things I have added is massive amounts of debugging code, turned on by compiler options, which should be able to resolve such discrepancies once and for all. It's just a matter of running a debug-compiled version on the PC, capturing the megabytes of debug logs and then manually replicating them on graph paper to see what goes wrong and where.

Then, I have a few extra bits of "polish". I'm considering a musical background noise. I wouldn't normally bother but it's v1.0 and it's a very simple thing to implement. The games seem to have more than enough spare capacity to play music too (the Palm port is noticeably slower on similar-MHz models) and I've found a couple of relaxing background tunes on a Creative Commons site, so when I get bored of the dreary work, I'll plug in some code for that. I've also considered short sound effects for such things as entering a game, pausing it, pressing a menu, clicking within the game etc. None of these things will affect anything if the player doesn't want them - they will be easily turned on and off and may even be off by default.

There is also supposed to be a way to "write down" the definition of a puzzle, so that you can pass it to friends. This is implemented as "Copy" in the desktop versions of these games but I consider it a bit pointless in the modern age. If the GP2X could do Bluetooth or something easily then I would use it as a basis for a "Beam game" option similar to that present on many Palm games. As it is, I might implement it as a menu option purely because it's so simple to do (call one function, get the "answer" as plain text, display it on the screen) but I don't see anybody using it. With the new menus, it can be put there without getting in the way.

The puzzles also have "print" options, but I'm deciding to leave those out entirely because I very much doubt anybody would bother to print them off, and adding the whole Postscript-creation routines into the puzzles is a bit overkill for a function nobody will use and nobody can use easily anyway. You would have to "print" the puzzle to a PS, or possibly PDF, file on the SD card, take it to a desktop machine (which has its own port of STPPC which understands the same savefile format and can print direct) and then send it to a printer. Because it's achievable in an identical amount of time, with an identical process, just by moving a savefile over to the PC version, I'm leaving it out.

I'm looking at downclocking the GP2X when it's not being taxed (i.e. most of the time it's actually in-game or in the menus, but NOT during puzzle creation when the poor thing struggles along) but I've still to experiment and test this thoroughly (when testing, you want to get into the parts you know are buggy as quickly as possible!). There would be a problem if it decided to throttle itself back on a certain game which would actually benefit from going faster and so just frustrate the user for no reason. I don't know if and how it would affect battery life but I hope that running at a slower clock speed is actually better for power consumption that just idling at a higher clock speed. There is a lot of take account of, though, because the cursor-motion and input is tied to the speed of certain timers and it would be very bad if they started getting lagged or overwhelmed because of the number of events happening while it was running slower.

There are lots of other things that I want to do but they are either trivial, cosmetic, only affect the source code, or they really aren't worth mentioning. I don't know when v1.0 will be finished but it will be finished sometime - every time I find myself in a programming mood, I'm on the GP2X straight away. Every time I play the GP2X, I suddenly find myself hacking on the code.

All in all, STPPC2x has been great fun so far and it's only recently that I've actually figured out how to win all the games in it! I still don't succeed every time, and Tents and Blackbox are often the most surprising ones - you get to the end by applying what you think is perfect logic and then you just cannot fathom the answer. Sometimes you can get it by restarting that same game but sometimes you just have to reach for that Solve option and then you start kicking yourself!

I do want to get out a v1.0, something to say "this is a real port of the games", I want it to be polished, I want it to be complete and I want it to stick around with the GP2X. Probably nobody else will continue the port to other platforms based on this particular code (although a couple of people have borrowed the early SDL code that got the games to a playable state for things like PlayStation Portable ports etc.) because there's such an abundance of power in modern consoles to just run the original versions or simple modifications - there are even Java ports of the collection now. But to have hold of a classic games console, quite rare, quite unique, very open, and to have a copy of a polished port for it that I know I wrote - that's something I want to be able to do. I don't care if nobody ever downloads it or nobody else owns the hardware to run it, but I'd like to be able to show it to people and say "I did that".

In twenty years, I want to find my GP2X in a dusty old box somewhere and spot an SD card with "STPPC2x v1.0". I want to plug it in and show any kids I have: This is what I did. This is how we used to make computers work. This is the code and tools I used to do it. This is what a lot of companies "these days" try to make you think is done in a magic box at Microsoft. Look, despite it being 20 years ago, I can still load this stuff up, I can still know how it works, I can still fix it when it goes wrong. Let's have a quick game of it before I put it back.