Ewww, MIPS! (Spectrum emulator)

xpectrum for the SPMP8000

A week or so ago I have ported xpectrum (also known as gp2xpectrum, iXpectrum, Xpectroid, and Aunt Irma ^{[citation needed]}) to the SPMP8000 platform. There is no support for ZIP files (if built with zlib all files, even non-compressed ones, fail to load), and the scaling in full-screen mode is done in software, and not very well. Other than that, it works quite nicely, sound and virtual keyboard included. You can download it here, source code is at the usual place.

The A3300 is... different.

I actually wanted to fix the two issues with xpectrum before announcing it here, but I didn't find the time because I've been had. My JXD A3300 has arrived at about the same time I finished the xpectrum port, and when I tried to run the emulator on it, it didn't work! Apparently, the A3300 is not an SPMP8000 device at all, but based on some obscure MIPS controller by "Actions Semiconductor". A little online research indicated that nobody has done any homebrewing on this platform yet. Bummer.

Not so fast!

I was unwilling to write off this purchase, though, so I went ahead and extracted the data from the encrypted firmware update image for the A3300. (Load it into the firmware update tool, attach OllyDbg, find the largest data segment, dump it, remove some superfluous bytes, and you've got yourself an SQLite3 database file.)
As it turned out, this system is a very, very different beast, but in many ways every bit as crappy as the good old SPMP8000/eCos platform.

The Actions Semiconductor OS

The Actions Semi OS appears to be based on uC/OS-II. It features a dynamic loader and virtual memory, but nonetheless almost all interfaces are implemented using system calls, even the GUI and libraries like SQLite. There even is a memcpy() syscall, although most applications have the good sense not to use it. On top of that they put something that seems to borrow many conventions and architectural features from Unix/Linux, such as devices as files, the POSIX threads API, or file name endings (".so" for dynamically loaded object and ".ko" for kernel modules), but it is clearly not derived from Linux. It's probably a custom job, either grown in-house at Actions Semi, or contracted from a third party. The latter seems more likely to me because the strings found in the binary files are almost entirely void of Chinglish, very much unlike the SPMP8000 OS.

Running our own code

From what I could tell there is no mechanism similar to the "native game" loading on the SPMP8000 that would allow you to run third-party native applications. There is, however, an update mechanism that checks if there are new emulators on the externally accessible flash partition. To check for updates, the OS tries to load certain files (libp1.so, libg1.so, etc.) with dlopen() and inspects an exported symbol to get a version number. If the number is higher than the one of the installed emulator (or there is no emulator of that type installed yet), it will copy the new file over the existing one on an internal flash partition and use it as an emulator for the corresponding types of files from now on.
The neat thing here is that dlopen() automatically runs the code in the .init section of the binary, so I wrote a little program ("installer") that resides entirely in the .init section, and if you put a file called new_libp1.so on the external flash, it will copy that file over the internal PS1 emulator, before any version numbers are being checked. Now all we need is a MIPS ELF binary that conforms to a few simple standards, and we can run your own code on this "closed" platform by choosing any file that is configured to be run by the PS1 emulator in the file browser.

Getting comfortable

Of course this is not very convenient: You don't want to copy files around and run the installer every time you want to use a different program. So I wrote another program ("launcher") tailored to the Actions Semi emulator interface that pretends to be an emulator up until the point where the emulator framework tells it what "ROM" it is supposed to play. At that point it opens that file and checks if it is an ELF binary. If it is, it dlopen()s the file and relays all information it has previously received from the emulator framework. At this point, the new binary has hooked itself into the emulator framework and takes over. If the file to be played is not an ELF binary, the same thing happens, with the exception that it's the PlayStation emulator that is loaded. Now we can simply copy our homebrew code to the console and run it from the file browser, provided we give it a PlayStation-ey file extension; .bin seems like the obvious choice.

Doing something useful

What we need now is a substantial demo. Since I had played around with xpectrum anyway, I just went ahead and ported that as the first homebrew program for the Actions Semiconductors platform. It works every bit as well as the SPMP8000 port, except that it's even faster if you disable throttling. Here's how to try it:

WARNING: This is alpha-quality code, use it at your own risk. It is unlikely to cause any serious damage to your system, but I am not making any promises. It may also work on devices like the JXD 3000, 5000, and M1000, but it has only been tested on my JXD A3300.

1. Download installer.elf and put it on the internal flash drive as "libp1.so".
2. Download launcher.elf and put it on the internal flash drive as "new_libp1.so".
3. Choose the game icon in the main menu to bring up the file browser; this will trigger the installation process.
4. Connect the console to your computer. What you should see is that the file new_libp1.so has been deleted, and the file backup_libp1.so (and possibly backup_libp1+.so) have been created.
5. Download real_libps1.so and put it on the internal flash drive. This is the PS1 emulator that will be used from now on. (You could use the backup files created previously instead, but the emulator has to be a specific version (R1.06), so it' s better to use the downloaded one.)
6. Put xpectrum in the GAME directory.

Now you should be able to launch xpectrum from the file browser.

The Code

The following components are available for your enjoyment from the actsemi github repository:

• libactsemi: A support library that provides access to the system call interface, which includes most standard C library functions and a lot of other things, from everyday stuff like semaphores and threads to a complete GUI interface, property lists, and text encoding functions, although many of them still lack prototypes and data type definitions.
• Baselibc: A customized version of the small C library Baselibc that provides the stuff that the operating system doesn't, such as string functions. I chose it over newlib because the Actions Semi OS already provides a rather high-level interface through system calls, and it's not easy to make newlib cooperate with another C library.
• installer: The binary injection program that replaces the PS1 emulator with our own code.
• launcher: A replacement for the PS1 emulator to be installed by launcher that is able to launch both native ELF binaries and a PS1 game images.
• demo: A simple moving-bar demo that also emits a lot of random debug output. I use it to test OS features.

The xpectrum code can be found in the actsemi branch of the SPMP8000 git repo.

To build all this goodness, you need a MIPS32 cross-toolchain. I built mine using crosstool-ng and this configuration file. Unfortunately, crosstool-ng doesn't pass the target CFLAGS that you set in the config file correctly when building newlib. While we don't use the newlib libc.a, this will still cause us grief when linking with the newlib math library. I worked around this by building newlib manually with CFLAGS_FOR_TARGET="-march=mips32r2 -G0 -Os".

Have fun!

Documentation, more libraries, and a new ScummVM build

A quick update:

• I doxygenized the code and you can find the resulting documentation here. There's a boatload of information on the various interfaces available, so if you have any interest in joining the fun, check it out.
• There are four new libraries in the 3rdparty directory: libmad, libogg, libvorbis, and Tremor.
• There is a new ScummVM build with zlib, MAD, and Tremor enabled, so you can play your compressed games now.

Have fun!

SPMP_SendSignal() Research (and ScummVM)

I did some rote work and added a new header file (mcatch_cmd.h) to the library that defines all the commands SPMP_SendSignal() accepts, 128 in total. I tried my best to guess reasonable names, but the only ones we can be sure about are those that are mentioned in debug strings. There is some really interesting stuff in there: power management, TV out, system update(!), SD card detection, various audio settings (speaker switching, volume, equalizer, Dolby), FM radio control, LCD panel control (suspend/resume, brightness), camera control, media playing, and much more.

SPMP_SendSignal() most often gets a pointer to a parameter structure that is used as input and/or output. So far I have only figured out the one used to read the real-time clock, in order to give ScummVM save games the proper timestamp.

And with that we come to our main attraction: ScummVM for SPMP8000 devices. This build only contains the SCUMM v1-v6 engine (memory is tight on this platform), but supports most features already, including sound and save games. The major feature that is not supported yet is compression, both for save games and game data. This is not difficult to fix, I simply haven't gotten around to it yet. Another feature I absolutely want to add is touchscreen support, but that will have to wait a bit; my A3300 is still in flight from China. :)

I have tested this build with Fate of Atlantis (VGA/Floppy) and Maniac Mansion (Enhanced), and they both work fine. It is quite conceivable that bigger games might not run due to a lack of memory, but that remains to be tested. Source code is at the usual place. Have fun!

[UPDATE: Here's the key mapping.

• F5 -> START (brings up the in-game menu)
• ESC -> SELECT (skip cutscenes)
• Mouse cursor -> d-pad
• Left mouse button -> O
• Right mouse button -> X

]

Win32 Toolchain (and a teaser)

I painstakingly built a Win32 (MinGW) arm-eabi toolchain, maybe that'll be of use to some people. Extract it in the root of your MinGW system. (Or, better yet, get yourself a real operating system, even if it's just Mac OS or something like that.)

Apart from the previously mentioned adbg and better support for building on Windows, the library has received a few small touchups:

• The startup code, header files, and linker script have been made C++-ready.
• The newlib/eCos glue translates stat()'s st_mode correctly now, so you can distinguish between files and directories.
• The utility function libgame_chdir_game() changes to a well-defined directory (the "GAME" folder), making it easier to find data files.

And these unassuming little fixes allow us to do this:

(Yes, that's ScummVM. The photos have been taken with the JXDs, my apologies. :)

Stay tuned...

Debugging SPMP8000 Programs Without Soldering

As everybody knows, the best way to debug code is to print lots of crap to the console, not the least because that requires very little platform support: All you need is a place where you can dump data and where it can be retrieved later. On the SPMP8000, I have so far used the file system for that purpose: Just redirect stderr to a file and write everything there. There is, however, a big problem with that approach: On a system without memory protection, programming errors usually result in a system crash, which in turn leads to total loss of the logged output. A better solution would be to have a channel of communication from which the debug output can be read as it is written, such as a serial port.

Luckily, all SPMP8000 microcontrollers come with an on-chip serial port, and breaking it out seems simple enough: Most devices appear to have soldering pads where you can attach wires. So I set out to perform this operation on my JXD 100 and promptly ripped the soldering pad off the board when I tried to verify whether the solder joint is good. Determined to make this work, I then attempted to solder the TX wire directly to the leads on the microcontroller, with a soldering iron that, at its narrowest, is about as wide as two pins. Predictably, I ended up shorting the UART TX and RX lines, and I suppose the JXD 100 is now happily talking to itself when printing debug output. I briefly tried to aggravate the situation by using a knife to scratch out the solder between the pins, but luckily came to my senses before causing more serious damage.

So what now? After some contemplation I came up with an alternative solution that does not require any hardware shenanigans and thus allows me to keep my white gloves on and my hardware alive: using the line out connector as a serial port.

The audio debugging interface (adbg)

The latest feature in libspmp8k is the adbg debug interface. It takes your debug output, serializes it and sends it to the headphone jack. When you connect that to the line input on your PC and run adbg_console.py there, it will decode the signal and print the debug output to your console. No soldering, no need to employ serial-to-USB converters, just plug in a cable and you're done.

The Catch(es)

Of course nothing is ever that simple, and there are a few things that you have to keep in mind:

• If you get no output at all, make sure to you have turned up the volume up to the maximum on both the transmitting and receiving ends. We want as much clipping as possible to get a nice square wave. You may also have to tell adbg_console.py which input it should listen to.
• If you usually get correct output, but there are bursts of garbled data, make sure you have good sound hardware with good drivers on the PC side. My computer at work (HP desktop with Intel HDA sound, Linux kernel 3.4.x) inserts random bursts of noise into the input signal every once in a while and is therefore largely unusable for this application. The machine at home (EVGA 780i SLI FTW board, also Intel HDA, kernel 3.1.x), on the other hand, works without a hitch. YMMV.
• Your computer may be oddly wired. If you receive output, but it's all garbled, you may have to invert the signal (adbg_console.py option "-i").
• There is no error detection or correction, nor is there a back channel, so this is not something you can or should use to transmit data that has to be 100% error-free.
• The devices I have seen so far (JXD 100 and A1000) seem to detect the presence of headphones by impedance. A line input has a much higher impedance that headphones and is therefore not detected. This is not a problem for signal transmission because the output on the headphone jack seems to be always on anyway, but it will be a problem for your ears because the internal speaker will not be switched off. You can, however, switch the speaker off manually in the "Settings" app.
• Finally, you have to disable sound in your program when debugging because it would interfere with the debug signal.

For an example of how to use audio debugging, check the adbg demo in the demos directory. The console program adbg_console.py can be found in tools.

The console application is written in Python, supports ALSA and PortAudio, and requires pyalsaaudio (for ALSA) and/or PyAudio (for PortAudio) to be installed. ALSA is of course Linux-only, but using PortAudio it should work on Windows and Mac OS X as well, although I have not tested that.

Have fun!

xrick for SPMP8000

Here's a port of xrick for the SPMP8000 platform. I always wanted to be the first to port the most ported game of all times to a "new" platform. :) Source code is at the usual place.
Please note that I have only tested this build on the JXD 100 because my wife brought the A1000 on a trip, so I can't use it for development ATM.

There are a few new features in the library as well:

• A couple of new test firmwares (JXD A16, 2000, and 300); all passed the tests without any changes.
• I have added two NativeGE hooks I found in the A16 firmware, NativeGE_SPUCommand() and NativeGE_getTPEvent(). I have no idea what the former does, but the latter obviously allows you to poll for touchscreen events. I do not have a touchscreen SPMP8000 device (yet), so I cannot test this right now.
• Followed a suggestion by Sea S to fall back to the HZX16 font if the Sunplus fonts are not available. This is a phenomenon I have never encountered in any official firmware, but there seem to be custom firmwares with those fonts removed.
• I have exposed _putc (and named it _diag_putc to avoid clashing with newlib). It's the character output function used by diag_printf(), which is used for all operating system debug output. By redirecting it to your own implementation you can, for instance, redirect system debug output to a file.

Have fun!

Better SPMP8000 Homebrewing (and PC Engine emulator!)

I recently got myself a JXD A1000 and found it quite an enjoyable device to develop for. The "official" API ("native game" interface) isn't any fun, though: Fixed screen size of 320x240, no access to all but three buttons, no support for generated audio, no access to the ROM file system, not even a way to list directories.

The firmware is, of course, able to do much better than that, but while all the functionality is there, we don't know how to access it: function and memory addresses are different from device to device and from firmware revision to firmware revision. While it was easy to adapt, for instance, Triple Oxygen's libemu for the Cybergame to my A1000 to get sound support, the result would again only run on a single machine. I didn't really want to spend a lot of time developing software that won't run on other people's systems, so I did some thinking and came up with a solution: On-the-fly firmware reverse-engineering.

Whenever I found something interesting in the firmware, I devised a simple heuristic that weasels its way through the running operating system, usually starting from a function the address of which we know because it's part of the "official" interface, and taking advantage of typical subroutine calls, address loads, debug strings, and function table offsets to get to the sweet stuff in a way that works on all SPMP8000 firmwares. To check that it really does work everywhere I availed myself of eleven different firmwares (and an additional JXD 100) and wrote a test harness that verifies that the heuristic actually arrives at the right spot.

The result is a library that provides you with a greatly extended API in a portable fashion, so your programs will (hopefully) run on any SPMP8k device. Here's the stuff you couldn't get before:

• Emulator interface: This is the API used by the built-in emulators. It provides hardware scaling from arbitrary resolutions, access to all keys on all controllers (both raw scan codes and standardized mappings), and streaming audio.
• Frame buffer access: Direct access to the currently active display device's frame buffers, with support for single and double buffering.
• eCos POSIX-like file API: Access to all files without restrictions or filtering. Full set of directory functions. (Seriously, with the official API you can't even read a directory!) I have glued this interface to newlib, so you can use stdio functions (fopen() etc.).
• High-resolution timer: The official API gives you mere 100 Hz; with the new libgame_utime(), you can get microsecond resolution. Indispensable for accurate speed and profiling.
• CPU scaling: Saves battery power, you can go down as low as 5 MHz. (Sorry, no overclocking.)
• Built-in fonts: A few simple routines I wrote to use the bitmap fonts in ROM. Not as black magic as the other stuff, but with full Unicode support. :)

There's more stuff in there, like the mysterious SPMP_SendSignal() function that takes dozens of commands, or the eCos threading interface, that I will unearth as needed. I also more or less worked out the signatures of all the as-yet undocumented functions in both the "native game" and the emulator API, if you want to experiment a little.

Using this souped-up libspmp8k, I have ported PC Engine emulator TGEmu, and it runs like hell (after some optimization) on both the JXD A1000 and 100, and probably on your SPMP8000 device, too. Check it out! (You need PC Engine ROMs, of course. Uncompressed, zipped, and gzipped files are supported.) The source code should also give you a good idea of how to use the library. Have fun!