QML error handling revisited

After releasing Nu, Pogodi! I learned the hard way that checking the QML runtime errors might be a good idea. For that particular application, simply checking the errors from QDeclarativeView after setting the main qml file was enough, because everything in qml file was statically declared. But what if you use QML Loader element, either explicitly or through some other qml element like PageStack from Qt Components, and something goes wrong?

Well, if you don't improve the error handling code, your application will silently fail in some places, which probably won't make the users happy. I didn't wanted to repeat the Nu, Pogodi! screw up when releasing Word Judge, so I've created a better error handling solution. First part is an error handler class:

// ----------------------------------------
//  qmlerrorhandler.h
// ----------------------------------------

class QmlErrorHandler : public QObject
{
    Q_OBJECT
public:
    explicit QmlErrorHandler(QDeclarativeView &view, QObject *parent = 0);
    bool errorOccured() const;

private slots:
    void handleQmlStatusChange(QDeclarativeView::Status status);
    void handleQmlErrors(const QList<QDeclarativeError>& qmlErrors);

private:
    QDeclarativeView &mView;
    bool mErrorOccured;

};

// ----------------------------------------
//  qmlerrorhandler.cpp
// ----------------------------------------

QmlErrorHandler::QmlErrorHandler(QDeclarativeView &view, QObject *parent) :
    QObject(parent),
    mView(view),
    mErrorOccured(false)
{
    connect(&view, SIGNAL(statusChanged(QDeclarativeView::Status)), SLOT(handleQmlStatusChange(QDeclarativeView::Status)));
    connect(view.engine(), SIGNAL(warnings(QList<QDeclarativeError>)), SLOT(handleQmlErrors(QList<QDeclarativeError>)));
}

void QmlErrorHandler::handleQmlStatusChange(QDeclarativeView::Status status)
{
    if (status == QDeclarativeView::Error) {
        handleQmlErrors(mView.errors());
    }
}

void QmlErrorHandler::handleQmlErrors(const QList<QDeclarativeError>& qmlErrors)
{
    QStringList errors;
    foreach (const QDeclarativeError& error, qmlErrors) {
        // Special case for bug in QtComponents 1.1
        // https://bugreports.qt-project.org/browse/QTCOMPONENTS-1217
        if (error.url().toString().endsWith("PageStackWindow.qml") && error.line() == 70)
            continue;

        errors.append(error.toString());
    }

    if (errors.isEmpty())
        return;

    mErrorOccured = true;

    QMessageBox msgBox;
    msgBox.setText("Uh oh, something went terribly wrong!");
    msgBox.setInformativeText("We're sorry, but it seems there are some problems "
                              "with running our application on your phone. Please "
                              "send us the following information to help us resolve "
                              "this issue:\n\n") +
                              errors.join("\n"));
    msgBox.exec();
    qApp->exit(-1);
}

bool QmlErrorHandler::errorOccured() const
{
    return mErrorOccured;
}

And here's how I use it in my applications:

int main(int argc, char *argv[])
{
    QScopedPointer<QApplication> app(createApplication(argc, argv));

    QScopedPointer<QmlApplicationViewer> viewer(QmlApplicationViewer::create());
    QmlErrorHandler errorHandler(*viewer);
    viewer->setMainQmlFile(QLatin1String("main.qml"));
    viewer->showExpanded();

    if (!errorHandler.errorOccured()) {
        return app->exec();
    } else {
        return -1;
    }
}

Basically we need to catch the runtime errors, which are emitted from QDeclarativeEngine in signal named for some unfathomable reason "warnings". Checking the errorOccured() in main() is ugly, but the qApp->exit() doesn't work until the event loop in main is started and that's the first thing which came to my mind. Please leave a comment if you know a simpler solution.

Note the lines 46-49 in QmlErrorHandler: we're catching all warnings and the qt components are not completely free of them. I had to add a special case to prevent triggering the handler on every orientation change. If you stumble upon some other errors that should be ignored, please let me know.

Word Judge for Symbian available in Nokia Store

Today I have finally pushed both versions of Word Judge through Nokia Store QA. The version with an english dictionary was available for over the week now, but I had to submit the polish version few times until it passed the Nokia tests.

Anyways, the applications are available for free in Nokia Store. Here's the link for english version:

And for polish version (might not be available in all countries - I had to add some restrictions to pass QA process):

Stay tuned for posts about development of Symbian version and enjoy your Scrabble games.

DAWG data structure in Word Judge

I'm the first to admit that the Word Judge is booooring application. Checking if the word can be used in a word game? Meh. From a programmer perspective however, there is one very interesting problem to solve - how to compress a large dictionary to reduce the size of the application package and at the same time be able to query this dictionary without using excessive amount of memory and CPU power?

First, let's settle on what is a "large dictionary". One of the languages supported by Word Judge is Polish, for which the valid word list has over 2 million entries and takes about 36MB after unpacking. Do we need to compress this data at all? Probably not. If you consider the average hardware spec and modern network speed, the 36MB is not much, but we can do so much better. Besides, it's fun!

One the other end of the spectrum is the "zip all the things" approach. It's not a good idea though - it puts a very heavy load on the CPU and doesn't compress the data that well. The zipped Polish dictionary takes 4.5MB.

The most important observation is that we're not compressing some random white noise, but words from real language. We can leverage this information to get a much better compression than some generic algorithm. Lot of words share the same prefix and suffix, which means they can be efficiently represented as a directed acyclic graph with shared nodes for common prefixes and suffixes. Let's see how the graph would look like for couple of words:

White letters on black background mark the nodes representing the ends of words. So the graph above represents the following words:

abject
abjection
abjections
abjectly
abjectness
ablate
ablated
ablation
ablations

The nodes for 'ab' prefix are of course shared, the 'ions', 's' suffixes nodes as well. For obvious reasons we cannot share the 't' node: in one group it marks the end of the word, in other it does not; the child nodes for each 't' nodes are also different. Nodes are equal, and thus can be shared, if and only if they represent the same letter, the End Of Word flag is the same for both of them, and the list of children is exactly the same.

This type of graph contains minimal number of nodes, but each node takes quite a lot of space. We need 8 bits for the letter, 1 bit for EOW flag, 8 bits for the children list length and some kind of reference, for example node index, to each child, which is log2(n) rounded up bits, where n is a number of nodes in the graph. As you can see the major contributor to a single node size is the children list: for TWL06 dictionary the number of nodes is in the 10k-100k order of magnitude, which means we need several bits per node index.

A guy called John Paul Adamovsky found a neat solution to that. Instead of keeping the full list of children, we can keep children in something similar to singly-linked list: let's store only the index of the first child, number the nodes in such way that the children always have consecutive indices and add a End Of Children List flag to each node. This way we need exactly 1 + log2(n) bits for child list. This way we can keep the entire node in one byte. What's the catch? We need to introduce few more nodes.

For example on the graph above we can no longer share the 'i' node in the 'ions' suffix: if it was shared it would have to have a number one greater than the number of both 'e' node in 'ablate' and 'n' node in 'abjectness' (this is, of course, assuming the 'i' node is the last on the child list of both 't' nodes; but it's impossible for any node ordering). Our graph would look like this:

The rule for node equality has to be extended: the nodes are equal if they represent the same letter, the flags are the same, the children are equal and the siblings further on the children list are equal. The last condition is a bit confusing, so I'll provide an example.

Let's add a 'absolution' and 'absolutions' to our dictionary. We can certainly share the 'ab' prefix and 'ons' suffix, but do we need a separate 'i' node as well? The 'i' node in 'absolution' must have a End-Of-Child-List flag set. If we arrange the child nodes in either 'ablat-' or 'abject-' part of graph in such way that the 'i' node is the last node, we can share the 'i' node between that part of graph and the newly added branch. Our graph would look like this:

This is the way we can squeeze the 36MB dictionary to 1.5MB. The structure also doesn't need any processing, it can be loaded to int array and used directly. If you're curious how to convert the dictionary into this kind of graph you can read the description of my dawggenerator project on github (C++) or the original description by John Paul Adamovsky (pure C).

Large raw assets in Android 2.2

My most recent application, Word Judge, contains full dictionary of word valid in word games. I went to great lengths to minimize the size of this data, but for some languages it's still quite large. For example polish dictionary is compressed from 35MB to 1.4MB. In Android 2.2 and earlier, if you add such large file as an asset and then try to open it, you'll get IOException. The exception itself doesn't contain any useful information, but the following text appears in logcat:

03-07 14:40:42.345: D/asset(301): Data exceeds UNCOMPRESS_DATA_MAX (1442144 vs 1048576)

With that information googling the workaround is easy. Some folks recommend splitting the file into multiple parts, or repackaging the apk, but the simplest solution is to change the file extension of your asset to one of the extensions listed in aapt tool sources as files that are not compressed by default:

/* these formats are already compressed, or don't compress well */
static const char* kNoCompressExt[] = {
    ".jpg", ".jpeg", ".png", ".gif",
    ".wav", ".mp2", ".mp3", ".ogg", ".aac",
    ".mpg", ".mpeg", ".mid", ".midi", ".smf", ".jet",
    ".rtttl", ".imy", ".xmf", ".mp4", ".m4a",
    ".m4v", ".3gp", ".3gpp", ".3g2", ".3gpp2",
    ".amr", ".awb", ".wma", ".wmv"
};

Of course if you target API level 9 or newer, you don't have to worry about this gotcha.

There is one more thing worth mentioning: the speed of IO operations in debug mode is atrocious. Loading the 1.4MB dictionary to memory takes over 10 seconds. Fortunately when the program is running without debugger attached, the loading time decreases to less than 1 second.

Introducing: Word Judge

I enjoy playing Scrabble and other word games like WordSquared of Word with Friends and I think I'm quite good at them for an amateur - my average score in 2-player Scrabble game is around 330 points. I do not have a tournament aspirations, because memorizing word lists or best stems doesn't fit my definition of fun, but I'm always in for a casual game.

The only thing that bothers me during these casual games are occasional, but unpleasant arguments - how do you spell given word or what is some 3 letter combination an obscure word or gibberish. Polish language have both declension and conjugation with a lot of irregularities, so there are plenty of things to argue about.

Most of the times we sort things out simply by checking the word spelling on the Internet, but there are situations when you can't do that, for example when you're abroad and don't have wi-fi access and roaming data access price is extraorbitant, or you're in the middle of the outback with no connectivity whatsoever. Without that possibility you have to find a compromise, and boy, that's not easy when you challenge someones 7-letter word on two triple-word bonus tiles.

That's why I created Word Judge - an ultimate way to settle all word-related disputes. The application contains full dictionary of valid words and short word list with definitions. Dictionary is shipped with the application, so you don't need internet connectivity to check if the word can be used in a word game. Currently the application is available only for Android devices, but I'm going to release Symbian (and maybe Harmattan?) version soon.



Unlike many other word game helpers, Word Judge doesn't contain anagrammer tool. It's not an oversight, it didn't add it for two reasons. First of all I don't think that such functionality should be bundled with the application, which is basically a dictionary. You use a dictionary to check the spelling, not to cheat. The second reason is more prosaic - I don't like the UI of any anagrammer tools I found. It's usually a jumble of checkboxes and radio buttons and two fields for board and rack tiles. Even if you figure out what all those controls do, these tools don't really solve the problem, which is finding the best move. It's not an easy problem, even if you find a convenient way to import current game state to the application (AR maybe?). It's certainly a good idea for a separate application, but not for a part of a dictionary app.

I released one version of my application for every supported language (currently only English and Polish). I considered setting up a service with downloadable dictionaries, but I decided against it, because it would needlessly complicate the application and I think that using multiple dictionaries is quite rare use case. Let me know if I'm wrong, I might change my mind. Also, if you like the idea of my application and want me to create a Word Judge for your language, send me an email. Unfortunately, since the app is free I cannot offer you anything more than a mention in application description and "About" window.

I was sure from the start that I won't charge money for this app, but I pondered for quite a long time if I should use ads. On one hand they are visually displeasing and take scarce screen real estate. On the other hand, I want to see for myself how much revenue they generate - the testimonies of other developers vary from ecstatic to disappointed. In the end I decided to add a small banner as an experiment.

This post is already getting too long, so let me just mention that in the next week or so I'm going to write about stuff I learned while working on this application. I hope you'll enjoy the reading. Meanwhile, dust of your Scrabble board, download my app and enjoy an argument-free game!

Customizing UI controls on Android

By customizing I mean "Oh, and can we have buttons with pink background and neon green text" requests you get from your customers. It turns out that such simple requests are in fact not simple. Take a look at the following screenshots taken on HTC Desire phone:

The "Nexus" button uses custom background selector copied from Android sources, so it looks exactly like a buttons on Nexus phones. The "HTC" button is using default background selector for HTC phones. As you can see there are few differences: highlight, corner rounding and slightly different padding. Other vendors also customize default look of UI controls: Motorola uses red highlight, and Samsung tablets use light blue highlight and there are probably some minor differences in padding and rounding as well.

Let's get back to the original problem, i.e. using pink background and neon green text. Obviously we cannot change just the text color, because it would be unreadable on HTC devices. Changing the background is also tricky, because you cannot reuse highlight graphics built into the platform resources: the different padding and corner rounding force you to use custom graphics for every state. Using layer list drawable to add some decoration is also out of the question, because of the differences in padding - most likely your decoration would be off by few pixels on some devices.

But now you have one completely custom UI control, which stands apart of built in controls. So for me it's rather "all or nothing" approach - either you customize all your UI controls (which might be very time consuming) or you use default controls.

I found one exception to this unfortunate situation: it's possible to create a ListView items with custom "normal" background and default highlight. Just use the following selector for list item background:

<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">
    <item android:state_selected="true" android:drawable="@android:color/transparent" />
    <item android:state_focused="true" android:drawable="@android:color/transparent" />
    <item android:state_pressed="true" android:drawable="@android:color/transparent" />
    <item android:drawable="@drawable/list_item" />
</selector>

DIP on BlackBerry

Android has this nice notion of density-independent pixel - a measurement unit which ensures the widgets have the same physical size on the devices with different screen size and different resolution. Basically 1dp is 1px on a device with 160dpi and is scaled appropriately on devices with different dpi.

Why is it important? Because if you design your UI in px on high resolution device and then run it on low resolution device, you'll end up with gigantic buttons which take 50% of the screen. If you do the opposite, you'll end up with button, which are too small to click on high res device.

Source: Android documentation

Unfortunately the BlackBerry doesn't support this concept out of the box. By default you specify the size and position of UI controls in regular pixels. The BlackBerry documentation suggests using this nugget of a code:

boolean lowRes = Display.getWidth() <= 320;

if (lowRes)
{
    // The device has a low resolution screen size
}
else
{
    // The device has a high resolution screen size
}

But I think there is a better way.

The net.rim.device.api.ui.Ui contains the static method convertSize that can be used, as you might have guessed, for size conversion between different measurement units. We'll of course convert our measurements to pixels, because that's the unit expected by most drawing methods. But what unit will we convert from?

Points are nice, because you probably use them for your font size already, but they are to coarse to be used for all components. Fortunately there is no need to write a wrapper for your own unit, because you can use built-in unit called centipoints. 100 centipoints = 1 point, so this unit should provide enough precision to layout elements just the way you want.