Figure 1: Login screen (Note locale es_ES.iso88591 at the bottom of submenu 1)
Harry Robinson
Sankar Chakrabarti
Internationalized applications employ two main principles [1]:
Internationalized applications are a boon to development groups, but testing these applications can become a nightmare for the test team. Simply put, since an application behaves slightly differently in each supported locale, the application should be tested in each of those locales. Doing this testing well and efficiently takes some effort.
Rather than expressing the test assertions in terms of hard-coded values, a test assertion for an internationalized test program could be expressed in locale-independent terms as follows:
If routine X executes properly, it will output the string indexed by (set-Id Y and message-Id Z) contained in the message catalog specific for the locale of execution.
It is important to note what we are doing here. We are asserting that the invariant in this relationship is the location of the string in the message catalog. We suggest that this invariant be used in deriving the test assertion as often it is practical. This practice would result in more robust and locale-independent tests.
Incorporating the message catalog locations into the test program might seem to tie the test program too closely to the application's implementation. On the contrary, we believe it is the least of several evils. It is true that the correspondence between the application and the test program will break if the application changes how it uses the message catalogs; it is far more likely, however, that the contents of the message catalogs themselves will change. Our program is immune to message catalog content changes because we do not concern ourselves with the contents, merely with the location in the catalog.
For each locale available on the test system, the test program performs the following actions:
The first hurdle in testing CDE in multiple locales is developing a method of setting locales automatically.
The user or test program logging in to CDE chooses the desired locale from language menus under the "Option" button of the login screen (see Figure 1). To select a locale, one must first know where the desired locale appears on the menu. Finding the desired locale is difficult for a test program because CDE constructs the language menus dynamically, based on how many locales are available on the system. The same locale can appear in different menu locations on different systems.
For example, Figures 1 and 2 show the login screens for two test machines. Figure 1 has two fewer locales installed than Figure 2. Note that es_ES.iso88591 (Spanish) locale appears in Figure 1 at the bottom of menu 1; in Figure 2, the same locale appears near the top of menu 2.
Figure 1: Login screen (Note locale es_ES.iso88591 at the bottom of submenu 1)
Test programs equipped with only static representations of the language menus cannot execute properly in these dynamic situations. The test must query the target environment at run-time to determine where to find the desired locale.
Our test tool, Synlib, [2] provides a structure known as a focusmap that allows us to navigate hierarchical menu systems by specifying menu locations. Once Synlib is given a selection's location, it computes the keystrokes needed to reach the targeted menu item.
Figure 2: Login screen (Note locale es_ES.iso88591 at the top of submenu 2)
However, since the structure of the menus and the position of each locale can only be determined at run-time, a static focusmap may be good for one system but may not be good for another system. The test program therefore uses a generic focusmap (Figure 3) for the language menus. The generic focusmap does not associate a locale name with any particular menu or position.
The executing test program queries the target machine with "locale -a" to determine how many locales are present on the machine. With this number, the test program calculates how many menus and submenus will appear on the login screen. By sorting the output of "locale -a", the test program also determines which menu and which item correspond to the target locale. For instance, if we want to run our tests in de_DE.iso88591, the above procedure might determine that there are 57 locales available on the target machine and that de_DE.iso88591 appears as Item 12 on Submenu 1. Our call to the focusmap utility would then specify that we wished to select the focus item:
Options.Options_menu.Language.Lang_submenu.menu1.menu1.12
Synlib then selects the appropriate item for us.
Figure 3: Generic Focusmap for the Option Menu and Submenus
This generic approach works well. The test program does not need to be told which locales or how many locales are present on the target system to be tested. Rather it is instructed to scan the system and determine what the system's language menu will look like. We can either choose a particular locale in which to test or, by feeding a loop construct with the output of "locale -a", we can cycle through all the available locales on a machine without knowing ahead of time what locales might be available. The latter technique was used in our automated test system.
The reader may recognize that the variable menu structure presented by the login screen is not strictly an effect of internationalization of CDE. Rather, the structure is determined by the number of locales configured on the target system. Nevertheless, setting the locale is a significant hurdle which had to be overcome to build a fully automated touch test suite for CDE in all locales.
In CDE, all applications have distinct titles. In our touch test, we open the application and verify that the application has the expected title. The test program should be constructed to retrieve the expected title string from the appropriate message catalog files. If an application window does not appear or displays an incorrect title, then the application has a bug. The example below illustrates the idea.
Here is a hard-coded version of a verification routine:
/* Wait for the window to map */
SynWaitWindowMap( dpy, "Editor", HOWLONG );
/* Fetch the title from the window */
SynGetWindowTitle( dpy, "Editor", &title_ptr);
/* Verify the window title */
if ( strstr(title_ptr, "Text Editor") != NULL )
isCorrectWindow = True;
else
isCorrectWindow = False;
The first level of verification, SynWaitWindowMap(), determines whether the application window actually appeared. The test reads the title of the target window using SynGetWindowTitle(). The string returned by this routine is then compared with the expected string. The problem is that the title strings are dependent on the locale of execution of the application (Table 1). The only way to make one test verify the behavior of the application in all its supported locales is to construct the test such that it can seek out the appropriate message catalog and then figure out the title string specified for that locale using the set-index and message-index for the desired string.
Table 1: Text Editor Title in Different Locales
Note that we are proposing here that the location of the window title string in the Text Editor message catalog be used as the test invariant, rather than the string itself. Once we know that the title is located in message set 7, message number 2 of the dtpad.cat catalog, we can re-code our test program as follows:
/* Open the message catalog */
my_cat=catopen("/usr/dt/lib/nls/msg/%L/dtpad.cat",NL_CAT_LOCALE);
/* Fetch the expected title string from the message catalog */
expected_title=catgets(my_cat,7,2,"Text Editor" );
/* Close the message catalog */
catclose(my_cat);
/* Wait for the window to map */
SynWaitWindowMap( dpy, "Editor", HOWLONG );
/* Fetch the title from the window */
SynGetWindowTitle( dpy, "Editor", &title_ptr);
/* Verify the window title */
if ( strstr(title_ptr, expected_title) != NULL )
isCorrectWindow = True;
else
isCorrectWindow = False;
This verification will now work in all supported locales. In fact, because
the test uses the message catalog locations, its results are immune to changes
in the strings themselves.
Take the example of the Text Editor's File menu. Figure 4 shows the File menu in the English and French locales. To close the application window in the English locale, the test program (or a human user) presses "C" for "Close". Here is the hard-coded version in Synlib:
/* Select the "Close" action */
SynClickKey( dpy, "C" );
The same input however will fail to close the application in the French locale; in French, the appropriate input to close the application is "F" for "Fermer". To verify the Close behavior, therefore, the test should press "F" and not "C''. How can the test provide the valid input in a given situation? Either all the possibilities are to be coded in the test -- which becomes increasingly complex with increasing number of locales; or the test may be constructed to consult the locale-specific message catalog to determine which is the valid input for the situation. It is our opinion that this latter approach is the cleanest and least expensive alternative.
Figure 4: File Menu in English (left) and French (right)
The illustration below shows how this test can be written in a locale-independent way using the information in the message catalogs. In the C locale, the message catalog contains the following:
$set 11 29 C 30 Close
In the fr_FR.iso88591 (French) locale, the dtpad.cat message catalog contains
$set 11 29 F 30 Fermer
Table 2: Mnemonic for the Close Action in Different Locales
As shown in Table 2, the location of the mnemonic is invariant across all locales. The test program can use this invariance to adapt itself to various locales.
/* Open the message catalog */
my_cat=catopen("/usr/dt/lib/nls/msg/%L/dtpad.cat",NL_CAT_LOCALE);
/* Get the mnemonic for the Close action */
sprintf(file_close, "<%s>", catgets(my_cat, 11, 29, "C") );
/* Close the message catalog */
catclose(my_cat);
/* Select the "Close" action */
SynClickKey( dpy, file_close );
The test suite in its current state is fairly simple. It merely sets the locale and then opens and closes the main CDE applications. In spite of its simplicity, this touch test suite uncovered a large number of defects of various levels of severity:
These defects were uncovered by our test suite even after the CDE system was thoroughly tested and debugged using other test suites.
A major suggestion of this work is that tests should use the structure of the message catalogs to compose test assertions. Without this approach there is very little hope of creating locale-independent tests and reducing the cost of testing target software. We propose that tests adopt the use of message catalogs in forming assertions as often as it is practical. The touch tests presented here for individual CDE applications were very simple. In most cases, the test consisted only of verifying that when launched in the locale of interest, the application started properly and displayed its window with the appropriate title. This test suite did not explore extensive functional testing of the applications concerned. It has become apparent to us, however, that the experience gained from this exercise can be easily adapted to create fully operational locale-independent functional tests suites for more thorough verification of the CDE applications.
[2] Sankar L. Chakrabarti: Testing X-Clients Using Synlib and FocusMaps. The X Resource, Issue 13, pp. 7-21, Winter Issue, 1995
Harry Robinson is currently a software test engineer with Microsoft's Intelligent Search Group
Sankar Chakrabarti is an R&D engineer with Hewlett Packard
