Introduction

Boost.Locale is a library that brings high quality localization facilities in C++ way. It uses std::locale, and std::locale facets in order to provide localization in transparent and C++ aware way to user.

C++ has quite a good base for localization via existing C++ locale facets: std::num_put, std::ctype, std::collate etc.. But they are very limited and sometimes buggy by design. The support of localization varies between different operating systems and incompatible between them.

On the other hand, there is great, well debugged, high quality, widely used ICU library that gives all of the goodies but, it has very old API that mimics Java behavior, it completely ignores STL and provides useful API only for UTF–16 encoded text, ignoring other popular Unicode encodings like UTF–8 and UTF–32, or limited national character sets like Latin1.

Boost.Locale provides the natural glue between C++ locales framework, iostreams and powerful ICU library in following areas:

Tutorial

Locale Generation

The major “container” of all localization information in C++ is class std::locale. It is designed to hold all general information about specific culture and can be easily extended with additional resources providing information about specific area: facets. Facets are classes derived from std::locale::facet that hold required resources.

Each locale is defined by specific locale identifier that contains a mandatory part—Language and optional pars Country, Variant, and keywords, when we use narrow strings (a.k.a. std::string) we need to specify encoding we use as well.

First we generate our locale with all required facets and then we can use it. Class boost::locale::generator provides us such tool. The simplest way to use generator is to create a locale and set it as global one:

#include <boost/locale.hpp>

using namespace boost::locale;
int main()
{
    generator gen;
    // Create locale generator 
    std::locale::global(gen(""));
    // Set system default global locale
}

Of course we can specify locale manually, using default system encoding:

std::locale loc = gen("en_US"); 
// Use English, United States locale

Or specify both locale and encoding independently or using POSIX locale specifier that includes both locale information and encoding:

std::locale loc = gen("ja_JP","UTF-8"); 
// Separation of locale and encoding
std::locale loc = gen("ja_JP.UTF-8");
// POSIX locale name with encoding

When you generate more then one locale, you may specify the default encoding used for std::string by calling octet_encoding member function of generator. For example:

generator gen;
gen.octet_encoding("UTF-8");

std::locale en=gen("en_US");
std::locale ja=gen("ja_JP");

Note: Even if your application uses wide strings anywhere it is recommended to specify 8-bit encoding that would be used for all wide stream IO operations like wcout or wfstream.

Tip: Prefer using UTF–8 Unicode encoding over 8-bit encodings like ISO–8859-X ones.

By default the locale generated for all supported categories and character types. However, if your application uses strictly 8-bit encodings, uses only wide character encodings only or it uses only specific parts of the localization tools you can limit facet generation to specific categories and character types, by calling categories and characters member functions of generator class.

For example:

generator gen;
gen.characters(wchar_t_facet);
gen.categories(collation_facet | formatting_facet);
std::locale::global(gen("de_DE.UTF-8"));

Collation

Boost.Locale provides collator class derived from std::collate that extends it with support of comparison levels: primary—the default one, secondary, tertiary and quaternary levels. They can be approximately defined as:

  1. Primary—ignore accents and characters’ case compare base characters only. For example “facade” and “Façade” are same.
  2. Secondary—ignore characters case but consider accents “facade” and “façade” are different but “Façade” and “façade” are same.
  3. Tertiary—do not ignore case and accents: “Façade” and “façade” are different
  4. Quaternary—the words are identical in terms of Unicode representation

There are two ways of using collator facet: direct by calling its member functions compare, transform and hash or indirect by using comparator template class in STL algorithms.

For example:

wstring a=L"Façade", b=L"facade";
bool eq = 0 == use_facet<collator<wchar_t> >(loc).compare(collator_base::secondary,a,b);
wcout << a <<L" and "<<b<<L" are " << (eq ? L"identical" : L"different")<<endl;

std::locale is designed to be useful as comparison class in STL collection and algorithms. In order to get similar functionality with addition of comparison levels you use comparator class.

std::map<std::string,std::string,comparator<char,collator_base::secondary> > strings;
// Now strings uses default system locale for string comparison

You can also set specific locale or level when creating and using comparator class:

comparator<char> comp(some_locale,some_level);
std::map<std::string,std::string,comparator<char> > strings(comp);

Conversions

There is a set of function that perform basic string conversion operations: upper, lower and title case conversions, case folding and Unicode normalization. The functions are called to_upper, to_lower, to_title, fold_case and normalize.

You may notice that there are existing functions to_upper and to_lower under in Boost.StringAlgo library, what is the difference? The difference is that these function operate over entire string instead of performing incorrect character-by-character conversions.

For example:

std::wstring gruben = L"grüßen";
std::wcout << boost::algorithm::to_upper_copy(gruben) << " " << boost::locale::to_upper(gruben) << std::endl;

Would give in output:

GRÜßEN GRÜSSEN

Where a letter “ß” was not converted correctly to double-S in first case because of limitation of std::ctype facet.

Notes:

Numbers, Time and Currency formatting and parsing

All formatting and parsing is performed via iostream STL library. Each one of the above information types is represented as number. The formatting information is set using iostream manipulators. All manipulators are placed in boost::locale::as namespace.

For example:

cout << as::currency << 123.45 << endl;
// display 123.45 in local currency representation.
cin >> as::currency >> x ;
// Parse currency representation and store it in x

There is a special manipulator as::posix that unset locale specific settings and returns back to ordinary, default iostream formatting and parsing methods. Please note, such formats may still be localized by default std::num_put and std::num_get facets.

Numbers and number manipulators

These are manipulators for number formatting:

Currency formatting

These are manipulators for currency formatting:

Note as::currency_XYZ manipulators do not affect on general formatting, but only on the format of currency, it is necessary to use both manipulators in order to use non-default format.

Date and Time formatting

Dates and times are represented as POSIX time. When date-time formatting is turned on in the iostream, each number is treated as POSIX time. The number may be integer, or double.

There are four major manipulators of Date and Time formatting:

For example:

double now=time(0);
cout << "Today is "<< as::date << now << " and tommorrow is " << now+24*3600 << endl;
cout << "Current time is "<< as::time << now << endl;
cout << "The current weekday is "<< as::ftime("%A") << now << endl;

There are also more fine grained control of date-time formatting is available:

These manipulators, when used together with as::date, as::time, as::datetime manipulators change the date-time representation. The default format is medium.

By default, the date and time is shown in local time zone, this behavior may be changed using following manipulators:

For example:

double now=time(0);
cout << as::datetime << as::locale_time << "Locale time is: "<< now << endl;
cout << as::gmt << "GMT Time is: "<< now <<endl;
cout << as::time_zone("EST") << "Eastern Standard Time is: "<< now <<endl;

The list of all available time zone IDs can be received as set<string> using all_zones static member function of boost::locale::time_zone class.

There is a list of supported strftime flags:

Unsupported strftime flags are: %C, %u, %U, %V, %w, %W. Also O and E modifiers are not supported.

General recommendations:

Internals

All formatting information is stored in stream class by using xalloc, pword, and register_callback member functions of std::ios_base. All the information is stored and managed using special object binded to iostream, all manipulators just change its state.

When a number is written to the stream or read from it. Custom Boost.Locale facet access to this object and checks required formatting information. Then it creates boost::locale::formatter object that actually formats the number and caches it in the iostream. When next time another number is written to the stream same formatter would be used unless some flags had changed and formatter object is invalid.

boost::locale::formatter<CharType> class generally used via special facets but it can be used directly as well. It has two kinds of member functions:

Generally direct using of formatter class is preferred when the performance is critical.

Messages Formatting

Introduction

Messages formatting is probably the most important part of localization—making your application to speak in users language.

Boost.Locale uses GNU Gettext localization model. It is recommended to read general documentation of GNU Gettext that may be out of scope of this document.

The model is following:

Loading dictionaries

All the dictionaries are loaded by generator class. So, in order to use localized strings in the application you need to specify following:

  1. The search path of the dictionaries
  2. The application domain (or name)

It is done by calling following member functions of generator class:

At least one domain and one path should be specified in order to load dictionaries.

For example, our first fully localized program:

#include <boost/locale.hpp>
#include <iostream>

using namespace std;
using namespace boost::locale;

int main()
{
    generator gen;

    // Specify location of dictionaries
    gen.add_messages_path(".");
    gen.add_messages_domain("hello");

    // Generate locales and imbue them to iostream
    locale::global(gen(""));
    cout.imbue(locale());

    // Display a message using current system locale
    cout << translate("Hello World") << endl;
}

Message Translation

These are basic translation functions

These functions return special Proxy object of type message. It holds all required information for string formatting. When this object is written to an output iostream it performs dictionary lookup of the id using locale imbued in iostream. If the message is found in the dictionary is written to the output stream, otherwise the original string is written to the stream.

Notes:

This allows postpone translation of the message to the place where translation is actually needed, even to different locale targets.

std::ofstream en,ja,he,de,ar;
std::wfstream w_ar;

void send_to_all(message const &msg)
{
    en << msg;
    ja << msg
    he << msg;
    de << msg;
    ar << msg;
    w_ar << ms;
}

main()
{
    ...
    send_to_all(translate("Hello World"));
}

Plural Forms

GNU Gettext catalogs has simple, robust and yet powerful plural forms support. It is recommended to read some original GNU documentation there.

Let’s try to solve a simple problem, display a message to user:

if(files == 1)
    cout << translate("You have 1 file in the directory") << endl;
else
cout < format(translate("You have {1} files in the directory")) % files << endl;

This quite simple task becomes quite complicated when we deal with language other then English. Many languages have more then two plural forms. For example, in Hebrew there are special forms for single, double, plural, and plural above 10. They can’t be distinguished by simple rule “n is 1 or not”.

The correct solution is:

cout << format(tranlsate("You have 1 file in the directory",You have {1} files in the directory",files)) % files << endl;

Where translate receives single, plural form of original string and the number it should be formatted for. On the other side, special entry in the dictionary specifies the rule to choose the correct plural form in the specific language, for example, for Slavic languages family there exist 3 plural forms, that can be chosen using following equation:

plural=n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

Such equation is written in the dictionary and it is evaluated during translation supplying the correct form. For more detailed information please refer to GNU Gettext: 11.2.6 Additional functions for plural forms.

Adding context information

The GNU Gettext model assumes that same source messages are translated to exactly same localized messages, but this may be wrong. For example a button label “open” is translated to “öffnen” in context of “opening file” or to “aufbauen” in context of opening internet connection in German.

Is such cases it is useful to add some context information to the original string by adding a comment.

button->setLabel(translate("#File#open"));

The comment is placed between first and the following hash symbol—‘#’. The comment is always extracted from the original string and not displayed, however it is a part of string identification. Translator should discard such comment and translate only “open” string.

For example, this how po file is expected to look like:

msgid "#File#open"
msgstr "öffnen"

msgid "#Internet Connection#open"
msgstr "aufbauen"

In order to insert ‘#’ as fist symbol you may just use double hash string, for example:

cout<< translate("$ - Dollar symbol") << endl
    << translate("## - Hash symbol") << endl;

Note: Hash based comments are extension of the GNU Gettext library.

Working with multiple messages domains

In some cases it is useful to work with multiple domains, for example if application consists of several independent modules, it may have several domains. For example, if application consists of modules “foo”, “bar” it is possible to specify which dictionary should be used.

There are two ways of using non-default domains:

Questions and Answers

Code-page conversions

Boost.Locale provides codepage conversion facets based on std::codecvt facet. This allows converting between wide characters encoding and 8-bit encodings like UTF–8, ISO–8859 or Shift-JIS encodings.

Most of compilers provide such facets, but:

Thus Boost.Locale provides an option to generate code-page conversion facets for using it with Boost.Iostreams filters or std::wfstream.

Limitations:

  1. Standard does not provides any useful information about std::mbstate_t type that should be used for saving intermediate code-page conversion states. It leave the definition the compiler implementation making it impossible to reimplement std::codecvt<wchar_t,char,mbstate_t> to any stateful encodings. Thus. Boost.Locae codecvt facet implementation may be used only with stateless encodings like UTF–8, ISO–8859, Shift-JIS, but not with stateful encodings like UTF–7 or SCSU.

  2. Standard requires that code page translation can be done by translating each wide character independently. This is not a problem for most fixed width encodings like ISO–8859 family, and this is not a problem when wchar_t represents a single code point, i.e.sizeof(wchar_t)=4 which is true for most POSIX platforms.

    But under Windows, sizeof(wchar_t)=2, and this it can represent only a single character in Base Multilingual Plane (BMP) where characters with code points above 0xFFFF are represented using surrogate pairs. Because, the conversion should be stateless (above limitation) and when wchar_t can’t represent single Unicode character, only UCS–2 encoding is supported, codecvt would fail on surrogate characters of UTF–16 strings.

    Same is valid for C++0x char16_t base streams.

So, if your system supports required encoding, it would be better to use it directly instead of Boost.Locale facet.

General Recommendation: Prefer Unicode UTF–8 encoding for char based strings and files in your application.

Boundary analysis

Boost.Locale provides boundary analysis tool allowing to split the text into characters, words, sentences and find appropriate places for line breaks.

Note: Characters are not equivalent to Unicode code points. For example a Hebrew word Shalom—“שָלוֹם” consists of 4 characters and 6 Unicode points, where two code points are used for vowels (diacritical marks).

Boundary analysis is done by creating an index of break points. boost::locale::boundary::index_type, which is std::vector<boundary::break_info>. The index is created my calling static member function map of boost::locale::boundary class.

Each vector element break_info has members:

For example:

std::string text;
getline(cin,text);
boundary::index_type indx=boundary::map(boundary::character,text);
cout << translate("All characters the the sentences are:") << endl;
for(unsigned i=0;i<index.size()-1;i++) {
    unsigned char_start = index[i].offset;
    unsigned char_end = index[i].offset;
    cout << text.substr(char_start,char_end-char_start) << endl;
}

Sometimes it is important to find what kind of word break point was found. Using next and prev members of break_info we can figure this out. We can use an “or” mask of flags that are interested us:

If prev or next are 0, that means that the break distinguish between white space or punctuation marks. For example, this how we count all the words in text.

fstream file("some_text.txt");
typedef map<string,int,locale> words_type;
words_type words;
// Create a map of number of occurrences of the words using
// collation.
while(!file.eof()) {
    string tmp;
    getline(file,tmp);
    boundary::index_type indx=boundary::map(boundary::word,text);
    for(unsigned i=0;i<indx.size();i++) {
        if( (indx[i].next & (boundary::letter | boundary::kana | boundary::ideo))==0)
            // Ingore non-word and numbers
            continue;
        unsigned word_start = index[i].offset;
        unsigned word_end = index[i].offset;
        string word = tmp.substr(char_start,char_end-char_start);
        if(words.find(word)==word.end())
            word[word]=1;
        else
            word[word]++;
    }
}
for(words_type::const_iterator p=words.begin();p!=words.end();++p) {
    cout << "Word "<<p->first<<" had "<<p->second <<" occurrences "<<end;
}

The additional member brk of break_info structure can be used for testing a type of line-break. It can be tested against a mask of following flags:

An operator < defined for break_info structure. It allows using this index in STL algorithms as “binary search”. For example: cut at most 100 characters of text at word boundary:

boundary::index_type indx=boundary::map(boundary::word,text);
boundary::index_type::iterator p=lower_range(indx.begin(),indx.end(),boundary::break_info(100));
return text.substr(0,p->offset);

Localized Text Formatting

The iostream manipulators are very useful but when we create a messages to the user, sometimes we need something like old-good printf or boost::format.

Unfortunately boost::format has several limitations in context of localization:

  1. It renders all parameters using global locale rather then target ostream locale.
  2. It knows nothing about new Boost.Locale manipulators.
  3. printf like syntax is very limited for formatting of complex localized data, not allowing formatting of dates, time or currency

Thus new class boost::locale::format was introduced. For example:

wcout << wformat(L"Today {1,date} I would meet {2} at home") % time(0) % name <<endl

Each format specifier is enclosed withing {} brackets. Each format specifier is separated with comma “,” and may have additional option after symbol ‘=’. The option may be simple ASCII text or quoted localized text with single quotes “’”. If quote should be inserted to the text, it may be represented with double quote.

For example, format string:

"Ms. {1} had shown at {2,ftime='%I o''clock'} at home. Exact time is {2,time=full}"

The syntax can be described with following grammar:

format : '{' parameters '}'
parameters: parameter | parameter ',' parameters;
parameter : key ["=" value] ;
key : [0-9a-zA-Z<>] ;
value : ascii-string-excluding-"}"-and="," | local-string ; 
local-string : quoted-text | quoted-text local-string;
quoted-text : '[^']*' ;

Following format key-value pairs are supported:

The constructor of format class may receive an object of type message allowing easier integration with localized messages. For example:

cout<< format(tranlsate("Adding {1} to {2}, we get {3}")) % a % b % (a+b) << endl;

Formatted string can be fetched directly using get(std::locale const &loc=std::locale()) member function. For example:

std::wstring de = (wformat(tranlsate("Adding {1} to {2}, we get {3}")) % a % b % (a+b)).str(de_locale);
std::wstring fr = (wformat(tranlsate("Adding {1} to {2}, we get {3}")) % a % b % (a+b)).str(fr_locale);

Important Note:

There is one significant difference between boost::format and boost::locale::format: Boost.Locale format converts its parameters only when it is written to ostream or when str() member function is called. It only saves a references to the objects that can be written to a stream.

This is generally not a problem when all operations are done in one statement as:

cout << format("Adding {1} to {2}, we get {3}") % a % b % (a+b);

Because temporary value of (a+b) exists until the format is actually written to the stream. But following code is wrong:

format fmt("Adding {1} to {2}, we get {3}");
fmt % a;
fmt % b;
fmt % (a+b);
cout << fmt;

Because temporary value of (a+b) is no longer exists when fmt is written to the stream. The correct solution would be:

format fmt("Adding {1} to {2}, we get {3}");
fmt % a;
fmt % b;
int a_and_b = a+b;
fmt % a_and_b;
cout << fmt;

Getting information about current locale

std::locale::name function provides quite limited information about locale. Thus additional facet was created for giving more precise information: boost::locale::info. It has following member functions:

Working with multiple locales

Boost.Locale allows you to work safely with multiple locales in the same process. As we mentioned before, the locale generation process is not a cheap one. Thus, when we work with multiple locales it is recommended to create all used locales at the beginning and then use them.

generator class has member function preload that allows you create locale and put it into cache. Then, next time you create locale, if it is exists it would be fetched from the existing preloaded locale set.

For example:

generator gen;
gen.octet_encoding("UTF-8");
gen.preload("en_US");
gen.preload("de_DE");
gen.preload("ja_JP");
// Create all locales

std::locale en=gen("en_US"); 
// Fetch existing locale from cache
std::locale ar=get("ar_EG");
// Because ar_EG not in cache, new locale is generated (but not cached)

Then these locales can be imbued to iostreams or used directly as parameters in various functions.

Recommendations and Myths

Recommendations

Myths

In order to use Unicode in my application I should use wide strings anywhere.

Unicode property is not limited to wide strings, in fact both std::string and std::wstring are absolutely fine to hold and process Unicode text. More then that the semantics of std::string is much cleaner in multi-platform application, because, if the string is “Unicode” string then it is UTF–8. When we talk about “wide” strings they may be “UTF–16” or “UTF–32” encoded, depending on platform.

So wide strings may be even less convenient when dealing with Unicode then char based strings.

UTF–16 is the best encoding to work with.

There is common assumption that it is one of the best encodings to store information because it gives “shortest” representation of strings.

In fact, it probably the most error prone encoding to work with it. The biggest issue is code points laying outside of BMP that are represented with surrogate pairs. In fact these characters are very rare and many applications are not tested with them.

For example:

So, UTF–16 can be used for dealing with Unicode, in-facet ICU and may other applications use UTF–16 as internal Unicode representation, but you should be very careful and never assume one-code-point == one-utf16-character.

Design Rationale

Why is it needed?

Why do we need localization library, standard C++ facets (should) provide most of required functionality:

So why do we need such library if we have all the functionality withing standard library?

Almost each(!) facet has some flaws in their design:

Also many features are not really supported by std::locale at all: timezones mentioned above, text boundary analysis, numbers spelling and many others. So it is clear that standard C++ locales are very problematic for real-world applications of internationalization and localization.

Why to use ICU wrapper instead of ICU?

ICU is very good localization library but it has several serious flaws:

For example: Boost.Locale provides direct integration with iostream allowing more natural way of data formatting. For example:

cout << "You have "<<as::currency << 134.45 << " at your account at "<<as::datetime << std::time(0) << endl;

Why the ICU API is not exposed to user?

It is true, all ICU API is hidden behind opaque pointers and user have no access to it. This is done for several reasons:

Why to use GNU Gettext catalogs for message formatting?

There are many available localization formats, most popular so far are: OASIS XLIFF, GNU gettext po/mo files, POSIX catalogs, Qt ts/tm files, Java properties, Windows resources. However, the last three are popular each one in its specific area, POSIX catalogs are too simple and limited so there are two quite reasonable options:

  1. Standard localization format OASIS XLIFF.
  2. GNU Gettext binary catalogs.

The first one generally seems like more correct localization solution but… It requires XML parsing for loading documents, it is very complicated format and even ICU requires preliminary compilation of it into ICU resource bundles.

On the other hand:

So, even thou GNU Gettext mo catalogs format is not officially approved file format:

Note: Boost.Locale does not use any of GNU Gettext code, it just reimplements tool for reading and using mo-files, getting rid of current biggest GNU Gettext flaw—thread safety when using multiple locales.

Why a plain number is used for representation of date-time instead of Boost.DateTime date of Boost.DateTime ptime?

There are several reasons:

  1. Gregorian Date is by definition can’t be used for representation of locale independent dates, because not all used calendars are Gregorian.

  2. ptime—is defiantly could be used unless it had several problems:

Thus, at this point, ptime is not supported for formatting localized date and time.