Debugging Tip: How to Set a Breakpoint In a Windows API Function

Introduction

This article is based on an EMail message I sent to the software development team at Vispero on August 26, 2008. I had just spent hours debugging an odd bug in JAWS®. When focus was in a particular window while JAWS was running I heard constant, rapid fire, beeping. I searched the JAWS source code for calls to the Beep and MessageBeep functions and was not able to determine the cause of the bug. I was at a loss. Then I asked the following question, is there a way I can just set breakpoints in the Beep and MessageBeep functions? This article discusses what I found during my quest for an answer to that question.

I am writing this article now because I encountered this issue again today and just spent five hours retracing my steps.

Note: After I wrote this article I learned that things are much easier in Visual Studio 2017 and Visual Studio 2019. In these versions of Visual Studio all that you need to do is open the “New Function Breakpoint” dialog box using the “Debug/New Breakpoint/Function Breakpoint…” menu item and enter the undecorated name of the function in the dialog box. If you are interested in learning how things used to be, follow me down the rabbit hole.

Falling down the rabbit hole

It turns out that it is possible to set a breakpoint in a Windows API function in Visual Studio. I do not know when this functionality was added to Visual Studio but I do know that it was available in Visual Studio 2005. The question is how.

The Set function breakpoints section of the Use breakpoints in the Visual Studio debugger page provides some relevant information; it provides very limited information on setting a breakpoint on a function with a known name. Simply use the “Debug/New Breakpoint/Function Breakpoint…” menu item to open the “New Function Breakpoint” dialog box and type a magical incantation in the “Function Name” edit.

Unfortunately, the information provided here is infuriating since it reveals that the feature is supported but it does not provide enough information to allow you to actually use the feature. In other words, it does not tell you what magical incantation to use.

Use the context operator in native C++.

{function, , [module]} [+]

Example: {MethodA, , App1.dll}+2

The article How Can I Debug Windows API Functions provides more information but it does not explain the context operator. It does reveal how to set a breakpoint in the MessageBeep function.

{,,USER32.DLL}_MessageBeep@4

This page mentions that you need to use the “decorated form of the function name” and suggests that you refer to the Viewing Decorated Names page for more information. The Viewing Decorated Names page in turn refers you to the Use a listing and Using DUMPBIN to View Decorated Names pages. Unfortunately, it turns out that neither of the techniques mentioned on the Viewing Decorated Names page will allow you to determine the decorated name for a function that is exported from a DLL. The DBH tool is supposed to be able to provide this information via the -d command-line option but in my tests using the following command it only provides undecorated names these days (bad Microsoft!).

dbh -s:srv*C:\SymbolCache*http://msdl.microsoft.com/Download/Symbols -d C:\Windows\System32\User32.dll enum *CreateWindowExW*

This command provides the following output.

Symbol Search Path: srv*C:\SymbolCache*http://msdl.microsoft.com/Download/Symbols

 index            address     name
     1            1007760 :   CreateWindowExW

Note that this is not the decorated name of the function since there is no @ character.

Unfortunately, I do not know an easy way to determine the “decorated form of the function name” other than to figure it out yourself using the information provided at Using Win32 calling conventions. Most Windows API function use the __stdcall calling convention. The general syntax for the decorated name of a __stdcall function is as follows.

_[UndecoratedFunctionName]@[SizeOfParametersInBytes]

This provides enough information for you to write the part of the magical incantation that comes after the } character, but what about the beginning of the magical incantation? Based on the example provided for the MessageBeep function one could assume that it should simply be the name of the DLL that implements the Windows API function, but is more information available?

Fortunately, the Context Operator in the Visual Studio Debugger (C++) page does provide a more detailed explanation for the first part of the magical incantation. This page reveals part of the syntax for the Context Operator.

{,,[module]} expression

Of course, this very annoyingly leaves a question unanswered. Why does this text between the { character and the } character start with two commas? What are you leaving out?

I have not been able to find official Microsoft documentation that answers that question yet. However, I do have an answer for you from Breaking on System Functions with the Context Operator. This page reveals that the full syntax of the Context Operator is as follows.

{[function],,[module]} expression

In most cases you will not need to worry about [function] and , you will only need to worry about the module and the expression.

Thus, the syntax for the magical incantation you will need to set the breakpoint is as follows.

{,,[module]}_[UndecoratedFunctionName]@[SizeOfParametersInBytes]

Pulling everything together.

To set a breakpoint in any Windows API function, such as the CreateWindowExW function, do the following.

  1. Look up the Windows API function in the Microsoft documentation. In the Requirements section the DLL the function is implemented in is provided; in this case it is User32.dll.
  2. Determine the size of the function parameters in bytes, as follows.
ParameterSize in bits on a 32-bit systemSize in bytes on a 32-bit systemSize in bits on a 64-bit systemSize in bytes on a 64-bit system
DWORD dwExStyle324324
LPCWSTR lpClassName324648
LPCWSTR lpWindowName324648
DWORD dwStyle324324
int X324324
int Y324324
int nWidth324324
int nHeight324324
HWND hWndParent324648
HMENU hMenu324648
HINSTANCE hInstance324648
LPVOID lpParam324648
Totals4868
  1. Fill in the blanks.

If you are debugging a 32-bit application the magical incantation is as follows.

{,,User32.dll}_CreateWindowExW@48

If you are debugging a 64-bit application the magical incantation is as follows.

{,,User32.dll}_CreateWindowExW@68
  1. Enter the appropriate magical incantation in the “Function Name” edit of the “New Function Breakpoint” dialog box and press the OK button.

Note that in order for this to work you will need to set up Visual Studio to use the Microsoft symbol server. To do this select the “Debug/Options…” menu item to open the Debugging section of the “Options” dialog box. Then select “Debugging/Symbols” in the tree view. Next press tab to move focus to the “Symbol file (.pdb) locations” check list box and check the “Microsoft Symbol Servers” item. Then press the OK button.

References

Down the Rabbit Hole (CC BY-SA 2.0) by Valerie Hinojosa

On writing code with debugging in mind

/*************************************************
*                                                *
* .=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-. *
* |                   ______                   | *
* |                .-"      "-.                | *
* |               /            \               | *
* |   _          |              |          _   | *
* |  ( \         |,  .-.  .-.  ,|         / )  | *
* |   > "=._     | )(__/  \__)( |     _.=" <   | *
* |  (_/"=._"=._ |/     /\     \| _.="_.="\_)  | *
* |         "=._"(_     ^^     _)"_.="         | *
* |             "=\__|IIIIII|__/="             | *
* |            _.="| \IIIIII/ |"=._            | *
* |  _     _.="_.="\          /"=._"=._     _  | *
* | ( \_.="_.="     `--------`     "=._"=._/ ) | *
* |  > _.="                            "=._ <  | *
* | (_/                                    \_) | *
* |                                            | *
* '-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=' *
*                                                *
*    ABANDON ALL HOPE, ALL YE WHO ENTER HERE     *
*************************************************/
ASCII art image of a skull and crossbones with the text “abandon all hope, all ye who enter here” displayed below the skull and crossbones.

One factor of code quality that does not get enough attention is how easy is it to debug the code. No matter how skilled the team that writes the software, there will be bugs. When these bugs are found some poor soul will need to debug the code to determine the cause of the bug. Unfortunately this can quickly turn into a nightmare because many functions are not debuggable. Sometimes even well designed code is so difficult to debug that it causes the poor soul who has to debug it to abandon all hope.

Non-Debuggable Function

The following function is very simple. At first glance it seems to be a concise and well designed function. However, it is a major contributor to the debugging this application is a nightmare problem.


inline bool CCoolAppDocument::IsValidStartOffset(long nOffset) const
{
    return nOffset >=0 && nOffset < GetLength();
}

You may be wondering, how can this innocent, one-line function cause someone to abandon all hope?

When the function works as expected (meaning it returns the value you expect) it cannot. However, when the function returns false when you think it should return true the nightmare begins. The most critical piece of information that might explain why this function is not returning the expected value, the document length, is hidden away within a function call. As a result, there is no way you can look at parameter and variable values in the debugger watch window and know why the function is behaving in the way that it is.

Debuggable Function

The following function does exactly the same thing but it can be debugged in a microsecond.


inline bool CCoolAppDocument::IsValidStartOffset(long nOffset) const
{
    const auto documentLength = GetLength();
    return (nOffset >=0 && nOffset < documentLength);
}

All you have to do to debug this function is look at the value of the nOffset parameter and the documentLength variable in the watch window and then the reason for the unexpected behavior is immediately obvious.

In the case of today’s nightmare that inspired this article, the problem was that both the nOffset parameter and the documentLength variable happened to be equal to 296.

Additional Comments

I know that some of you are wondering whether I am aware that some debuggers will provide the return value of a function that was just called. The answer is of course I am aware of that. However, as far as I know not all debuggers support this functionality. In addition, this functionality is not as easy to use or as intuitive as the watch variable feature.

To me it just makes sense to make a very small change to make it possible to diagnose problems with the more intuitive and easier to use watch variable feature of the debugger.

Using a select element to change the active stylesheet for a web page.

This article discusses the use of a select element to change the active stylesheet for a web page. It also provides step by step instructions on how to create the select element.

  • Add a link element to the head of your web page. Set the rel attribute to “stylesheet,” the href attribute to the URL of the default stylesheet you wish to use for your web page, and the id attribute to a unique value.

    Code Example:

    <link
      rel="stylesheet"
      href="https://www.w3.org/StyleSheets/Core/Swiss"
      id="W3C-Core-Styles" />
    

  • Add the select element. Set the value attribue of each option element to the URL of the stylesheet and the contents of each option element to the text you wish to be displayed for the combo box.

    Code Example:

    <select id="SeclectW3CCoreStyle">
    <option
      value="https://www.w3.org/StyleSheets/Core/Chocolate">
    Chocolate
    </option>
    <option
      value="https://www.w3.org/StyleSheets/Core/Midnight">
    Midnight
    </option>
    <option
      value="https://www.w3.org/StyleSheets/Core/Modernist">
    Modernist
    </option>
    <option
      value="https://www.w3.org/StyleSheets/Core/Oldstyle">
    Oldstyle
    </option>
    <option
      value="https://www.w3.org/StyleSheets/Core/Steely">
    Steely
    </option>
    <option
      value="https://www.w3.org/StyleSheets/Core/Swiss">
    Swiss
    </option>
    <option
      value="https://www.w3.org/StyleSheets/Core/Traditional">
    Traditional
    </option>
    <option
      value="https://www.w3.org/StyleSheets/Core/Ultramarine">
    Ultramarine
    </option>
    </select>
    

    Typically, I add this select element, along with a lable and the Set Style button, to the footer at the bottom of the page.

  • Add a Set Style button, as follows.

    Code Example:

    <input
      id="SetW3CCoreStyleButton"
      onclick="SetStyle('SeclectW3CCoreStyle', 'W3C-Core-Styles')"
      type="button"
      value="Set Style" />
    

  • Add the following JavaScript code to a script element in your page or to a .js file on your website.

    function isEmptyOrSpaces(str){
      if (str == null){
        return true;
      }
      if (str.length == 0){
        return true;
      }
      if (str.trim() === ''){
        return true;
      }
      return false;
    }
    
    function SetStyle(selectStyleElementId, linkElementId){
      console.log(
        "SetStyle('" + selectStyleElementId
        + "', '" + linkElementId
        + "') called.");
      let seclectStyleElement =  document.getElementById
        selectStyleElementId);
      if (!seclectStyleElement){
        console.log(
          "Could not find the '" + selectStyleElementId
          + "' element.");
        return false;
      }
      let seclectedStyle = seclectStyleElement.value;
      if (isEmptyOrSpaces(seclectedStyle)){
        console.log("Could not get the selected style.");
        return false;
      }
      let linkElement = document.getElementById(linkElementId);
      if (!linkElement){
        console.log(
          "Could not find the '" + linkElementId
          + "' element.");
        return false;
      }
      linkElement.setAttribute("href", seclectedStyle);
      return true;
    }
    

This example is fully functional. It does lack one key feature, the ability to store the selected style in a cookie. The article Store and Retrieve Cookies demonstrates the use of the localStorage.setItem and localStorage.getItem functions.

Modify the SetStyle function by adding the folloing line just above the ‘return true;’ line at the bottom of the function.


window.localStorage.setItem(
  "SeclectedStyle", seclectedStyle);

Note that the localStorage object is only supported on modern web browsers. That said, the Local storage with Window.localStorage page provides JavaScript code to emulate the feature for older browsers.

The next step is to set the style and the selected item in the select element when the web page loads. To acomplish this you need to add the following function.


function SetStyleFromCookie(selectStyleElementId, linkElementId){
  console.log(
    "SetStyleFromCookie('" + selectStyleElementId + "', '" + linkElementId + "') called.");
  let seclectedStyle = window.localStorage.getItem(
    "SeclectedStyle");
  if (isEmptyOrSpaces(seclectedStyle)){
    console.log(
      "Failed to get the value of the 'SeclectedStyle' cookie. Using the default value.");
    seclectedStyle = 'https://www.w3.org/StyleSheets/Core/Swiss';
  }
  let linkElement = document.getElementById(linkElementId);
  if (!linkElement){
    console.log("Could not find the '" + linkElementId + "' element.");
    return false;
  }
  linkElement.setAttribute("href", seclectedStyle);
  let seclectStyleElement =  document.getElementById(selectStyleElementId);
  if (!seclectStyleElement){
    console.log("Could not find the '" + selectStyleElementId + "' element.");
    return false;
  }
  seclectStyleElement.value = seclectedStyle;
  return true;
}

Finally, use the following code to call the SetStyleFromCookie function when the web page is loaded.


window.addEventListener('DOMContentLoaded', (event) => {
  SetStyleFromCookie('SeclectW3CCoreStyle', 'W3C-Core-Styles');
});

References

To see this code in action, go to Using a select element to change the active stylesheet for a web page.

Providing Accessibility Information in a .NET Windows Forms application

Windows Forms applications provide two mechanisms for providing accessibility information.

In many cases the required accessibility information can be provided using the following properties of the Control Class in the System.Windows.Forms Namespace: the Control.AccessibleDefaultActionDescription Property, the Control.AccessibleDescription Property, the Control.AccessibleName Property, and the Control.AccessibleRole Property. This feature is almost enough to meet WCAG Success Criterion 4.1.2: Name, Role, Value. WCAG Success Criterion 4.1.2: Name, Role, Value requires that state information be provided as well. Unfortunately, there is not a property in the Control Class for state information.

In many cases, the built in Windows Forms support is sufficient and therefore there is no need to specifically provide state information.

If you do need to provide state information, it will take a bit more effort. In this case, you will need to use the AccessibleObject Class and the Control.CreateAccessibilityInstance Method.

Using the Accessible* Properties of the Control Class

The following code demonstrates the use of the Control.AccessibleDescription Property and the Control.AccessibleName Property of the Control Class.

In this case, the checkbox object provides the state. However, since an image is used instead of text, it does not provide the name. Therefore the Control.AccessibleDescription Property and the Control.AccessibleName Property of the Control Class are used. In addition, these properties suffice to meet WCAG Success Criterion 4.1.2: Name, Role, Value.

References

Using Direct Annotation in C++

Sometimes you will encounter a situation in which the existing accessibility support for an user interface element is almost good enough and you need to just make small changes to the existing support. For example, perhaps you just need to override the Name or Role properties. In these cases, fully implementing the IAccessible interface as described in Implementing a Microsoft Active Accessibility (MSAA) Server Using the Windows API and C++ may seem like a great deal of effort to achieve these improvements.

Fortunately, starting with Windows 7 there is an easier alternative, the Direct Annotation feature of the Dynamic Annotation API. The Dynamic Annotation API is extension to Microsoft Active Accessibility that makes it possible to customize existing IAccessible support without using subclassing or wrapping techniques. Direct Annotation is one of several features provided by the Dynamic Annotation API; it is the simplest of the features provided.

Direct Annotation makes it possible to customize the following properties.

Since Name, Role, Value, and State are included in the list of properties supported by the Direct Annotation feature, this feature is sufficient to meet WCAG Success Criterion 4.1.2: Name, Role, Value.

Implementation Details

The first step to using the Direct Annotation feature of the Dynamic Annotation API is to create an [IAccPropServices][] object using either the CoCreateInstance function or CoCreateInstanceEx function. Then you can use either the IAccPropServices::SetHwndProp method or the IAccPropServices::SetHwndPropStr method.

The following sample from Using Direct Annotation demonstrates the technique.



More complete examples can be found in the following locations.

Note that using the IAccPropServices::SetHwndProp method or the IAccPropServices::SetHwndPropStr method will not send WinEvents. It is your responsibility to send appropriate events by calling the NotifyWinEvent function after the IAccPropServices::SetHwndProp method or the IAccPropServices::SetHwndPropStr method has been called.

For example, if you use IAccPropServices::SetHwndPropStr method to set the Name property of an element, send an EVENT_OBJECT_NAMECHANGE event for that object after SetHwndPropStr returns.

The CWndWithCustomizedAccessibleInfo class and DirectAnnotation class both make the necessary calls to the NotifyWinEvent function for you.

Implementing a Microsoft Active Accessibility (MSAA) Server Using the Windows API and C++

The oldest and simplest of the Accessibility APIs used in Microsoft Windows is Microsoft Active Accessibility or MSAA. This article provides a brief introduction on how to implement a MSAA Server using the Windows API and the C++ programming language.

The IAccessible interface

The heart of MSAA is the IAccessible interface. This interface exposes methods and properties that allow an accessibility server to make a user interface element and its children accessible to accessibility client applications. The methods and properties of the IAccessible interface allow a Windows API application to meet WCAG Success Criterion 4.1.2: Name, Role, Value, which requires that an accessibility client be able to programmatically determine the Name, Role, Value, and States of all user interface components (including but not limited to: form elements, links and components generated by scripts).

The following table provides information the methods of the IAccessible interface that are used to expose each of these properties.

Property Method
Name IAccessible::get_accName
Role IAccessible::get_accRole
State IAccessible::get_accState
Value IAccessible::get_accValue

See the IAccessible interface page for a complete list of the members of the interface and a detailed description of each method. The Content of Descriptive Properties page provides information on the expected value of these properties.

The Name, Role, and State properties are required for all user interface components that can gain focus. The Value property may be required for some user interface components but not for others. More information on which members of the IAccessible interface should be implemented for a given user interface component can be found in the Choosing Which Properties to Support page.

How an accessibility client obtains an IAccessible object

An accessibility client use one of the following functions to obtain an IAccessible object.

See the Getting an Accessible Object Interface Pointer page for more information on these functions.

What happens when an accessibility client requests an IAccessible object?

When an accessibility client requests an IAccessible object using the AccessibleObjectFromEvent, AccessibleObjectFromPoint, and AccessibleObjectFromWindow functions Microsoft Active Accessibility sends the WM_GETOBJECT message to the appropriate window procedure within the appropriate server application with the lParam parameter set to OBJID_CLIENT. Your Microsoft Active Accessibility (MSAA) server needs to handle this message.

Handling the WM_GETOBJECT Message

When your Microsoft Active Accessibility (MSAA) Server receives the WM_GETOBJECT message it should return the value obtained by passing the IAccessible interface of the object to the LresultFromObject function as follows.



The above code is from the Windows API Accessibility Server sample application.

For more information on handling the WM_GETOBJECT message see the following.

Events

Implementing the IAccessible interface is just one part of the puzzle. You also need to notify fire event notifications. This is done using the NotifyWinEvent function. See Event Constants, System-Level and Object-Level Events, and What Are WinEvents? for more information.

Additional Resources

Additional information can be found in the following locations.

Using Character or String Constants in a Template function

Using Character or String Constants in a Template function

Ben Key:

October 1, 2013; November 10, 2018

 


This article provides several techniques that may be used when creating template functions in which the character type is a template parameter and you must use character or string constants. It also provides the source code for several template functions and macros that may be used to implement one of the proposed solutions.

Problem Description

There are various data types that may be used to represent characters in C++. The most common of these are char and wchar_t. It often is necessary to write code that is capable of handling either type of character.

One alternative is to simply implement the function once for each character type that needs to be supported. There are obvious problems with this approach. The first problem is that this approach causes unnecessary code duplication. It also opens up the possibility that the different implementations of the function will diverge over time as changes are made in one implementation, for example in order to fix bugs, but not in the other implementations of the function.

Consider the following example.

The CommandLineToArgVector function parses a command line string such as that which might be returned by the GetCommandLine function. This function depends on a number of character constants: specifically NULCHAR (’\0’), SPACECHAR (’ ‘), TABCHAR (’), DQUOTECHAR (’”‘), and SLASHCHAR (’\’).

In order to support characters of both char and wchar_t it is necessary to implement the function twice as follows.


inline size_t CommandLineToArgVector(
    const char* commandLine,
    std::vector<std::string>& arg_vector)
{
    arg_vector.clear();
    /* Code omitted. */
    return static_cast<size_t>(arg_vector.size());
}

inline size_t CommandLineToArgVector(
    const wchar_t* commandLine,
    std::vector<std::wstring>& arg_vector)
{
    arg_vector.clear();
    /* Code omitted. */
    return static_cast<size_t>(arg_vector.size());
}

The obvious solution is to implement this as a template function that accepts the character type as a template parameter as follows.


template<typename CharType>
size_t CommandLineToArgVector(
    const CharType* commandLine,
    std::vector< std::basic_string<CharType> >& arg_vector)
{
    arg_vector.clear();
    /* Code omitted. */
    return static_cast<size_t>(arg_vector.size());
}

The only thing that prevents us from doing that is the existence of the character constants. How do you represent the constants so that they are can be of either data type?

Possible Solutions

As is often the case, there are many possible solutions to this problem. This article discusses several of them.

Widen

One possible solution is to make use of std::ctype::widen as follows.


template <typename CharType>
CharType widenChar(
    const char ch, const std::locale& loc = std::locale())
{
    const auto& cType = std::use_facet<std::ctype<CharType>>(loc);
    return cType.widen(ch);
}

This same technique can be extended for use with string literals as follows.


template<typename CharType>
std::basic_string<CharType> widenString(
    const char* str, const std::locale& loc = std::locale())
{
    std::basic_string<CharType> ret;
    if (str == nullptr || str[0] == 0)
    {
        return ret;
    }
    const auto& cType = std::use_facet<std::ctype<CharType>>(loc);
    auto srcLen = std::strlen(str);
    auto bufferSize = srcLen + 32;
    auto tmpPtr = yekneb::make_unique<CharType[]>(bufferSize);
    auto tmp = tmpPtr.get();
    cType.widen(str, str + srcLen, tmp);
    ret = tmp;
    return ret;
}

Then in the template function you can use the character constants in a character type neutral way as follows.


const CharType NULCHAR = widenChar<CharType>('\0');
const CharType SPACECHAR = widenChar<CharType>(' ');
const CharType TABCHAR =  widenChar<CharType>('\t');
const CharType DQUOTECHAR = widenChar<CharType>('\"');
const CharType SLASHCHAR =  widenChar<CharType>('\\');

The problem is that this requires a multibyte to wide character conversion for the wchar_t case. This may not be much of an issue for functions that are not used very often and are not performance critical. However the cost of the multibyte to wide character conversion would not be acceptable in performance critical situations.

The question is how to solve this problem without these performance issues.

Algorithm Traits

Another option is to create an algorithm traits class that includes functions that return each of the constants as follows.


template <typename CharType>
struct CommandLineToArgVector_Traits
{
    CharType NULCHAR();
    CharType SPACECHAR();
    CharType TABCHAR();
    CharType DQUOTECHAR();
    CharType SLASHCHAR();
    CharType* STRING();
};

template<>
struct CommandLineToArgVector_Traits<char>
{
    char NULCHAR()
    {
        return '\0';
    }
    char SPACECHAR()
    {
        return ' ';
    }
    char TABCHAR()
    {
        return '\t';
    }
    char DQUOTECHAR()
    {
        return '\"';
    }
    char SLASHCHAR()
    {
        return '\\';
    }
    char* STRING()
    {
        return "String";
    }
};

template<>
struct CommandLineToArgVector_Traits<wchar_t>
{
    wchar_t NULCHAR()
    {
        return L'\0';
    }
    wchar_t SPACECHAR()
    {
        return L' ';
    }
    wchar_t TABCHAR()
    {
        return L'\t';
    }
    wchar_t DQUOTECHAR()
    {
        return L'\"';
    }
    wchar_t SLASHCHAR()
    {
        return L'\\';
    }
    wchar_t* STRING()
    {
        return L"String";
    }
};

As you can see, it is a simple matter to incorporate string constants in the algorithm traits structure.

The algorithm traits structure can then be used as follows in your template function.


CommandLineToArgVector_Traits<CharType> traits;
const CharType NULCHAR = traits.NULCHAR();
const CharType SPACECHAR = traits.SPACECHAR();
const CharType TABCHAR =  traits.TABCHAR();
const CharType DQUOTECHAR = traits.DQUOTECHAR();
const CharType SLASHCHAR =  traits.SLASHCHAR();
const CharType* STRING = traits.STRING();

This solution does not have the same performance issues that the widenChar solution does but these performance gains come at the cost of a significant increase in the complexity of the code. In addition, there is now a need to maintain two implementations of the Algorithm Traits structure, which means that we are right back where we started from, though admittedly maintaining two implementations of the Algorithm Traits structure is a lot less work than maintaining two implementations of the algorithm.

It would be ideal to develop a solution that has the convenience and simplicity of the widenChar solution without the performance issues. The question is how.

Preprocessor and Template Magic

Fortunately it is possible to solve this problem using some preprocessor and template magic. I found this solution in the Stack Overflow article How to express a string literal within a template parameterized by the type of the characters used to represent the literal.

The solution is as follows.


template<typename CharType>
CharType CharConstantOfType(const char c, const wchar_t w);

template<>
char CharConstantOfType<char>(const char c, const wchar_t /*w*/)
{
    return c;
}
template<>
wchar_t CharConstantOfType<wchar_t>(const char /*c*/, const wchar_t w)
{
    return w;
}

template<typename CharType>
const CharType* StringConstantOfType(const char* c, const wchar_t* w);

template<>
const char* StringConstantOfType<char>(const char* c, const wchar_t* /*w*/)
{
    return c;
}
template<>
const wchar_t* StringConstantOfType<wchar_t>(const char* /*c*/, const wchar_t* w)
{
    return w;
}

#define _TOWSTRING(x) L##x
#define TOWSTRING(x) _TOWSTRING(x)
#define CHAR_CONSTANT(TYPE, STRING) CharConstantOfType<TYPE>(STRING, TOWSTRING(STRING))
#define STRING_CONSTANT(TYPE, STRING) StringConstantOfType<TYPE>(STRING, TOWSTRING(STRING))

Then in the template function you can use the character constants in a character type neutral way as follows.


const CharType NULCHAR = CHAR_CONSTANT(CharType, '\0');
const CharType SPACECHAR = CHAR_CONSTANT(CharType, ' ');
const CharType TABCHAR =  CHAR_CONSTANT(CharType, '\t');
const CharType DQUOTECHAR = CHAR_CONSTANT(CharType, '\"');
const CharType SLASHCHAR =  CHAR_CONSTANT(CharType, '\\');
const CharType* STRING = STRING_CONSTANT(CharType, "String");

Abracadabra. The problem is solved without any code duplication or performance issues.

Article Source Code

The source code for this article can be found on SullivanAndKey.com. The relevant files may be found at the following locations.

  • widen.h: Contains the source of widenChar and widenString.
  • ConstantOfType.h: Contains the source of CharConstantOfType and StringConstantOfType.
  • CmdLineToArgv.h: The header file for the CommandLineToArgVector function.
  • CmdLineToArgv.cpp: The source code for the CommandLineToArgVector function.

 

Cross Platform Conversion Between string and wstring

Cross Platform Conversion Between string and wstring

Ben Key

October 31, 2013; November 09, 2018

 


This article describes a cross platform method of converting between a STL string and a STL wstring. The technique described in this article does not make use of any external libraries. Nor does it make use of any operating system specific APIs. It uses only features that are part of the standard template library.

Problem Description

The Standard Template Library provides the basic_string template class to represents a sequence of characters. It supports all the usual operations of a sequence and standard string operations such as search and concatenation.

There are two common specializations of the basic_string template class. They are string, which is a typedef for basic_string<char>, and wstring, which is a typedef for basic_string<wchar_t>. In addition there are two specializations of the basic_string template class that are new to C++11. The new specializations of the basic_string template class are u16string, which is a typedef for basic_string<char16_t>, and u32string, which is a typedef for basic_string<char32_t>.

A question that is often asked on the Internet is how to convert between string and wstring. This may be necessary if calling an API function that is designed for one type of string from code that uses a different type of string. For example your code might use wstring objects and you may need to call an API function that uses string objects. In this case you will need to convert between a wstring to a string before calling the API function.

Unfortunately the Standard Template Library does not provide a simple means of doing this conversion. As a result, this is a commonly asked question on the Internet. A Google search for convert between string and wstring returns about 76,300 results. Unfortunately many of the answers that are provided are incorrect.

The Incorrect Answer

The most common incorrect answer found on the Internet can be found in the article Convert std::string to std::wstring on Mijalko.com. The solution provided in that article is as follows.


// std::string -> std::wstring
std::string s("string");
std::wstring ws;
ws.assign(s.begin(), s.end());

// std::wstring -> std::string
std::wstring ws(L"wstring");
std::string s;
s.assign(ws.begin(), ws.end());

The problem is that this solution compiles and runs and for the simple string constants used in this example code, appears to yield the correct results. Many sub standard computer programmers will assume that since it works for these very simple strings they have found the correct solution. Then they translate their application to other languages besides English and they are extremely surprised to discover that their application is riddled with bugs that only affect their non English speaking customers!

The problem is that this solution only works correctly for ASCII characters in the range of 0 to 127. If your strings contain even a single character with a numerical value greater than 127, this simple solution will yield incorrect results. In other words, this simple solution will yield incorrect results if your strings contain any characters other than a through z, A through Z, the numerals 0 through 9, and a few punctuation marks.

This means that any application that uses this technique will not support Chinese. It will not even properly support Spanish since the Spanish alphabet contains several characters that are outside of the ASCII character set such as ch (ce hache), ll (elle), and ñ (eñe).

The Correct Solution

This article describes a cross platform solution for this problem that has the following characteristics.

The Functions


namespace yekneb
{

template<typename Target, typename Source>
inline Target string_cast(const Source& source)
{
    return source;
}

template<>
inline std::wstring string_cast(const std::string& source)
{
    std::locale loc;
    return ::yekneb::detail::string_cast::s2w(source, loc);
}

template<>
inline std::string string_cast(const std::wstring& source)
{
    std::locale loc;
    return ::yekneb::detail::string_cast::w2s(source, loc);
}

}

Three additional functions are provided that allow the user to specify the locale parameter.

The functions are used as follows.


std::wstring wIn(L"Hello World! It is truly a wonderful day to be alive.");
std::string sIn("Hello World! It is truly a wonderful day to be alive.");
std::string sOut = yekneb::string_cast<std::string>(wIn);
std::wstring wOut = yekneb::string_cast<std::wstring>(sIn);

The details

The functions s2w and w2s are implemented using the following features of the STL.

The implementation of these functions is as follows.


namespace yekneb
{

namespace detail
{
namespace string_cast
{

inline std::string w2s(const std::wstring& ws, const std::locale& loc)
{
    typedef std::codecvt<wchar_t, char, std::mbstate_t> converter_type;
    typedef std::ctype<wchar_t> wchar_facet;
    std::string return_value;
    if (ws.empty())
    {
        return "";
    }
    const wchar_t* from = ws.c_str();
    size_t len = ws.length();
    size_t converterMaxLength = 6;
    size_t vectorSize = ((len + 6) * converterMaxLength);
    if (std::has_facet<converter_type>(loc))
    {
        const converter_type& converter = std::use_facet<converter_type>(loc);
        if (converter.always_noconv())
        {
            converterMaxLength = converter.max_length();
            if (converterMaxLength != 6)
            {
                vectorSize = ((len + 6) * converterMaxLength);
            }
            std::mbstate_t state;
            const wchar_t* from_next = nullptr;
            std::vector<char> to(vectorSize, 0);
            std::vector<char>::pointer toPtr = to.data();
            std::vector<char>::pointer to_next = nullptr;
            const converter_type::result result = converter.out(
                state, from, from + len, from_next,
                toPtr, toPtr + vectorSize, to_next);
            if (
              (converter_type::ok == result || converter_type::noconv == result)
              && 0 != toPtr[0]
              )
            {
              return_value.assign(toPtr, to_next);
            }
        }
    }
    if (return_value.empty() && std::has_facet<wchar_facet>(loc))
    {
        std::vector<char> to(vectorSize, 0);
        std::vector<char>::pointer toPtr = to.data();
        const wchar_facet& facet = std::use_facet<wchar_facet>(loc);
        if (facet.narrow(from, from + len, '?', toPtr) != 0)
        {
            return_value = toPtr;
        }
    }
    return return_value;
}

inline std::wstring s2w(const std::string& s, const std::locale& loc)
{
    typedef std::ctype<wchar_t> wchar_facet;
    std::wstring return_value;
    if (s.empty())
    {
        return L"";
    }
    if (std::has_facet<wchar_facet>(loc))
    {
        std::vector<wchar_t> to(s.size() + 2, 0);
        std::vector<wchar_t>::pointer toPtr = to.data();
        const wchar_facet& facet = std::use_facet<wchar_facet>(loc);
        if (facet.widen(s.c_str(), s.c_str() + s.size(), toPtr) != 0)
        {
            return_value = to.data();
        }
    }
    return return_value;
}

}
}

}

A GNU/Linux Specific Bug

The above functions function well on Microsoft Windows and MAC OS X. However they do not work as well on GNU/Linux. Specifically the w2s function will fail with one of two errors, depending on whether debugging is enabled.

If debugging is enabled the error will be a double free or corruption error that occurs when the output buffer is being deallocated. The error occurs do to the fact that the call to converter.out will cause the output buffer to be corrupted and far more bytes will be written to the output buffer than the number of bytes that were allocated for the buffer, even if you over allocate the output buffer by a power of 2.

If debugging is not enabled the error will be as follows.

../iconv/loop.c:448: internal_utf8_loop_single: Assertion `inptr - bytebuf > (state->__count & 7)' failed.

These bugs bug only occur when the active locale uses UTF-8.

I have been unable to resolve this issue using just the STL. However, the Boost C++ Libraries found at www.boost.org provide an acceptable solution, the boost::locale::conv::utf_to_utf function. If using Boost is an option for you this problem can be resolved as follows.

First, add the IsUTF8 function to the ::yekneb::detail::string_cast namespace. This function may be used to determine whether or not a given locale uses UTF-8 and thus whether or not the w2s function should use the boost::locale::conv::utf_to_utf function.

The source code for the IsUTF8 function is as follows.


inline bool IsUTF8(const std::locale &loc)
{
    std::string locName = loc.name();
    if (! locName.empty() && std::string::npos != locName.find("UTF-8"))
    {
        return true;
    }
    return false;
}

Then simply add the following code to the w2s function just after the if (ws.empty()) code block.


if (IsUTF8(loc))
{
    return_value = boost::locale::conv::utf_to_utf<char>(ws);
    if (! return_value.empty())
    {
        return return_value;
    }
}

For consistency a similar code block should be added to the s2w function as well.

Article Source Code

The complete source code for this article can be found on SullivanAndKey.com in the header file StringCast.h. You can also see the code in action on ideone.com.


 

Variable Expansion in Strings

Variable Expansion in Strings

Ben Key:

December 6, 2013; November 7, 2018

 


A common task in C and C++ is to build a string out of a template string containing variable placeholders, often called format specifiers, and additional data. This article describes several options that are available for solving this problem and introduces two versions of an ExpandVars function as an alternative to those solutions.

Problem Description

The simplest description of the problem can be best described via an expansion of the classic Hello World program that is so often the first program one learns to write when learning a new programming language.

The simplest implementation of Hello World in C++ is as follows.


#include <iostream>
 
int main()
{
    std::cout << "Hello, world!\n";
    return 0;
}

Source: Variable Expansion in Strings – Example 1


What if you wanted to modify this program to first ask you for your name and then display a more personal greeting? One way to do this is as follows.


#include <iostream>
#include <string>
 
int main()
{
    std::cout << "What is your name?\n";
    std::string name;
    std::getline(std::cin, name);
    std::cout << "Hello, " << name << "!\n";
    return 0;
}

Source: Variable Expansion in Strings – Example 2


The problem with this approach is that it is not very extensible. It can also be very unwieldy when you have multiple variables that you need to print out.

For example, imagine you have the following person structure and you want to display a message containing all of the fields in the person structure.


struct person
{
    std::string firstName;
    std::string middleName;
    std::string lastName;
    std::string streetAddress1;
    std::string streetAddress2;
    std::string city;
    std::string state;
    std::string zip;
};

Source: Variable Expansion in Strings – Example 3


One possible explanation of a function to display a message containing all of the fields in the person structure is as follows.


void PrintPersonWithStream(const person& p)
{
    std::cout
        << "First Name: " << p.firstName << "\n"
        << "Middle Name: " << p.middleName << "\n"
        << "Last Name: " << p.lastName << "\n"
        << "Street Address 1: " << p.streetAddress1 << "\n"
        << "Street Address 2: " << p.streetAddress2 << "\n"
        << "City: " << p.city << "\n"
        << "State: " << p.state << "\n"
        << "Zip: " << p.zip << "\n";
}

Source: Variable Expansion in Strings – Example 3


As you can see, this function is rather unwieldy. It would be far simpler to be able to write something like the following.


void PrintPerson(const person& p)
{
    const char FormatString[] = R"(
First Name: {FormatSpecifier}
Middle Name: {FormatSpecifier}
Last Name: {FormatSpecifier}
Street Address 1: {FormatSpecifier}
Street Address 2: {FormatSpecifier}
City: {FormatSpecifier}
State: {FormatSpecifier}
Zip: {FormatSpecifier}
)";
    std::string message = SomeFormatFunction(
        FormatString,
        p.firstName, p.middleName, p.lastName,
        p.streetAddress1, p.streetAddress2,
        p.city, p.state, p.zip);
    std::cout << message << "\n";
}

In the above example {FormatSpecifier} will be replaced with a bit of text that causes the text of the appropriate variable to be inserted at the appropriate place in the final string. It will vary depending on the solution you use.

The benefit of this type of solution is that it is far less verbose and it is far less work to change the order of variables in the final output and to add a variable.

You might ask why being able to change the order of variable counts. A simple answer is if your program supports several languages and you need to change the order of items such as dates to account for standards used by a given language.

Using the Standard C Library Functions printf or sprintf

One option is to simply make use of the Standard C library functions printf, or if you need to store the output in a string, sprintf. The printf function writes formatted data to stdout. The sprintf function writes formatted data to a string. These functions can be used as follows.

First, add the following function to the person structure.


static std::string GetPrintfFormatString()
{
    static const char FormatString[] = R"(
First Name: %s
Middle Name: %s
Last Name: %s
Street Address 1: %s
Street Address 2: %s
City: %s
State: %s
Zip: %s
)";
    return FormatString;
}

Source: Variable Expansion in Strings – Example 4


Then the functions can be defined as follows.


void PrintPersonWithPrintf(const person& p)
{
    ::printf(
        p.GetPrintfFormatString().c_str(),
        p.firstName.c_str(), p.middleName.c_str(), p.lastName.c_str(),
        p.streetAddress1.c_str(), p.streetAddress2.c_str(),
        p.city.c_str(), p.state.c_str(), p.zip.c_str());
}
 
void PrintPersonWithSPrintf(const person& p)
{
    std::string formatString = p.GetPrintfFormatString();
    size_t outputLen = formatString.length() + p.firstName.length()
        + p.middleName.length() + p.lastName.length()
        + p.streetAddress1.length() + p.streetAddress2.length()
        + p.city.length() + p.state.length() + p.zip.length()
        + 20;
    std::vector<char> buffer(outputLen, 0);
    ::sprintf(
        buffer.data(), formatString.c_str(),
        p.firstName.c_str(), p.middleName.c_str(), p.lastName.c_str(),
        p.streetAddress1.c_str(), p.streetAddress2.c_str(),
        p.city.c_str(), p.state.c_str(), p.zip.c_str());
    std::string out = buffer.data();
    std::cout << out;
}

Source: Variable Expansion in Strings – Example 4


Limitations of sprintf

One limitation of using the sprintf function is that it is not very flexible for international applications. Often the order of words differ from one language to another. One often discussed example is a time and date string.

For example, in the United States date strings are written as {Month}/{Day}/{Year} while in France date strings are written as {Day}/{Month}/{Year} and in Japan date strings are written as {Year}/{Day}/{Month}. There are many other instance in which word order varies from language to language. For more information refer to the Word order article on Wikipedia, [The origin and evolution of word order][], and The Typology of the Word Order of Languages.

One problem with the sprintf function is that it is not possible to change the order of words in the final output by simply changing the order of words in the format string. That is due to the fact that the order of parameters in the code would need to be changed as well.

One solution to this problem is the use of positional specifiers for format strings.

Positional Specifiers for Format Strings

POSIX compatible systems implement an extension to the printf family of functions to add support for positional specifiers for format strings. This extension allows the conversion specifier character % to be is replaced by the sequence “%n$”, where n is a decimal integer in the range [1, {NL_ARGMAX}], giving the position of the argument in the argument list. For more information see the following resources.

The problem for this solution is that this is not universally supported. For example, on Microsoft Windows, the printf family of functions does not support positional specifiers for format strings. Instead this functionality is supported in the printf_p family of functions: see printf_p Positional Parameters and _sprintf_p, _sprintf_p_l, _swprintf_p, _swprintf_p_l. This makes writing cross platform code unnecessarily difficult.

The following code demonstrates the use of positional specifiers for format strings to write a function that will properly format a date string for the United States, France, and Japan.


std::string GetDateFormatString(const std::string& langCode)
{
    if (
       0 == langCode.compare(0, 2, "en")
       || 0 == langCode.compare(0, 2, "EN")
       )
    {
        return std::string("%1$i/%2$i/%3$i");
    }
    else if (
       0 == langCode.compare(0, 2, "fr")
       || 0 == langCode.compare(0, 2, "FR")
       )
    {
        return std::string("%2$i/%1$i/%3$i");
    }
    else if (
       0 == langCode.compare(0, 2, "ja")
       || 0 == langCode.compare(0, 2, "JA")
       || 0 == langCode.compare(0, 2, "jp")
       || 0 == langCode.compare(0, 2, "JP")
       )
    {
        return std::string("%3$i/%2$i/%1$i");
    }
    return std::string("%1$i/%2$i/%3$i");
}
 
std::string GetDateString(
    const std::string& langCode,
    int month, int day, int year)
{
    std::string fmt = GetDateFormatString(langCode);
    std::array<char, 32> buffer;
    buffer.fill(0);
#if defined(_WIN32)
    ::_sprintf_p(buffer.data(), 32, fmt.c_str(), month, day, year);
#else
    ::sprintf(buffer.data(), fmt.c_str(), month, day, year);
#endif
    std::string ret = buffer.data();
    return ret;
}

Source: Variable Expansion in Strings – Example 5


Note that the there is one major drawback of the above GetDateString function, the presence of that nasty #if/#else/#endif block. This is far from ideal. Unfortunately, due to the fact that the _sprintf_p function expects an additional sizeOfBuffer parameter. Therefore you cannot simply do the following.


#if !defined(_WIN32)
#  define _sprintf_p sprintf
#endif

std::string GetDateString(
    const std::string &langCode,
    int month, int day, int year)
{
    std::string fmt = GetDateFormatString(langCode);
    std::array<char, 32> buffer;
    buffer.fill(0);
    // This will not compile on non Windows systems due to the extra parameter.
    _sprintf_p(buffer.data(), 32, fmt.c_str(), month, day, year);
    std::string ret = buffer.data();
    return ret;
}

The following will work as an acceptable alternative, however.


#if defined(_WIN32)
#  define sprintfp _sprintf_p
#else
#  define sprintfp snprintf
#endif

std::string GetDateString(
    const std::string &langCode,
    int month, int day, int year)
{
    std::string fmt = GetDateFormatString(langCode);
    std::array<char, 32> buffer;
    buffer.fill(0);
    sprintfp(buffer.data(), 32, fmt.c_str(), month, day, year);
    std::string ret = buffer.data();
    return ret;
}

Source: Variable Expansion in Strings – Example 6


This leaves one problem that all of the solutions I have discussed so far unsolved. This function uses C style strings. That is, the first parameter of _sprintf_p is expected to be a pre-allocated char array. It does not natively make use of the C++ basic_string class.

The Boost Format library

The Boost C++ Libraries are a collection of free peer-reviewed portable C++ source libraries that work well with the C++ Standard Library and enhance the capabilities of the C++ Standard Library. In fact, some of the features of the C++ Standard Library were first implemented in the Boost C++ Libraries and the Boost C++ Libraries are designed so that they are suitable for eventual standardization.

One of the components of Boost is The Boost Format library. The Boost home page describes The Boost Format library as follows.

The format library provides a class for formatting arguments according to a format-string, as does printf, but with two major differences:

  • format sends the arguments to an internal stream, and so is entirely type-safe and naturally supports all user-defined types.
  • The ellipsis (…) can not be used correctly in the strongly typed context of format, and thus the function call with arbitrary arguments is replaced by successive calls to an argument feeding operator%

The format specification strings used by the Boost Format library use the Unix98 Open-group printf precise syntax. Further information on the format specification strings used by the Boost Format library can be found in the Boost printf format specifications section of the Boost Format library documentation. Note that these are essentially the same format specification strings that are used by the _sprintf_p function. As a result, the GetDateFormatString function can be used with The Boost Format library.

The following function shows how this can be done.


std::string GetDateStringBoost(
    const std::string &langCode, int month, int day, int year)
{
    std::string fmt = GetDateFormatString(langCode);
    std::string ret = boost::str(boost::format(fmt) % month % day % year);
    return ret;
}

Source: Variable Expansion in Strings – Example 7


Self Documenting Format Specification Strings

One problem with the printf style format specification strings is that they require some form of supporting documentation to indicate which part of the format specification string corresponds to which variable. For example, in order for the GetDateFormatString function to be considered complete, a comment should be added to specify that the %1$ component corresponds to the month, the %2$ component corresponds to the day, and the %3$ component corresponds to the year.

It would be idea if this documentation was an inherent part of the format specification string. Consider the following syntax for a format string: “\((month)/\)(day)/$(year).” In this string their is no need for supporting documentation to indicate the meaning of each component of the format specification string.

This technique is commonly referred to as String interpolation or variable interpolation, variable substitution, or variable expansion. Some programming languages have this functionality built in.

For example, the Python programming language supports the Literal String Interpolation feature since Python 3.6. This makes the following possible.


apples = 4
print(f"I have {apples} apples")

Another example is in the C# programming language. C# 6 added the interpolated string feature.


string name = "Mark";
var date = DateTime.Now;
Console.WriteLine($"Hello, {name}! Today is {date.DayOfWeek}, it's {date:HH:mm} now.");

The ExpandVars Function

The YekNeb C++ Code snippets library provides two versions of the ExpandVars function, which provides string interpolation functionality for C++. One version of the function uses nothing beyond the STL. Another version of the function uses the Boost Xpressive library. Both versions of the function return a string in which the variables are expanded based on the values specified in either an environment map or the environment variables. The following formats are supported for variable names.

  • %VarName%
  • %(VarName)
  • %[VarName]
  • %{VarName}
  • $(VarName)
  • $[VarName]
  • ${VarName}
  • #(VarName)
  • #[VarName]
  • #{VarName}

Bash style variable names in the form of $VarName are not supported.

The variable names used by the ExpandVars function may contain word characters, space characters, the ( character, and the ) character. Note that if the variable includes either the ( character or the ) character you should not use the %(VarName) or $(VarName) syntax.

The following is a simplified version of the STL only ExpandVars function.


bool FindVariableString(
    const std::string &str,
    const std::string::size_type pos,
    std::string::size_type &beginVarStringPos,
    std::string::size_type &endVarStringPos,
    std::string::size_type &beginVarNamePos,
    std::string::size_type &endVarNamePos)
{
    const char *TestString = "%$#";
    const char PercentSign = '%';
    const char LeftParenthesis = '(';
    const char LeftSquareBracket = '[';
    const char LeftCurlyBracket = '{';
    const char RightParenthesis = ')';
    const char RightSquareBracket = ']';
    const char RightCurlyBracket = '}';
    beginVarStringPos = std::string::npos;
    endVarStringPos = std::string::npos;
    beginVarNamePos = std::string::npos;
    endVarNamePos = std::string::npos;
    if (str.empty())
    {
        return false;
    }
    beginVarStringPos = str.find_first_of(TestString, pos);
    if (std::string::npos == beginVarStringPos)
    {
        return false;
    }
    if (beginVarStringPos >= str.length() - 1)
    {
        return false;
    }
    char ch = str[beginVarStringPos];
    char ch1 = str[beginVarStringPos + 1];
    if (
       PercentSign == ch
       && LeftParenthesis != ch1 && LeftSquareBracket != ch1
       && LeftCurlyBracket != ch1
       )
    {
        beginVarNamePos = beginVarStringPos + 1;
        endVarStringPos = str.find(PercentSign, beginVarNamePos);
        if (std::string::npos == endVarStringPos)
        {
            return false;
        }
    }
    else if (
       LeftParenthesis != ch1 && LeftSquareBracket != ch1
       && LeftCurlyBracket != ch1
       )
    {
        return false;
    }
    else
    {
        beginVarNamePos = beginVarStringPos + 2;
        char closeChar = 0;
        if (LeftParenthesis == ch1)
        {
            closeChar = RightParenthesis;
        }
        else if (LeftSquareBracket == ch1)
        {
            closeChar = RightSquareBracket;
        }
        else if (LeftCurlyBracket == ch1)
        {
            closeChar = RightCurlyBracket;
        }
        endVarStringPos = str.find(closeChar, beginVarNamePos);
        if (std::string::npos == endVarStringPos)
        {
            return false;
        }
    }
    endVarNamePos = endVarStringPos - 1;
    return true;
}
 
bool StringContainsVariableStrings(const std::string &str)
{
    std::string::size_type beginVarStringPos = 0;
    std::string::size_type endVarStringPos = 0;
    std::string::size_type beginVarNamePos = 0;
    std::string::size_type endVarNamePos = 0;
    bool ret = FindVariableString(str, 0, beginVarStringPos, endVarStringPos, beginVarNamePos, endVarNamePos);
    return ret;
}
 
std::string GetVariableValue(
    const std::string &varName,
    const std::map<std::string, std::string> &env,
    bool &fromEnvMap, bool &valueContainsVariableStrings)
{
    typedef std::map<std::string, std::string> my_map;
    fromEnvMap = false;
    valueContainsVariableStrings = false;
    std::string ret;
    my_map::const_iterator itFind = env.find(varName);
    if (itFind != env.end())
    {
        ret = (*itFind).second;
        if (!ret.empty())
        {
            fromEnvMap = true;
            valueContainsVariableStrings = StringContainsVariableStrings(ret);
        }
    }
    if (ret.empty())
    {
        ret = ::getenv(varName.c_str());
    }
    return ret;
}
 
std::string ExpandVars(
    const std::string &original,
    const std::map<std::string, std::string> &env)
{
    std::string ret = original;
    if (original.empty())
    {
        return ret;
    }
    bool foundVar = false;
    std::string::size_type pos = 0;
    do
    {
        std::string::size_type beginVarStringPos = 0;
        std::string::size_type endVarStringPos = 0;
        std::string::size_type beginVarNamePos = 0;
        std::string::size_type endVarNamePos = 0;
        foundVar = FindVariableString(ret, pos, beginVarStringPos, endVarStringPos, beginVarNamePos, endVarNamePos);
        if (foundVar)
        {
            std::string::size_type varStringLen = endVarStringPos - beginVarStringPos + 1;
            std::string varString = ret.substr(beginVarStringPos, varStringLen);
            std::string::size_type varNameLen = endVarNamePos - beginVarNamePos + 1;
            std::string varName = ret.substr(beginVarNamePos, varNameLen);
            bool fromEnvMap;
            bool valueContainsVariableStrings;
            std::string value = GetVariableValue(varName, env, fromEnvMap, valueContainsVariableStrings);
            if (!value.empty())
            {
                ret = ret.replace(beginVarStringPos, varStringLen, value);
                pos = beginVarStringPos;
            }
            else
            {
                pos = endVarStringPos + 1;
            }
        }
    } while (foundVar);
    return ret;
}

Source: Variable Expansion in Strings – Example 8

The following code demonstrates the use of the ExpandVars function.

std::string GetDateFormatStringExpandVars(const std::string& langCode)
{
    if (
       0 == langCode.compare(0, 2, "en")
       || 0 == langCode.compare(0, 2, "EN")
       )
    {
        return std::string("${month}/${day}/${year}");
    }
    else if (
       0 == langCode.compare(0, 2, "fr")
       || 0 == langCode.compare(0, 2, "FR")
       )
    {
        return std::string("${day}/${month}/${year}");
    }
    else if (
       0 == langCode.compare(0, 2, "ja")
       || 0 == langCode.compare(0, 2, "JA")
       || 0 == langCode.compare(0, 2, "jp")
       || 0 == langCode.compare(0, 2, "JP")
       )
    {
        return std::string("${year}/${day}/${month}");
    }
    return std::string("${month}/${day}/${year}");
}
 
std::string GetDateStringExpandVars(
    const std::string &langCode, int month, int day, int year)
{
    std::string fmt = GetDateFormatStringExpandVars(langCode);
    std::map<std::string,std::string> env{
        {"month", std::to_string(month)},
        {"day", std::to_string(day)},
        {"year", std::to_string(year)}
    };
    std::string ret = ExpandVars(fmt, env);
    return ret;
}

Source: Variable Expansion in Strings – Example 8


The following is a simplified version of a version of the ExpandVars function that uses the Boost Xpressive regex_replace function.


::boost::xpressive::sregex GetRegex()
{
    namespace xpr = ::boost::xpressive;
    xpr::sregex ret =
        "%" >> (xpr::s1 = +(xpr::_w | xpr::_s | "(" | ")")) >> '%'
        | "%(" >> (xpr::s1 = +(xpr::_w | xpr::_s)) >> ')'
        | "%[" >> (xpr::s1 = +(xpr::_w | xpr::_s | "(" | ")")) >> ']'
        | "%{" >> (xpr::s1 = +(xpr::_w | xpr::_s | "(" | ")")) >> '}'
        | "$(" >> (xpr::s1 = +(xpr::_w | xpr::_s)) >> ')'
        | "$[" >> (xpr::s1 = +(xpr::_w | xpr::_s | "(" | ")")) >> ']'
        | "${" >> (xpr::s1 = +(xpr::_w | xpr::_s | "(" | ")")) >> '}'
        | "#(" >> (xpr::s1 = +(xpr::_w | xpr::_s)) >> ')'
        | "#[" >> (xpr::s1 = +(xpr::_w | xpr::_s | "(" | ")")) >> ']'
        | "#{" >> (xpr::s1 = +(xpr::_w | xpr::_s | "(" | ")")) >> '}';
    return ret;
}

struct string_formatter
{
    typedef std::map<std::string, std::string> env_map;
    env_map env;
    mutable bool valueContainsVariables;
    string_formatter()
    {
        valueContainsVariables = false;
    }
    template<typename Out>
    Out operator()(::boost::xpressive::smatch const& what, Out out) const
    {
        bool fromEnvMap;
        bool valueContainsVariableStrings;
        std::string value = GetVariableValue(
            what.str(1), env, fromEnvMap, valueContainsVariableStrings);
        if (fromEnvMap && !value.empty() && valueContainsVariableStrings)
        {
            valueContainsVariables = true;
        }
        if (value.empty())
        {
            value = what[0];
        }
        if (!value.empty())
        {
            out = std::copy(value.begin(), value.end(), out);
        }
        return out;
    }
};

std::string ExpandVarsR(
    const std::string &original,
    const std::map<std::string, std::string> &env)
{
    std::string ret = original;
    if (original.empty())
    {
        return ret;
    }
    string_formatter fmt;
    fmt.env = env;
    fmt.valueContainsVariables = false;
    ::boost::xpressive::sregex envar = GetRegex();
    ret = ::boost::xpressive::regex_replace(original, envar, fmt);
    if (fmt.valueContainsVariables)
    {
        std::string newValue;
        std::string prevValue = ret;
        do
        {
            fmt.valueContainsVariables = false;
            newValue = ::boost::xpressive::regex_replace(prevValue, envar, fmt);
            if (0 == prevValue.compare(newValue))
            {
                break;
            }
            prevValue.erase();
            prevValue = newValue;
        }
        while (fmt.valueContainsVariables);
        if (0 != ret.compare(newValue))
        {
            ret = newValue;
        }
    }
    return ret;
}

Source: Variable Expansion in Strings – Example 9


The source code of the full version of the STL only implementation of the ExpandVars function can be found in ExpandVars.h and ExpandVars.cpp. The source code of the Boost implementation of the ExpandVars function can be found in boost/ExpandVars.h and boost/ExpandVars.cpp.

 

Musings on the Formatting of C and C++ Code

Ben Key: Ben.Key@YekNeb.com

October 4, 2013; October 26, 2018

 

I was watching the video The Care and Feeding of C++’s Dragons. I found it to be very interesting. I especially found the CLang based program that reformats code to be promising. However, there were some things about the tool that I find disturbing. It seems to use some formatting patterns that I think are huge mistakes.

To illustrate, some of the sample code looks like the following.

int functionName(int a, int b, int c,
                 int d, int e, int f,
                 int g, int h, int i) const {
  /* Code omitted. */
  if (test()) {
    /* Code omitted. */
  }
}

There are two problems that I see with formatting code in this way. The first is that on the line where the list of variables is continued, there is a great deal of what I see as entirely unnecessary white space. In my opinion it should be as follows.

int functionName(int a, int b, int c,
  int d, int e, int f,
  int g, int h, int i) const {

Or, my personal preference would be the following.

int functionName(
  int a, int b, int c,
  int d, int e, int f,
  int g, int h, int i) const {

My reasoning is quite simple.

Examine what happens when the name of the function is changed to something longer, as follows, and then it is reformated. The code becomes as follows.

int aMuchLongerFunctionName(int a, int b, int c
                            int d, int e, int f,
                            int g, int h, int i) const {

If you do a diff of the original version of the code and the new code, it will show that three lines have changed instead of showing that only one line has changed. I am aware of the fact that many diff tools have an ignore white space option and that if this option is set it will only show one line as having changed. However, not all diff tools have that option. In addition, some companies and organizations have a strict policy that every line of code that changes must be changed for a purpose associated with a given task. And these companies do not accept changes that are only related to reformatting as being associated with a given task. In essence, if the changed line does not affect the functionality of the code, it is not an acceptable change. And these organizations will deliberately not turn on the ignore white space option and will turn a deaf ear to the argument that they should just enable that option (can you tell that I am speaking from experience?).

If you are in such a situation and you change the name of a function that initially is formatted with the parameter list aligned with the end of the function name and you adhere to a strict “every changed line must have a functional purpose” rule you will inevitably end up with the following.

int aMuchLongerFunctionName(int a, int b, int c
                 int d, int e, int f,
                 int g, int h, int i) const {

This just looks wrong!

There is also another reason for not aligning the parameters with the end of the function name. Consider the following.

int aVeryLongFunctionNameThatGoesBeyondTheEdgeOfTheScreen(int a, int b, int c,
                                                          int d, int e, int f,
                                                          int g, int h, int i) const {

In this case, you cannot see the parameters at all without wasting your time scrolling across the screen.

If you always begin the parameter list on its own line that is indented one level deep as follows, you would not ever have to scroll the screen just to see the parameter list.

int aVeryLongFunctionNameThatGoesBeyondTheEdgeOfTheScreen(
  int a, int b, int c,
  int d, int e, int f,
  int g, int h, int i) const {

The second issue I have is with putting braces at the end of the line. In C and C++ braces are optional for some statements such as if. And lets face the facts, C and C++ is often inconsistently indented. Putting braces at the end leads to more work in the following scenario.

if (aVeryLongTestThatGoesPastTheEdgeOfTheScreen()) {
  /*
       Thousands
 of
         inconsistently indented
    lines of
 code /*
}

Putting the brace at the end of the if line forces someone who is reading the code to hit the end key to determine how much code will only be called if the condition is true, one line or thousands of lines, when they might not give a damned about seeing the end of the test because the beginning of it is enough to tell them whether or not the condition can be true in the scenario they are working on. What if the person knows that in the case they are working on, the function “aVeryLongTestThatGoesPastTheEdgeOfTheScreen” will return false. They really do not need to see the end of the test in this case except to find out how many lines they need to skip past in order to get to code that is relevant to their task. Why not just put the brace on a line by itself and make everyone’s life so much easier? Why force someone to hit the end key just so they can answer the following question. How many lines do I need to skip to get to code that is relevant to my task?

Until C and C++ do as they did in the Go language and make the braces mandatory, I believe braces should never be at the end of the line.

In Go, where the braces are mandatory, it does not matter as much to me because I know that if the code compiles the brace is there and I do not care if I cannot see it. But in C and C++, I do not want you to force me to find and hit the end key just so I can tell where your if statement ends. Of course, that does not mean that I think that the decision to put the brace at the end was a good one for Go. I often use a paren matching feature to skip past the irrelevant code in the scenario I have described. That requires that the caret be on the opening paren. In Go I need to hit the end key anyway just to get the caret on the brace so I can use the paren matching feature to skip past code I do not care about. Why? If the brace were on a line by itself, I do not need to locate and hit the end key. I can just arrow down to the brace line and use the paren matching feature.

I know that these arguments are only relevant to the placement of braces for conditional statements and that they are not relevant to the placement of braces at the beginning of functions. However, I still feel that the opening brace of a function should be on a line by itself for the sake of consistency.

I cannot believe other people have adopted code formatting patterns that to me are so obviously mistakes. Is there something I am missing that makes my arguments invalid?

And before you say, “just hit the end key, it is not that hard,” consider the fact that some people are hunt an peck typists. For some people, any extra key they need to hunt for unnecessarily is an aggravation that interrupts their work flow. I am certain that for some people who are touch typists, hitting one additional key is no big deal, but for hunt and peck typists, it can be.

I for one am a hunt and peck typist despite the fact that I began using computers in 1985 and for me finding the end key just to find out how many lines of code will only get called in the condition is true case is enough of a disruption that I find it to be extremely annoying.

When I first wrote this article back in 2013, I was not aware of the many options for customizing the behavior of clang-format.

Fortunately, you can easily customize the behavior of clang-format. There are numerous Clang-Format Style Options available. For example, you can instruct clang-format to “always break after an open bracket, if the parameters don’t fit on a single line” by setting AlignAfterOpenBracket to AlwaysBreak.

When you use clang-format to format a file it will search for a “.clang-format file located in one of the parent directories of the source file” and load the various formatting options from there. Clang-format also has a number of predefined coding styles to choose from: LLVM, Google, Chromium, Mozilla, and WebKit. You can use the -fallback-style and -style command line arguments to specify the coding style you wish to use. For more information see the ClangFormat manual.

I have begun using clang-format for my own open source projects, and I am pleased with the results. If you are interested, you can take a look at my SnKOpen .clang-format file.

There are various websites that will help you to generate the perfect .clang-format file for your project. One of the best is the clang-format configurator. The Unformat project, which generates a .clang-format file from example codebase may also be worth investigating.