Showing posts with label Size/type mismatch. Show all posts
Showing posts with label Size/type mismatch. Show all posts

Monday, November 26, 2012

... and that's the way it is!

One of the errors reported by Coverity on our code was this:
Logical vs. bitwise operator (CONSTANT_EXPRESSION_RESULT)
~sense is always 1/true regardless of the values of its operand. This occurs as the bool operand of assignment. Did you intend to use '!' rather than '~'


The code it referred to was something like:

bool sense;
...
if(~sense)
...


Note: If we write:
bool result = ~sense;
instead, we get the same result and same error.

Now this was quite surprising. The size of bool is 1-bit, and for years now, we have understood that though it is not a good practice to use bit-negation instead of logical negation for a one-bit quantity, the result is same (it's obviously wrong for multi-bit).
Refusing to believe coverity, we wrote a small CPP code:
======================

  bool sense = true;

  cout << "Size: " << sizeof(sense) << endl;

  if(~sense)
    cout << "Always true" << endl;
  else
    cout << "Can be false" << endl;

  bool result = ~sense;
  if(result)
    cout << "Always true" << endl;
  else
    cout << "Can be false" << endl;


======================
 


The result was totally unexpected, and stumped three of us who saw it:

Size: 1
Always true
Always true


Of course, any variable will be aligned to the word (8-bit boundary), but we expected any 0-padding to take place after the negation operation, and not before.

Our understanding that the bit-negation and logical negation yield same result for single bit, has perhaps resulted from years of conditioning of wroking with Verilog. It appears that C/C++ does not work that way. and we just have to accept it.

Wednesday, June 27, 2007

My 64-bit porting experiences - VI

6. Excessive code-reuse

We have always been instructed to reuse code, as it makes the code more maintainable. But it can sometimes be taken too far (too much of a good thing ?!).

6.1 Objects of different sizes in a union

Class A{
int getVal {return bVal;}
exp *getExpr {return bExpr;}
int hasExpr {return isExpr;}
union {
int bVal;
exp *bExpr;
}
int isExpr;
};

The object of this class ‘A’ can have a data-member that is either a constant value (bVal), or an expression (bExpr) if the value is not constant. When the value is not constant, ‘isExpr’ is set to ‘1’, and the user is expected to use the expression returned by ‘getExpr’ call.

In the following piece of code , ‘obj_a’ is an object of class ‘A’:

int tmp = 0;
exp *tmpExpr = NULL;
if (cond)
tmp = 1;
else
tmp = obj_a->getVal();

if(obj_a->hasExpr())
{
if(tmp == 1)
tmpExpr = createExpr(1);
else
tmpExpr = obj_a->getExpr();
}
else
tmpExpr = createExpr(tmp);

Incorrect values in a testcase, in 64-bit mode, were traced to this piece of code. Even though ‘obj_a’ contained expression (which represented a value other than ‘1’), and ‘cond’ was ‘0’, this code inferred the value represented by ‘obj_a’ to be ‘1’.

Reason: ‘obj_a’ represented an expression, so obj_getVal() should not have been used. But it was called, and it returned ‘1’. In 64-bit mode, the members of the union – int and pointer – have sizes 32 and 64 bits respectively. When the integer value was queried, the higher 32 bits were returned, which constituted ‘1’ in integer value [the pointer was 0x1_hhhh_hhhh]. This code worked without problems in 32-bit, because int and pointer have same size, and a pointer cannot be 0x1.


6.2 Overriding functions through implicit casting

This problem was encountered at many places in the code, in different forms.

A hash-table to hash pointers had been implemented, and widely used. The interface functions to insert new records, or find existing ones, accepted char** arguments (and dereferenced them to obtain the pointer). Later on, there was a requirement to hash integer values as well. Instead of implementing a new hash-table (or even writing new interface functions), the previous table and its interface was reused – by passing int* on the actual. When the interface functions dereferenced the pointer passed at the actual, the resulting value was int, and therefore not a valid address.

Sunday, June 17, 2007

My 64-bit porting experiences -V

5. Use of compatible, but incorrect datatypes

An API function, ‘func1’ returns int*, which actually points to an array of two integers. The part of code that uses this function is as follows:

long *lp = func1(…);
int lval = lp[0];
int rval = lp[1];

The results were correct in 32-bit, because int and long are same size. On 64-bit, the return values were incorrect, because of the difference in size of int and long. ‘lp[0]’ represents a 64-bit value, which contains both the int values that are returned by ‘func1’. When it is assigned to ‘lval’, the lower 32 bits are assigned to ‘lval’, which was the value that needed to be assigned to ‘rval’. ‘lp[1]’ reads next 64 bits from memory and assigns the lower 32 bits of it to ‘rval’. So, ‘lval’ gets incorrect value, while ‘rval’ gets junk.

Sunday, June 10, 2007

My 64-bit porting experiences - IV

4. Masking operations
The following may be considered as example of oversight, or assumption on datatype size. I have made this an independent section, because masking operations are frequently used in C code for fast storage and access of data. Masks are usually implemented through macros, and problems in these macros are difficult to debug.

4.1 Storing additional information in a pointer
It may be considered that manipulation of pointer addresses will be very prone to errors. But given the fact that addresses are always word-aligned, the last bit in a pointer value is always 0. To take advantage of this fact, applications sometimes store a boolean property in the last bit of the address, and save runtime memory. As long as it takes care to reset the last bit of such pointers before dereferencing them, the applications are safe. The set and reset operations are usually coded as macros in ‘C’ code, in the interest of runtime.

In the following code snippet, ‘x’ is a pointer under manipulation. M_IS_LIST queries the property on the pointer, M_SET_LIST sets the property on the pointer, and M_GET_LIST retrieves the pointer when before it is de-referenced.
#define M_IS_LIST(x) (((unsigned long)(x)) & 0x1U)
#define M_SET_LIST(x) ((x) |= 0x1U)
#define M_GET_LIST(x) (((unsigned long)(x)) & ~0x1U)

This works for 32-bit mode. In 64-bit, I was getting corrupt pointers at some point in code, which was traced to these macros after considerable effort.

Reason: In its own context, a constant is treated as integer value. So, 0x1U is considered as a 32 bit value. Bit-wise AND with a pointer results in a 32-bit quantity (the smaller of the values being AND’ed).

Solution: To make the mask the same size as the variable:
#define M_IS_LIST(x) (((unsigned long)(x)) & 0x1UL)
#define M_SET_LIST(x) ((x) |= 0x1UL)
#define M_GET_LIST(x) (((unsigned long)(x)) & ~0x1UL)


4.2 Manipulating addresses

Yet another memory corruption was caused by the following code snippet, which purported to provide the next pointer to be processed:

if (((unsigned long)pstr) & 0x3U)
pstr = (char *)(((unsigned long)(pstr + 4)) & ~0x3U);

It was very difficult to understand the function of this simple line of code, due to use of constants, and absence of comments. I cannot stress enough the necessity of documentation in code, especially with tricky calculations like this.

Well, it re-aligns a pointer, based on the understanding that the addresses are aligned at 4 bytes (last two bits are 0). However, in 64-bit mode, the addresses are aligned at 8 bytes (last 3 bits are 0). Therefore, calculations needed to be modified [though I wish it could be more generic], as follows:

#ifdef 64_BIT_BUILD
#define addrMask 0x7UL
#else
#define addrMask 0x3UL
#endif

if (((unsigned long)pstr) & addrMask)
pstr = (char *)(((unsigned long)(pstr + sizeof(long))) & ~addrMask);

Wednesday, May 9, 2007

My 64-bit porting experiences - III

3. Assumptions on datatypes

3.1 Assumptions on size of built-in datatypes

I traced crashes on some platforms to a corruption in hash-tables (though I am still wondering why these testcases did not crash on the rest of the platforms). In the hash-table designated for a certain kind of records, a different kind of record was stored.

The reason, which proved to be very difficult to determine [I spent almost three days on this!], was an assumption on size of built-in datatypes in the hash function (computation for the bin number):

int hash_func(long key)
{
int size = NUM_BINS;

long hi = key >> 16, lo = key;
return (int)(hi ^ lo) % size;
}

The long type argument ‘key’ is derived from a pointer by casting it to long. On 32-bit systems, XOR-ed the higher and lower 16 bits, cast the result to int and determined the bin number by applying modulus function. Since both long and int are 32-bits in size, the cast operation did not affect the size in actual.

In 64-bit mode, when ‘key’ (which actually referred to a pointer address) was cast to int after XOR operation, the 64-bit long variable was truncated to 32 bits, and the 32nd MSB was interpreted as the sign-bit of the int. This MSB was ‘1’, so the resulting integer was a negative number. The modulus function retains the sign of the first operand, therefore the result of the modulus was also a negative number. This resulted in a negative bin number! When the record was stored into this bin, it was stored into the space of another variable, which was incidentally also a hash-table, but of a different kind of record.

The fix was simple:
return (int)((hi ^ lo) % size);

Instead of casting the result of XOR operation to int, I cast the result of modulus to int.


3.2 Assumptions on size and format of user-defined datatypes

Assumption on size and format of user-defined dataypes can be equally fatal. Consider a structure defined as follows:

typedef struct
{
long value;
char *name;
int type;
} mystruct;

In 32-bit architecture, the size of ‘mystruct’ is 12 bytes, as expected by adding the size of the individual members. In 64-bit architecture, however, the size is 24 bytes, rather 20 bytes (because of the 8-byte alignment).

Let n = sizeof(mystruct) [n = 12 on 32-bit and n = 24 on 64-bit].

Now, suppose a sequence of long-pointer-int values is stored in a chunk of memory, and the user wants to populate an array of ‘mystruct’ from this chunk. The user reads first ‘n’ bytes from the chunk and assigns it to an element of the array, reads next ‘n’ bytes from the chunk and assigns it to the next element, and so on. This works fine on 32-bit architecture, because the numbers of bytes read from the memory is exactly the size of the structure. However, on 64-bit architecture, when the user reads first ‘n’ bytes and assigns it to memory, he actually retrieves 4 bytes more than required, misaligning the next set of values to be read.


3.3 Use of non-standard calls on API datastructures

APIs frequently need to provide interface functions to external users to allow them to manipulate the API data. If the API is coded using C++, the internal implementation of the API can be hidden from the users very effectively, using data encapsulation and function overriding capabilities of Object-oriented programming. However, if the API code is written in C, sometimes it becomes inevitable that the internal data-structures are exposed to the user. In this case, if the data-structures need to be changed because of new requirements, the users may be required API is assumed to function in a certain way, instead of using the API interface.

Consider that the API defines a type ‘a_type’, which can be initialized with a value ‘a_null’, or through an interface function ‘a_init’. However, ‘a_null’ is equivalent to NULL, and so the users frequently use NULL to initialize variables of type ‘a_type’. Now, if the definition of ‘a_type’ is changed so that it is no longer compatible with NULL or ‘0’, and the definition of ‘a_null’ and ‘a_init’ is also changed accordingly, the users’ code will still fail to compile, because it was not using the API interface.

Friday, April 27, 2007

My 64-bit porting experiences - II

2. Assumptions on size of predefined datatypes

2.1 Explicit assumption

Ideally, any implementation should not place an assumption on the size of pre-defined datatypes, because these sizes are not standard, and are usually at the discretion of OS implementation provided some boundary conditions are satisfied (e.g., an int shall at least be 16 bits). Most of the code that I was porting, was based on the premise that int, long and pointer are all of the same size, and long long was twice this size. Assertions were placed in the code to ensure this:

assert( sizeof(long) == sizeof(int) );

assert( 2 * sizeof(char *) == sizeof(unsigned long long));

On 64-bit, these assertions were violated. In fact, I had to rearchitecture a significant part of the code.

2.2 Implicit assumption

Implicit assumption often takes the form of casting between different datatypes.

For a hash-table that hashes pointers, the key computation uses the pointer address:

unsigned int key = (unsigned)ptr >> 2;

This specific usage (casting pointer to integer) was treated in a different manner on different platforms. On AIX, there was no warning message; on Linux, the compiler issued a warning; on Solaris, the C compiler gave no warning, while C++ compiler issued an error.

Tuesday, April 17, 2007

My 64-bit porting experiences - I

1. Oversight

1.1 Implicit functions

Following code examples functioned harmlessly while the code was compiled and run on 32-bit, but crashed on 64-bit. Many of them could have been avoided, if compiler and/or lint warnings were attended to.

char *str = (char *)malloc(n * sizeof(char));

This is a perfectly valid way to allocate a string of ‘n’ characters dynamically, and assign it to ‘str’, which is a pointer of the appropriate type. Strangely, on 64-bit, the pointer address returned was 32-bit, and crashed with SIGBUS. Several hits and trials yielded that the ‘C’ file that contained this code did not include stdlib.h, the standard library that provides the definition of ‘malloc’.

ptr = func_ret_ptr(…);

The function, ‘func_ret_ptr’ returns a pointer of a specific type, which is assigned to a pointer variable of the same type. The function comes from another library file, which also has a corresponding header file. The header file is included in the file that contains the above line of code. In this case also, on 64-bit, the value assigned to ‘ptr’ was 32-bit. Using another similar function correctly returned a 64-bit value. It took hours of debugging to notice that the declaration of this particular function was missing from the header file.

In yet another case, a function that returned a pointer was defined in one file, and used in another. An extern declaration provided the declaration to the code file that used the function. However, the extern declaration incorrectly declared the return type of the function to be int.

The explanation is rather obvious. When the declaration of a function (or variable) is not found, the ‘C’ compiler implicitly defines them as int. In 32-bit mode, int and pointer are of same size, therefore an integer value assigned to a pointer represents a valid address. But in 64-bit mode, a 32-bit value stored by an int is not a valid address and therefore a pointer assigned this value cannot be successfully de-referenced. [It is only now, when I am writing this document, it has become obvious to me that all these cases are similar. Earlier, when I worked on these problems, I had thought that the malloc issue was due to difference in compiler behavior with the standard libraries.]

1.2 Needless casting

Type * ptr_var = (Type *)(int)(func_ret_ptr(…));

There is no need to cast the value returned by the function to int. It functioned in 32-bit mode, but in 64-bit, caused 32 MSBs to be lost from the pointer returned by the function.