Fix malloc_vsnprintf handling of %o, %u and %x

These flags take unsigned values, but they were fed with signed values
taken with va_arg, and that led to sign extension in cases where the
corresponding value has the most significant bit set.