Update mallocx() OOM test to deal with smaller hugemax.

Depending on virtual memory resource limits, it is necessary to attempt
allocating three maximally sized objects to trigger OOM rather than just
two, since the maximum supported size is slightly less than half the
total virtual memory address space.

This fixes a test failure that was introduced by
0c516a00 (Make *allocx() size class
overflow behavior defined.).

This resolves #379.