Fix rtree_subkey() to use uintptr_t rather than unsigned for key bitmasking. This regression was introduced by 4a346f55 (Replace rtree path cache with LRU cache.).
