IB/mthca: Avoid recycling old FMR R_Keys too soon

When a FMR is unmapped, mthca resets the map count to 0, and clears
the upper part of the R_Key which is used as the sequence counter.

This poses a problem for RDS, which uses ib_fmr_unmap as a fence
operation.  RDS assumes that after issuing an unmap, the old R_Keys
will be invalid for a "reasonable" period of time. For instance,
Oracle processes uses shared memory buffers allocated from a pool of
buffers.  When a process dies, we want to reclaim these buffers -- but
we must make sure there are no pending RDMA operations to/from those
buffers.  The only way to achieve that is by using unmap and sync the
TPT.

However, when the sequence count is reset on unmap, there is a high
likelihood that a new mapping will be given the same R_Key that was
issued a few milliseconds ago.

To prevent this, don't reset the sequence count when unmapping a FMR.

Signed-off-by: Olaf Kirch <olaf.kirch@oracle.com>
Signed-off-by: Roland Dreier <rolandd@cisco.com>

diff --git a/drivers/infiniband/hw/mthca/mthca_mr.c b/drivers/infiniband/hw/mthca/mthca_mr.c
index 3538da16e3fe9ec4cea862fa7b0100fb3d3225c7..820205dec560954562b22c570f7eb13861ec8824 100644 (file)
--- a/drivers/infiniband/hw/mthca/mthca_mr.c
+++ b/drivers/infiniband/hw/mthca/mthca_mr.c
@@ -818,15 +818,9 @@ int mthca_arbel_map_phys_fmr(struct ib_fmr *ibfmr, u64 *page_list,
 
 void mthca_tavor_fmr_unmap(struct mthca_dev *dev, struct mthca_fmr *fmr)
 {
-       u32 key;
-
        if (!fmr->maps)
                return;
 
-       key = tavor_key_to_hw_index(fmr->ibmr.lkey);
-       key &= dev->limits.num_mpts - 1;
-       fmr->ibmr.lkey = fmr->ibmr.rkey = tavor_hw_index_to_key(key);
-
        fmr->maps = 0;
 
        writeb(MTHCA_MPT_STATUS_SW, fmr->mem.tavor.mpt);
@@ -834,16 +828,9 @@ void mthca_tavor_fmr_unmap(struct mthca_dev *dev, struct mthca_fmr *fmr)
 
 void mthca_arbel_fmr_unmap(struct mthca_dev *dev, struct mthca_fmr *fmr)
 {
-       u32 key;
-
        if (!fmr->maps)
                return;
 
-       key = arbel_key_to_hw_index(fmr->ibmr.lkey);
-       key &= dev->limits.num_mpts - 1;
-       key = adjust_key(dev, key);
-       fmr->ibmr.lkey = fmr->ibmr.rkey = arbel_hw_index_to_key(key);
-
        fmr->maps = 0;
 
        *(u8 *) fmr->mem.arbel.mpt = MTHCA_MPT_STATUS_SW;