IB/mthca: Work around kernel QP starvation

With mthca, RC QPs can starve each other and even UD QPs on the same
hardware schedule queue.  As a result, userspace MPI can starve
e.g. IPoIB traffic, with netdev watchdog warnings getting printed out,
and TCP connections getting stuck or failing.

Reduce the chance of this happening by using three separate hardware
schedule queues: one for userspace RC QPs, one for kernel RC QPs, and
one for all other QPs.

Signed-off-by: Michael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: Roland Dreier <rolandd@cisco.com>

diff --git a/drivers/infiniband/hw/mthca/mthca_qp.c b/drivers/infiniband/hw/mthca/mthca_qp.c
index 8fe6fee7a97ae21f036c4866e2236256e794a0e1..fee60c852d1441e032293bab94c2d011cceb517a 100644 (file)
--- a/drivers/infiniband/hw/mthca/mthca_qp.c
+++ b/drivers/infiniband/hw/mthca/mthca_qp.c
@@ -701,6 +701,19 @@ int mthca_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask,
                qp_param->opt_param_mask |= cpu_to_be32(MTHCA_QP_OPTPAR_PRIMARY_ADDR_PATH);
        }
 
+       if (ibqp->qp_type == IB_QPT_RC &&
+           cur_state == IB_QPS_INIT && new_state == IB_QPS_RTR) {
+               u8 sched_queue = ibqp->uobject ? 0x2 : 0x1;
+
+               if (mthca_is_memfree(dev))
+                       qp_context->rlkey_arbel_sched_queue |= sched_queue;
+               else
+                       qp_context->tavor_sched_queue |= cpu_to_be32(sched_queue);
+
+               qp_param->opt_param_mask |=
+                       cpu_to_be32(MTHCA_QP_OPTPAR_SCHED_QUEUE);
+       }
+
        if (attr_mask & IB_QP_TIMEOUT) {
                qp_context->pri_path.ackto = attr->timeout << 3;
                qp_param->opt_param_mask |= cpu_to_be32(MTHCA_QP_OPTPAR_ACK_TIMEOUT);