Re: [PATCH] platform/chrome: cros_ec_spi: Transfer messages at high priority

[Matthias Kaehlcke]

On Tue, Apr 02, 2019 at 08:17:03PM -0700, Guenter Roeck wrote:
> On Tue, Apr 2, 2019 at 4:38 PM Doug Anderson <dianders@xxxxxxxxxxxx> wrote:
> >
> > Hi,
> >
> > On Tue, Apr 2, 2019 at 4:19 PM Matthias Kaehlcke <mka@xxxxxxxxxxxx> wrote:
> > >
> > > Hi Doug,
> > >
> > > On Tue, Apr 02, 2019 at 03:44:44PM -0700, Douglas Anderson wrote:
> > > > The software running on the Chrome OS Embedded Controller (cros_ec)
> > > > handles SPI transfers in a bit of a wonky way.  Specifically if the EC
> > > > sees too long of a delay in a SPI transfer it will give up and the
> > > > transfer will be counted as failed.  Unfortunately the timeout is
> > > > fairly short, though the actual number may be different for different
> > > > EC codebases.
> > > >
> > > > We can end up tripping the timeout pretty easily if we happen to
> > > > preempt the task running the SPI transfer and don't get back to it for
> > > > a little while.
> > > >
> > > > Historically this hasn't been a _huge_ deal because:
> > > > 1. On old devices Chrome OS used to run PREEMPT_VOLUNTARY.  That meant
> > > >    we were pretty unlikely to take a big break from the transfer.
> > > > 2. On recent devices we had faster / more processors.
> > > > 3. Recent devices didn't use "cros-ec-spi-pre-delay".  Using that
> > > >    delay makes us more likely to trip this use case.
> > > > 4. For whatever reasons (I didn't dig) old kernels seem to be less
> > > >    likely to trip this.
> > > > 5. For the most part it's kinda OK if a few transfers to the EC fail.
> > > >    Mostly we're just polling the battery or doing some other task
> > > >    where we'll try again.
> > > >
> > > > Even with the above things, this issue has reared its ugly head
> > > > periodically.  We could solve this in a nice way by adding reliable
> > > > retries to the EC protocol [1] or by re-designing the code in the EC
> > > > codebase to allow it to wait longer, but that code doesn't ever seem
> > > > to get changed.  ...and even if it did, it wouldn't help old devices.
> > > >
> > > > It's now time to finally take a crack at making this a little better.
> > > > This patch isn't guaranteed to make every cros_ec SPI transfer
> > > > perfect, but it should improve things by a few orders of magnitude.
> > > > Specifically you can try this on a rk3288-veyron Chromebook (which is
> > > > slower and also _does_ need "cros-ec-spi-pre-delay"):
> > > >   md5sum /dev/zero &
> > > >   md5sum /dev/zero &
> > > >   md5sum /dev/zero &
> > > >   md5sum /dev/zero &
> > > >   while true; do
> > > >     cat /sys/class/power_supply/sbs-20-000b/charge_now > /dev/null;
> > > >   done
> > > > ...before this patch you'll see boatloads of errors.  After this patch I
> > > > don't see any in the testing I did.
> > > >
> > > > The way this patch works is by effectively boosting the priority of
> > > > the cros_ec transfers.  As far as I know there is no simple way to
> > > > just boost the priority of the current process temporarily so the way
> > > > we accomplish this is by creating a "WQ_HIGHPRI" workqueue and doing
> > > > the transfers there.
> > > >
> > > > NOTE: this patch relies on the fact that the SPI framework attempts to
> > > > push the messages out on the calling context (which is the one that is
> > > > boosted to high priority).  As I understand from earlier (long ago)
> > > > discussions with Mark Brown this should be a fine assumption.  Even if
> > > > it isn't true sometimes this patch will still not make things worse.
> > > >
> > > > [1] https://crbug.com/678675
> > > >
> > > > Signed-off-by: Douglas Anderson <dianders@xxxxxxxxxxxx>
> > > > ---
> > > >
> > > >  drivers/platform/chrome/cros_ec_spi.c | 107 ++++++++++++++++++++++++--
> > > >  1 file changed, 101 insertions(+), 6 deletions(-)
> > > >
> > > > diff --git a/drivers/platform/chrome/cros_ec_spi.c b/drivers/platform/chrome/cros_ec_spi.c
> > > > index ffc38f9d4829..101f2deb7d3c 100644
> > > > --- a/drivers/platform/chrome/cros_ec_spi.c
> > > > +++ b/drivers/platform/chrome/cros_ec_spi.c
> > > >
> > > > ...
> > > >
> > > > +static int cros_ec_pkt_xfer_spi(struct cros_ec_device *ec_dev,
> > > > +                             struct cros_ec_command *ec_msg)
> > > > +{
> > > > +     struct cros_ec_spi *ec_spi = ec_dev->priv;
> > > > +     struct cros_ec_xfer_work_params params;
> > > > +
> > > > +     INIT_WORK(&params.work, cros_ec_pkt_xfer_spi_work);
> > > > +     params.ec_dev = ec_dev;
> > > > +     params.ec_msg = ec_msg;
> > > > +
> > > > +     queue_work(ec_spi->high_pri_wq, &params.work);
> > > > +     flush_workqueue(ec_spi->high_pri_wq);
> > >
> > > IIRC dedicated workqueues should be avoided unless they are needed. In
> > > this case it seems you could use system_highpri_wq + a
> > > completion. This would add a few extra lines to deal with the
> > > completion, in exchange the code to create the workqueue could be
> > > removed.
> >
> > I'm not convinced using the "system_highpri_wq" is a great idea here.
> > Using flush_workqueue() on the "system_highpri_wq" seems like a recipe
> > for deadlock but I need to flush to get the result back.  See the
> > comments in flush_scheduled_work() for some discussion here.
> >
> 
> Given that high priority workqueues are used all over the place, and
> system_highpri_wq is only rarely used, hijacking the latter doesn't
> seem to be such a good idea to me either.

It *might* be that system_highpri_wq is used less often because it was
introduced later than high priority workqueues in general, also many
folks might still think of workqueues as 'one thread per CPU' (like
myself until I was recently pointed to cmwq) and prefer to have their
own wq for this reason.

> I also recall that we had to drop using system qorkqueues at a
> previous company and replace them with local workqueues because we
> got into timing trouble when using system workqueues.

IIUC this shouldn't be an issue with cwmq (which has worker pools),
unless a work is a CPU hog, in which case it probably shouldn't use
a system workqueue in the first place.

Not saying that the problem you mentioned didn't exist, just sharing
what is my current understanding, maybe somebody will point out that
I'm plain wrong and I/we learn something new about wqs ;-)

Matthias