Re: [PATCH tip/core/rcu 04/21] docs/memory-barriers.txt: Rewrite "KERNEL I/O BARRIER EFFECTS" section

[Will Deacon]

Hi Akira,

On Fri, Apr 05, 2019 at 12:58:36AM +0900, Akira Yokosawa wrote:
> On Tue, 2 Apr 2019 14:03:46 +0100, Will Deacon wrote:
> > On Tue, Mar 26, 2019 at 04:41:16PM -0700, Paul E. McKenney wrote:
> >> From: Will Deacon <will.deacon@xxxxxxx>
> >>
> >> The "KERNEL I/O BARRIER EFFECTS" section of memory-barriers.txt is vague,
> >> x86-centric, out-of-date, incomplete and demonstrably incorrect in places.
> >> This is largely because I/O ordering is a horrible can of worms, but also
> >> because the document has stagnated as our understanding has evolved.
> >>
> >> Attempt to address some of that, by rewriting the section based on
> >> recent(-ish) discussions with Arnd, BenH and others. Maybe one day we'll
> >> find a way to formalise this stuff, but for now let's at least try to
> >> make the English easier to understand.
> >>
> >> Cc: "Paul E. McKenney" <paulmck@xxxxxxxxxxxxx>
> >> Cc: Benjamin Herrenschmidt <benh@xxxxxxxxxxxxxxxxxxx>
> >> Cc: Michael Ellerman <mpe@xxxxxxxxxxxxxx>
> >> Cc: Arnd Bergmann <arnd@xxxxxxxx>
> >> Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
> >> Cc: Andrea Parri <andrea.parri@xxxxxxxxxxxxxxxxxxxx>
> >> Cc: Palmer Dabbelt <palmer@xxxxxxxxxx>
> >> Cc: Daniel Lustig <dlustig@xxxxxxxxxx>
> >> Cc: David Howells <dhowells@xxxxxxxxxx>
> >> Cc: Alan Stern <stern@xxxxxxxxxxxxxxxxxxx>
> >> Cc: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
> >> Cc: "Maciej W. Rozycki" <macro@xxxxxxxxxxxxxx>
> >> Cc: Mikulas Patocka <mpatocka@xxxxxxxxxx>
> >> Signed-off-by: Will Deacon <will.deacon@xxxxxxx>
> >> Signed-off-by: Paul E. McKenney <paulmck@xxxxxxxxxxxxx>
> >> ---
> >>  Documentation/memory-barriers.txt | 115 ++++++++++++++++++------------
> >>  1 file changed, 70 insertions(+), 45 deletions(-)
> > 
> > If somebody could provide an Ack on this patch, I'd really appreciate it,
> > please. Whilst the portable ordering guarantees that I've documented are
> > fairly conservative, I do think that this change is a big improvement and
> > gives you what you need if you're writing a portable device driver for a new
> > piece of hardware. I'm tackling the removal of MMIOWB as a separate series.
> > 
> > I think Paul now requires an Ack before he'll send a patch to mainline,
> > hence the grovelling.
> 
> I'm afraid I'm not that qualified to provide an Ack to this patch,
> but please find a nit fix below.

Oh well, thanks for having a look anyway!

> >> + (*) insX(), outsX():
> >> +
> >> +     As above, the insX() and outX() accessors provide the same ordering
>                                   outsX()

Thanks; I'll fix that.

> >> +     guarantees as readsX() and writesX() respectively when accessing a mapping
> >> +     with the default I/O attributes.
> >>  
> >>   (*) ioreadX(), iowriteX()
> >>  
> >>       These will perform appropriately for the type of access they're actually
> >>       doing, be it inX()/outX() or readX()/writeX().
> >>  
> >> +All of these accessors assume that the underlying peripheral is little-endian,
> >> +and will therefore perform byte-swapping operations on big-endian architectures.
> >> +
> >> +Composing I/O ordering barriers with SMP ordering barriers and LOCK/UNLOCK
> >> +operations is a dangerous sport which may require the use of mmiowb(). See the
> >> +subsection "Acquires vs I/O accesses" for more information.
> >>  
> >>  ========================================
> >>  ASSUMED MINIMUM EXECUTION ORDERING MODEL
> >> -- 
> >> 2.17.1
> >>
> 
> JFYI, there is another document Documentation/driver-api/device-io.rst,
> which is somewhat related to this update. It looks like this one also needs
> some update, as Jon commented in transforming to .rst format in commit
> 8a8a602fdb83 ("docs: Convert the deviceio template to RST"):
> <quote>
>     Like the rest of our documentation, this one could use some work.  There's
>     no mention of ioremap() and friends, no mention of io_read*() and friends.
>     But we have nice documentation for all those folks writing new drivers that
>     do port I/O :).
> </quote>
> 
> This commit was merged in v4.11 cycle. And there has been no update whatsoever
> since. mmiowb() is lightly mentioned therein. IMHO, just updating
> memory-barriers.txt would widen the gap of information.
> 
> Thoughts?

I have a subsequent patch which kills mmiowb() entirely:

https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/commit/?h=for-next/mmiowb&id=3c1a2050c08fb8193777b60b49e60320254a156c

and that one does hit device-io.rst.

Will