Re: On guest free page hinting and OOM

[David Hildenbrand]

On 01.04.19 16:09, David Hildenbrand wrote:
>>> Thinking about your approach, there is one elementary thing to notice:
>>>
>>> Giving the guest pages from the buffer while hinting requests are being
>>> processed means that the guest can and will temporarily make use of more
>>> memory than desired. Essentially up to the point where MADV_FREE is
>>> finally called for the hinted pages.
>>
>> Right - but that seems like exactly the reverse of the issue with the current
>> approach which is guest can temporarily use less memory than desired.
>>
>>> Even then the guest will logicall
>>> make use of more memory than desired until core MM takes pages away.
>>
>> That sounds more like a host issue though. If it wants to
>> it can use e.g. MAD_DONTNEED.
> 
> Indeed. But MADV_DONTNEED is somewhat undesired for performance reasons.
> You want to do the work when swapping not when hinting.
> 
> But what I wanted to say here: Looking at the pure size of your guest
> will at least not help you to identify if more memory than desired will
> be used.
> 
>>
>>> So:
>>> 1) Unmodified guests will make use of more memory than desired.
>>
>> One interesting possibility for this is to add the buffer memory
>> by hotplug after the feature has been negotiated.
>> I agree this sounds complex.
> 
> Yes it is, and it goes into the direction of virtio-mem that essentially
> does that. But bad news: memory hotplug is complicated stuff, both on
> the hypervisor and guest side. And things like NUMA make it more involved.
> 
> But even then, malicious guest can simply fake feature negotiation and
> make use of all hotplugged memory. Won't work, at least not for
> malicious guests.
> 
>>
>> But I have an idea: how about we include the hint size in the
>> num_pages counter? Then unmodified guests put
>> it in the balloon and don't use it. Modified ones
>> will know to use it just for hinting.
> 
> These are the nightmares I was talking about. I would like to decouple
> this feature as far as possible from balloon inflation/deflation.
> Ballooning is 4k based and might have other undesirable side effect.
> Just because somebody wants to use page hinting does not mean he wants
> to use ballooning. Effectively, many people will want to avoid
> ballooning completely by using page hinting for their use case.
> 
>>
>>
>>> 2) Malicious guests will make use of more memory than desired.
>>
>> Well this limitation is fundamental to balloon right?
> 
> Yep, it is the fundamental issue of ballooning. If memory is available
> right from the boot, the system is free to do with it whatever it wants.
> (one of the main things virtio-mem will do differently/better)
> 
>> If host wants to add tracking of balloon memory, it
>> can enforce the limits. So far no one bothered,
>> but maybe with this feature we should start to do that.
> 
> I think I already had endless rants about why this is not possible.
> Ballooning as it is currently implemented by virtio-balloon is broken by
> design. Period. You can and never will be able to distinguish unmodified
> guests from malicious guests. Please don't design new approaches based
> on broken design.
> 
>>
>>> 3) Sane, modified guests will make use of more memory than desired.
>>>
>>> Instead, we could make our life much easier by doing the following:
>>>
>>> 1) Introduce a parameter to cap the amount of memory concurrently hinted
>>> similar like you suggested, just don't consider it a buffer value.
>>> "-device virtio-balloon,hinting_size=1G". This gives us control over the
>>> hinting proceess.
>>>
>>> hinting_size=0 (default) disables hinting
>>>
>>> The admin can tweak the number along with memory requirements of the
>>> guest. We can make suggestions (e.g. calculate depending on #cores,#size
>>> of memory, or simply "1GB")
>>
>> So if it's all up to the guest and for the benefit of the guest, and
>> with no cost/benefit to the host, then why are we supplying this value
>> from the host?
> 
> See 3), the admin has to be aware of hinting behavior.
> 
>>
>>> 2) In the guest, track the size of hints in progress, cap at the
>>> hinting_size.
>>>
>>> 3) Document hinting behavior
>>>
>>> "When hinting is enabled, memory up to hinting_size might temporarily be
>>> removed from your guest in order to be hinted to the hypervisor. This is
>>> only for a very short time, but might affect applications. Consider the
>>> hinting_size when sizing your guest. If your application was tested with
>>> XGB and a hinting size of 1G is used, please configure X+1GB for the
>>> guest. Otherwise, performance degradation might be possible."
>>
>> OK, so let's start with this. Now let us assume that guest follows
>> the advice.  We thus know that 1GB is not needed for guest applications.
>> So why do we want to allow applications to still use this extra memory?
> 
> If the application does not need the 1GB, the 1GB will be hinted to the
> hypervisor and are effectively only a buffer for the OOM scenario.
> (ignoring page cache discussions for now).
> 
> "So why do we want to allow applications to still use this extra memory"
> is the EXACT same issue you have with your buffer approach. Any guest
> can make use of the buffer and you won't be able to detect it. Very same
> problem. Only in your approach, the guest might agree to play nicely by
> not making use of the 1G you provided. Just as if the application does
> not need/use the additional 1GB.
> 
> The interesting thing is most probably: Will the hinting size usually be
> reasonable small? At least I guess a guest with 4TB of RAM will not
> suddenly get a hinting size of hundreds of GB. Most probably also only
> something in the range of 1GB. But this is an interesting question to
> look into.
> 
> Also, if the admin does not care about performance implications when
> already close to hinting, no need to add the additional 1Gb to the ram size.

"close to OOM" is what I meant.


-- 

Thanks,

David / dhildenb