[Planetlab-devel] Re: [PL #45504] VSYS script for adjusting txqueuelen parameter on tap interface

[Thom Haddow]

Sapan Bhatia wrote:
>> That's understandable, if this issue is widespread for other people. Would
>> it be possible to roll the change out to a couple of nodes to confirm the
>> fix?
> 
> We can do this, but again, preferably after the NSDI deadline.

Ok. I'll take a crack at the VSYS script then, as we believe this is crippling 
our performance. You're probably right about it affecting other experiments - 
having done a quick survey across ~250 machines, each one has lost on average 
2.5 million packets this way (I guess since the last reboot).

>> I suspect this is what I'd like to do eventually, as that would also enable
>> us to solve our NAT problem as well. I'm a little unsure on what it is
>> you've done to create this 'multiplexed' tap interface, since it's different
>> on PlanetLab compared to how it works on regular systems (in which the tap
>> interface doesn't exist until you try to open it in userspace) - could you
>> point me to the scripts that configure this originally? I'm finding the
>> intra-
>> node network plumbing is otherwise proving to be a bit of black-box.
> 
> Ah, I should have been more clear here. I was suggesting that for your
> slice, and for slices that need to change the configuration of the
> interface, we use the standard tun/tap device, not the
> Planetlab-specific one with multiplexing. So you would bring it up
> just like on a regular Linux box.

So the /dev/net/tun device isn't otherwise special and will behave as on normal 
Linux platforms?

> This is something that we have wanted to do for some time now, since
> it also lifts the restriction of having only 1 tap device per slice,
> which some users have
> complained about.
> 
>> What do you mean by "not taken?" - I assume the 10/8 addresses on for the
>> tap0 devices on each node are unique, but do you know how they were
>> generated
>> in the first place?
> 
> Right, those are generated in a way that they are unique for every
> slice. But if we bring up a standard tap device on demand, then we'll
> need to check that the address that we are going to assign to it is
> unused, probably by walking the list of active interfaces.

Thanks, that should be easy enough. If I put them into the 172.16/12 space that 
should avoid any trouble with the routing tables too (since the existing taps 
are masked /8). Does that seem reasonable?


Thanks for your help!

Thom

>>> On Tue, Sep 22, 2009 at 7:12 AM, Thom Haddow <thaddow at doc.ic.ac.uk> wrote:
>>>> Hello,
>>>>
>>>> Any update on this?
>>>>
>>>> thanks,
>>>>
>>>> Thom
>>>>
>>>> Thom Haddow wrote:
>>>>> Sapan Bhatia wrote:
>>>>>> Hi Thomas,
>>>>>>
>>>>>> Increasing the default queue length seems to be the right approach
>>>>>> here. I am cc-ing the users mailing list, in case this is a problem
>>>>>> for somebody else using tun/tap on PlanetLab.
>>>>>>
>>>>>> What do you need txqueuelen to be increased to?
>>>>> Thanks for getting back to me. I think 5000 would be safe.
>>>>>
>>>>> cheers,
>>>>>
>>>>> Thom
>>>>>
>>>>>
>>>>>> On Wed, Sep 16, 2009 at 12:21 PM, Thom Haddow <thaddow at doc.ic.ac.uk>
>>>>>> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> We're having trouble reliably reading from the tap0-/dev/net/tun
>>>>>>> device
>>>>>>> under load bursts, as detailed below. I was hoping that increasing
>>>>>>> txqueuelen
>>>>>>> on the tap0 device via a VSYS script would smooth out these load
>>>>>>> bursts
>>>>>>> sufficiently, but it seems this may be complicated by the fact that
>>>>>>> the tap0 interface is apparently multiplexed between slices. I was
>>>>>>> directed
>>>>>>> to this list hoping that someone might have some suggestions about how
>>>>>>> this
>>>>>>> might be achieved?
>>>>>>>
>>>>>>> Failing that, would there be any particular impact from increasing the
>>>>>>> size
>>>>>>> of txqueuelen on tap0 by default?
>>>>>>>
>>>>>>> thanks,
>>>>>>>
>>>>>>> Thom
>>>>>>>
>>>>>>> Sapan Bhatia via RT wrote:
>>>>>>>> Email Recipients (see http://www.planet-lab.org/Support )
>>>>>>>>     Owner: sapanb
>>>>>>>>     Requestor: t.haddow08 at imperial.ac.uk
>>>>>>>>
>>>>>>>>
>>>>>>>> ==================================================
>>>>>>>>
>>>>>>>> Hi Thomas,
>>>>>>>>
>>>>>>>> Are you on the PlanetLab devel mailing list? I suggest that you post
>>>>>>>> requests of this type there. You will get more feedback, and most
>>>>>>>> likely your requests will be acted on more promptly.
>>>>>>>>
>>>>>>>> Your script seems reasonable. The problem is that there is only 1
>>>>>>>> multiplexed tap interface on a PL node. I am not sure how to address
>>>>>>>> this problem. But if you post the question on devel, somebody might
>>>>>>>> have ideas.
>>>>>>>>
>>>>>>>> Sapan
>>>>>>>>
>>>>>>>> On Tue, Sep 15, 2009 at 10:48 PM, Haddow, Thomas via RT
>>>>>>>> <support at planet-lab.org> wrote:
>>>>>>>>> Email Recipients (see http://www.planet-lab.org/Support )
>>>>>>>>>    Requestor: t.haddow08 at imperial.ac.uk
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ==================================================
>>>>>>>>>
>>>>>>>>> Tue Sep 15 22:48:58 2009: Request 45504 was acted upon.
>>>>>>>>> Transaction: Ticket created by t.haddow08 at imperial.ac.uk
>>>>>>>>>
>>>>>>>>> Subject: VSYS script for adjusting txqueuelen parameter on tap
>>>>>>>>> interface
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> Following on from the earlier UDP socket buffering issue, we've
>>>>>>>>> discovered that there is another buffering issue in our overlay
>>>>>>>>> implementation. We believe that under short load bursts we are
>>>>>>>>> unable
>>>>>>>>> to read from the tap0 device regularly enough, resulting in local
>>>>>>>>> packet drops (as indicated by ifconfig). Hopefully this would be
>>>>>>>>> fixable by increasing the txqueuelen parameter on the interface, but
>>>>>>>>> we don't have the permissions to set this option. Would it be
>>>>>>>>> possible
>>>>>>>>> to deploy a VSYS script to let us tune this parameter for our own
>>>>>>>>> tap
>>>>>>>>> interface? Attached is a script that is hopefully suitable, although
>>>>>>>>> I'm not sure how the root slice would refer to our slice-specific
>>>>>>>>> tap0
>>>>>>>>> device.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Thom
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> PlanetLab Support Mail Reflector
>>>>>>>>> support at planet-lab.org
>>>>>>>>> https://lists.planet-lab.org/mailman/listinfo/support-community
>>>>>>>>>
>>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel at lists.planet-lab.org
>>>>>>> https://lists.planet-lab.org/mailman/listinfo/devel
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Devel mailing list
>>>>>> Devel at lists.planet-lab.org
>>>>>> https://lists.planet-lab.org/mailman/listinfo/devel
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at lists.planet-lab.org
>>>>> https://lists.planet-lab.org/mailman/listinfo/devel
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel at lists.planet-lab.org
>>>> https://lists.planet-lab.org/mailman/listinfo/devel
>>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel at lists.planet-lab.org
>>> https://lists.planet-lab.org/mailman/listinfo/devel
>> _______________________________________________
>> Devel mailing list
>> Devel at lists.planet-lab.org
>> https://lists.planet-lab.org/mailman/listinfo/devel
>>
> 
> _______________________________________________
> Devel mailing list
> Devel at lists.planet-lab.org
> https://lists.planet-lab.org/mailman/listinfo/devel