[Planetlab-users] question about packet loss

Mon Aug 11 05:43:05 EDT 2008

HI,

Great, I'm glad the fix running on the beta nodes will work for you.
It should be installed on all of the nodes sometime this week.

To see the bandwidth limitation on your slice, you can install the
'util-vserver-pl' package and run a utility called bwlimit.  Drop the
.repo file below into /etc/yum.repos.d/ and run 'sudo yum install
util-vserver-pl'.

----- /etc/yum.repos.d/planetlab.repo -----
[planetlab-rollout]
name=PlanetLab 4.2 rollout
baseurl=https://boot.planet-lab.org//install-rpms/planetlab-rollout
enabled=1
gpgcheck=0
----- /etc/yum.repos.d/planetlab.repo -----

Then '/usr/sbin/bwlimit get <slicename>' will produce output something like:

[princeton_iias at planetlab3 ~]$ /usr/sbin/bwlimit get princeton_iias
princeton_iias 1 1kbit 10mbit 1kbit 1gbit 553 0

The fourth field (e.g., 10mbit) is your bandwidth limit to
destinations on the public Internet.  The sixth field (e.g., 1gbit) is
your bw limit to destinations on Internet2 (if this node is at an
I2-connected institution).  The seventh and eighth fields show how
many bytes your slice has sent to non-I2 and I2 destinations
respectively.  The bwlimits are not user-settable.

Cheers,
Andy

On Fri, Aug 8, 2008 at 10:41 PM, Daekyeong Moon <dkmoon at cs.berkeley.edu> wrote:
> Hi Andy,
>
> Sorry. I misunderstood.
>
> I used planetlab3.csail.mit.edu as the receiver and
> hptest-1.cs.princeton.edu as the server.
> I repeated 10 times, and it seems to work. Thanks!
>
> Can you apply the fix to other regular nodes?
> My experiment needs to randomly select 16 nodes every iteration. So, I need
> more nodes.
>
> By the way, is there anyway to see and set my bandwidth limitation?
> I'm wondering whether the net_max_rate (kbps) attribute can be set by a
> user.
>
> Thanks again,
>
> DK
>
>
> On Aug 8, 2008, at 11:40 AM, Andy Bavier wrote:
>
>> Hi,
>>
>> Did you try it on some beta nodes?  It should be OK on them.  You can
>> get a list of beta nodes by logging into the PL web site and clicking
>> the Node Groups link.
>>
>> Andy
>>
>> On 8/8/08, Daekyeong Moon <dkmoon at cs.berkeley.edu> wrote:
>>>
>>> Hi Andy,
>>>
>>> Thanks.
>>> I tried on the same nodes with the toy udp sender/receiver, but it
>>> seems that I still have the issue.
>>> One interesting thing is such loss always starts from the sequence
>>> number 60.
>>> (As you can see in the program I sent you, the dumb toy program has
>>> nothing to do with the seq 60)
>>>
>>> (28 byte udp/ip header + 4 byte payload) / 1ms = 32KBps = 256Kbps
>>> I believe this is not much bw usage. Right?
>>>
>>>
>>> 57 4
>>> 58 4
>>> 59 4
>>> 797 4
>>> 798 4
>>> 799 4
>>>
>>>
>>> 57 4
>>> 58 4
>>> 59 4
>>> 469 4
>>> 470 4
>>> 471 4
>>>
>>>
>>> 57 4
>>> 58 4
>>> 59 4
>>> 1404 4
>>> 1405 4
>>> 1406 4
>>>
>>>
>>>
>>> Thanks,
>>>
>>> DK
>>>
>>>
>>> On Aug 8, 2008, at 8:52 AM, Andy Bavier wrote:
>>>
>>>> Hi,
>>>>
>>>> Ok I see what you mean. I believe the cause is a problem with the way
>>>> that the bandwidth limiter is configured on the nodes, and is fixed in
>>>> the software update running on the beta nodes. Can you try your test
>>>> on these nodes and see if it works as expected?
>>>>
>>>> Cheers
>>>> Andy
>>>>
>>>> On 8/8/08, Daekyeong Moon <dkmoon at cs.berkeley.edu> wrote:
>>>>>
>>>>> Hi Andy,
>>>>>
>>>>> Yeah, all nodes are planetlab nodes.
>>>>>
>>>>> Unfortunately, it does not always happen, and thus difficult to
>>>>> reproduce manually.
>>>>>
>>>>> Then, I'm wondering what if the receiver is highly overloaded.
>>>>> (say, w
>>>>> command show around 8.0)
>>>>> Can it drop packets for a long time? (e.g., more than a second)
>>>>>
>>>>> I did another very simple experiment.
>>>>> I used 216.165.109.79 as the receiver and 216.165.109.81 as the
>>>>> sender.
>>>>> (I guess they're on the same network.)
>>>>>
>>>>> The load average of the receiver was around 8, and the receiver
>>>>> dropped about 700 packets out of 6000, when the sender injected a 4
>>>>> byte udp packet every ms.
>>>>> Especially, it dropped consecutive packets. Here's a log snippet from
>>>>> the receiver.
>>>>> The first column is the seq number and the second one is the received
>>>>> payload size.
>>>>>
>>>>> ...
>>>>> 58 4
>>>>> 59 4
>>>>> 770 4  (big gap occurs here!! what happened!?)
>>>>> 771 4
>>>>> 772 4
>>>>> ...
>>>>>
>>>>> I wiresharked on the sender node and I could see the missing sequence
>>>>> numbers.
>>>>> (For some reason, wireshark on the receiver node didn't show packets
>>>>> at all)
>>>>>
>>>>> This is normal despite the same network?
>>>>>
>>>>> This experiment is fortunately reproduciable at this moment.
>>>>> The program I used is fairly simple udp server/client program with
>>>>> just 88 LOC.
>>>>> If you need it, I can send it to you (not to this group)
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> DK
>>>>>
>>>>>
>>>>> On Aug 7, 2008, at 11:46 PM, Andy Bavier wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In your experiment, are both the sender *and* receiver PlanetLab
>>>>>> nodes?  If so, can you retry your experiment with only the sender or
>>>>>> receiver being a PL node?  I would expect the packet loss to occur
>>>>>> on
>>>>>> the sending side, since PL doesn't do any limiting of receive
>>>>>> bandwidth.
>>>>>>
>>>>>> If you can give me a way of reproducing your problem on some nodes,
>>>>>> I'll look into it more closely.
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>> On Fri, Aug 8, 2008 at 8:02 AM, Daekyeong Moon
>>>>>> <dkmoon at cs.berkeley.edu> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm seeing weird udp loss.
>>>>>>> I'm replaying traffic traces from some Internet application.
>>>>>>> The replayer is a very simple udp application injecting packets
>>>>>>> from a pcap
>>>>>>> file according to the packet interval recorded in the file.
>>>>>>>
>>>>>>> Each sender generates at most 50 udp packets per second for 2
>>>>>>> minutes, and
>>>>>>> each packet is 300 bytes on average.
>>>>>>> So, each sender consumes about 120Kbps, which is not much.
>>>>>>>
>>>>>>> A receiver receives packets from 7 senders (approximately 840Kbps),
>>>>>>> and
>>>>>>> reports timestamp using SO_TIMESTAMP.
>>>>>>> But, I found that the receiver sometimes doesn't receive any packet
>>>>>>> for more
>>>>>>> than 10 seconds.
>>>>>>> (select() call with 10 second timeout also fires.)
>>>>>>>
>>>>>>> Since, the traces don't have such big gap and the receiver receives
>>>>>>> from the
>>>>>>> 7 senders, this is quite abnormal.
>>>>>>> I repeated 200 times with random planet lab nodes in the US, and I
>>>>>>> don't
>>>>>>> think it's the issue of particular nodes.
>>>>>>>
>>>>>>> I'm just guessing it could be related to the receiver's bandwidth
>>>>>>> limitation.
>>>>>>> (e.g. dropping all packets until the next time window if bandwidth
>>>>>>> usage in
>>>>>>> a certain time window exceeds the Shared BW limit)
>>>>>>>
>>>>>>> Anybody experienced the same issue?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> DK
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list: Users at lists.planet-lab.org
>>>>>>> https://lists.planet-lab.org/mailman/listinfo/users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list: Users at lists.planet-lab.org
>>>>>> https://lists.planet-lab.org/mailman/listinfo/users
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list: Users at lists.planet-lab.org
>>>>> https://lists.planet-lab.org/mailman/listinfo/users
>>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list: Users at lists.planet-lab.org
>>>> https://lists.planet-lab.org/mailman/listinfo/users
>>>
>>> _______________________________________________
>>> Users mailing list: Users at lists.planet-lab.org
>>> https://lists.planet-lab.org/mailman/listinfo/users
>>>
>>
>> _______________________________________________
>> Users mailing list: Users at lists.planet-lab.org
>> https://lists.planet-lab.org/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list: Users at lists.planet-lab.org
> https://lists.planet-lab.org/mailman/listinfo/users
>