[Planetlab-users] probing for bandwidth measurements
Gaurav Gupta
mail2gaurav at gmail.com
Fri May 13 04:25:30 EDT 2005
Hi,
I are facing problems sniffing packets using libpcap/ tcpdump, while
sending packets from one packet lab machine to another.
Normal sockets and raw sockets seem to work fine, but we tried to
capture using libpcap, we are unable to sniff any packets, also
tcpdump does not capture any packets, however it works fine when we
try to capture ICMP echo replies sent to any other node from that
node.
-------------------------------------------------------------------------------------
int main(void)
{
char *dev;
char errbuf[PCAP_ERRBUF_SIZE];
pcap_t* descr;
const u_char *packet;
struct pcap_pkthdr hdr; /* pcap.h */
struct ether_header *eptr; /* net/ethernet.h */
struct bpf_program fp; /* hold compiled program */
bpf_u_int32 maskp; /* subnet mask */
bpf_u_int32 netp; /* ip */
dev = pcap_lookupdev(errbuf);
if(dev == NULL)
{ fprintf(stderr,"%s\n",errbuf); exit(1); }
pcap_lookupnet(dev,&netp,&maskp,errbuf);
descr = pcap_open_live(dev,BUFSIZ,0,-1,errbuf);
if(descr == NULL)
{ printf("pcap_open_live(): %s\n",errbuf); exit(1); }
if(pcap_compile(descr,&fp,"port 5000",0,netp) == -1)
{ fprintf(stderr,"Error calling pcap_compile\n"); exit(1); }
if(pcap_setfilter(descr,&fp) == -1)
{ fprintf(stderr,"Error setting filter\n"); exit(1); }
pcap_loop(descr,UDP_TRAIN_LEN -1,my_callback,NULL);
return(0)
}
void my_callback(u_char *useless,const struct pcap_pkthdr*
pkthdr,const u_char* packet)
{
printf("iside callback\n");
double time = (pkthdr->ts.tv_sec) + (pkthdr->ts.tv_usec)/1000000.0;
time_r(time);
}
-------------------------------------------------------------------------------------
On 5/11/05, Neil Spring <nspring at cs.umd.edu> wrote:
> Manish,
>
> This seems consistent with what I understand -- that pcap is the only
> way to go to measure delays on busy machines -- though I wildly
> underestimated the resistance folks would have toward techniques that
> require running as root on non-planetlab machines. I guess I assumed
> that those who knew how "interrupt coalescing" could affect a
> measurement tool wouldn't even think about using gettimeofday().
>
> At the same time, it's not immediately clear to me why you're never
> able to send 20 0.2ms-spaced packets in a row, as that should only
> hold the processor for 4ms, so perhaps there's more going on.
>
> For what it's worth, the functionality of kernel timestamps for
> incoming packets can be achieved, without pcap and without root, by
> using the SIOCGSTAMP ioctl after recvfrom(). The following code from
> scriptroute grabbed receive timestamps from the "silk" socket in the
> days before Mark made libpcap work:
>
> readbytes = recv(sock, packet_buf, BUFFER_SIZE, 0);
> if(ioctl(sock, SIOCGSTAMP, &tv) < 0) {
> (void)gettimeofday(&tv, NULL);
> }
>
> That code fragment (with appropriate error checking I removed before
> posting) might help you run a receiver as non-root on busy machines.
> There's also an SO_TIMESTAMP socket option I have not used,
> contrasted with this ioctl at http://nms.lcs.mit.edu/software/CM/doc/
> cmappnotes.html.
>
> -neil
>
>
> On May 11, 2005, at 10:37 PM, Manish Jain wrote:
>
> >
> > Last week, several questions were raised about the usefulness of
> > planet-lab for bandwidth estimation using existing tools. I have
> > done a small study to answer this question and the results are
> > described below.
> >
> > Most available bandwidth estimation tools have two requirements
> > from the end host - first, the capability to send packet trains at
> > certain rate and second, to be able to time-stamp the arrival of
> > packets. Depending on how loaded the end hosts are, these
> > requirements may be hard to achieve. In this report, we present
> > results of accuracy of sending and receiving packet trains on
> > planet-lab nodes, using a stripped down version of pathload. Most
> > of the publicly available tools use "gettimeofday" to time stamp
> > packets. Since, planet-lab provides access to "pcap", which gives
> > more accurate timestamping, we use both, gettimeofday and pcap
> > timestamps, in this study.
> >
> > The two main observations are- first, it is hard to achieve a pre-
> > specified sending rate non-planetlab nodes. Second, packet arrival
> > can not be accurately timestamped using gettimeofday. Though using
> > "pcap" improves the accuracy, most bandwidth estimation tools do
> > not use pcap, as it requires root privileges.
> >
> > Next we briefly describe a typical mechanism to transmit and
> > receive packet train at certain rate. A sender process transmits
> > train with rate R by uniformly spacing packets by time=(packet_size/
> > R). For example, if R=50Mbps and packet size is 1500 bytes, then
> > all the consecutive packets in a train must be spaced by 240 micro-
> > seconds (usec). In pathload, we achieve this spacing in two ways.
> > If the desired spacing is smaller than the minimum granularity of
> > timer, then spacing is achieved by doing the "busy wait".
> > Otherwise, we use a combination of timer and "busy wait". Packets
> > in the train are timestamped using gettimeofday(), just before
> > "sendto()" call. At receiving host, packets are received using
> > "recvfrom()" system call, and timestamped using "gettimeofday".
> >
> > For this study, we selected 10 nodes at different planet-lab sites
> > (mostly in continental US) and one non-planetlab node at Georgia
> > Tech, named sirius, which has negligible CPU load. To examine the
> > effect of end host on sending packet train with certain rate, we
> > run sender on planet-lab nodes and receiver on sirius. Similarly,
> > to examine the effect of end host on the accuracy of time stamping
> > arrival of packets, we run sender on sirius and receiver on planet-
> > lab nodes. In the stripped down version of pathload, the only task
> > sender does is to transmit a train of timestamped packets and
> > receiver receives and timestamps each arriving packet. Packet size
> > used in the experiment is 1500 bytes and time stamping is done
> > using "gettimeofday".
> >
> > In the first set of experiments, when sender was running on planet-
> > lab node, sender side spacing in several packets is significantly
> > higher than the requested spacing. This can happen if the sender
> > process is "switched out" after transmitting i-th packet and before
> > transmitting (i+1)-th packet. Then the spacing between these two
> > packets will correspond to the time sender process was "switched
> > out", which is larger than the requested spacing. Therefore, the
> > actual train rate will be smaller than the requested rate. One may
> > think about using "pcap" to circumvent this problem. However, use
> > of "pcap" would not solve this problem, since it would not change
> > the rate at which sending process is generating packets.
> >
> > Figure <http://www.cc.gatech.edu/~jain/send_disp.eps> shows actual
> > spacing (dispersion) between packets at sender, when desired
> > spacing was 120 usec. Note that the packets in train almost
> > periodically have large spacing, about 10 msec, which may due to
> > context switching. The requested rate and actual rate for this
> > packet train are 60Mbps and 6.6Mbps, respectively. Figure <http://
> > www.cc.gatech.edu/~jain/send_disp_5ms.eps> shows similar graph,
> > when desired spacing was 5 msec. In this case too, packet spacing
> > is not accurate on few occasions and the requested rate and actual
> > rate for this packet train are 2.4Mbps and 2.3Mbps, respectively.
> >
> > In the second set of experiments, when receiver was running on
> > planetlab node, we observe that significant number of packets
> > arrived with a spacing of 8-20 usec. This spacing is much smaller
> > than the reception latency of 100Mbps NIC, which is 120 usec, on
> > these hosts. One reason this can happen is- if the receiver process
> > is "switched out" while packets arrive, some of the packets will be
> > buffered in the kernel. When receiver process is "switched in", the
> > buffered packets are transferred from the kernel to receiving
> > process with a spacing equal to the latency of "recvfrom" system
> > call. Figure <http://www.cc.gatech.edu/~jain/recv_disp.eps> shows
> > receiver dispersion, measured as difference of arrival timestamp of
> > consecutive packets. Most packets are received with a spacing of
> > 16 usec, which is the latency of "recvfrom" on the receiver node.
> >
> > Next we show use of "pcap" to improve arrival time-stamp accuracy.
> > Figure <http://www.cc.gatech.edu/~jain/recv_disp_pcap.eps> shows
> > the receiver dispersion measured with "pcap". Here, we do not
> > observe any packet spacing less than the NIC reception latency.
> > This is because "pcap" timestamps the packets even when the
> > receiver process is switched out.
> >
> > To verify our conjecture that high CPU load, and therefore "context
> > switch", is causing inaccuracies, we used a tool called "cotop" to
> > verify the load on the system before and after we ran sender/
> > receiver process on planetlab node. cotop gives the total CPU load
> > and idle CPU time, averaged over 3 sec duration. Observations from
> > cotop were consistent with observed inaccuracies, i.e. cotop showed
> > heavy CPU utilization each time we observed consecutive packets
> > with small receiver side dispersion, or large sender side dispersion.
> >
> > To summarize the results of the above experiments, first
> > observation is that it is very difficult to accurately generate
> > train of certain rate on planetlab. Secondly, context switching at
> > receiver will affect the accuracy of all available bandwidth
> > estimation tools.
> >
> > Effects of context switching at receiver can be mitigated by the
> > use of "pcap". However, there is a practical issue with the use of
> > "pcap". Using "pcap" requires root privilege, which is not
> > available in many cases outside planetlab. Tool developers may want
> > to use planetlab only for testing their tools, hence need to use
> > "pcap" for testing on planet-lab and release tools with
> > "gettimeofday". Therefore, tool validation results will be
> > stronger than that can be achieved by their released version.
> >
> > Finally, a note of caution to those using available bandwidth
> > estimation tools on planet-lab. Pathload uses techniques to detect
> > whether the send (receive) process experienced significant context
> > switching while the train were sent (received). In that case, the
> > tool typically aborts the measurements or raises warnings. However,
> > most other tools do not test for context switch before reporting
> > final result. In such cases, users of such tools will never know if
> > their measurements were affected by the load on the node.
> > Consequently, one may draw wrong conclusions based on such
> > observations.
> >
> >
> > Manish Jain
> > www.cc.gatech.edu/~jain
> >
> > On Thu, 5 May 2005, Neil Spring wrote:
> >
> >
> >>
> >> On May 5, 2005, at 11:13 AM, Manfred Georg wrote:
> >>
> >>> I'll look into libpcap. And for the record, I was having trouble
> >>> getting
> >>> accurate measurements on linux too. The problem was just aggrevated
> >>> on planetlab.
> >>>
> >>
> >> Ahh. Then it might be good, long-term, to revive and extend the
> >> PeriScope work (where packet pairs are handed in one system call
> >> to the kernel) with an eye toward pushing it into Linux and
> >> Planetlab.
> >>
> >> I'm looking forward to Manish's report -- I'd like to know what
> >> train he can get on plain linux but not on planetlab. It's tough
> >> to send packets uniformly at 5ms spacing with any other process
> >> running even on a plain linux machine.
> >>
> >> To be clear, my position, without much evidence, is that PlanetLab
> >> is at least as good as a shared linux machine, (ignoring the rate-
> >> limits) and that defensive programming is required everywhere to
> >> ensure that what you think you send is really what you sent. I
> >> believe that Pathrate is the hardest possible tool to get right on
> >> Planetlab because of its need for precise timing over relatively
> >> long durations (unlike, say, packet chirps which need less time).
> >>
> >> Would someone who attended last week's meeting summarize:
> >>
> >> http://www.planet-lab.org/Talks/2005-05-01/
> >> planetLab_meeting_050105.ppt
> >>
> >> It seems some folks have very recent experience with pathrate on
> >> planetlab.
> >>
> >> thanks,
> >> -neil
> >>
> >> _______________________________________________
> >> Users mailing list: Users at lists.planet-lab.org
> >> https://lists.planet-lab.org/mailman/listinfo/users
> >>
> >
>
> _______________________________________________
> Users mailing list: Users at lists.planet-lab.org
> https://lists.planet-lab.org/mailman/listinfo/users
>
--
gaurav at iitd.ernet.in
gaurav at cse.iitd.ernet.in
All the world's a stage and most of us are desperately unrehearsed.
-- Sean O'Casey
More information about the Users
mailing list