[Planetlab-devel] Network virtualization

Wed Feb 21 11:37:47 EST 2007

Hello Daniel,

The link between the two machines is Gigabit ethernet, connected via
Broadcom NetXtreme cards and a gigabit switch.

The tools are a small set of scripts in C/perl that I wrote myself because I
wanted to do best-effort sending (which i-perf doesn't support) and thought
I'd need more control (especially in the routing cases) than Netperf has to
offer. Basically, we send UDP packets in a loop and measure the jitter, etc.
on the server.

What is the meaning of the Native Linux - 19, NetNS - 1, OpenVz - 7 ?

It means that native linux hits a CPU utilization of 100% (on one core) when
the number of concurrent senders is raised to 19. For NetNS, this number is 1
and for OpenVZ it's 7. NetNS and OpenVZ were compiled using different kernels
(2.6.20 and 2.6.16
 respectively) so there might have been some differences there.

Do you have some graphics showing the throughput/cpu/jitter vs nb
> containers ?

I do for CPU utilization and jitter. I'll rerun the experiments once and
post them.

I am very surprised to read the throughput is not impacted by the
> containers number while the CPU utilization is much more important. Do
> you have any explication on that ?

The explication, to my mind, is as follows.

- Basically, 1 NETNS/OpenVZ process can saturate a gigabit link (because it
doesn't add an overly large ninstruction overhead)
- When you have N processes running in parallel, since only one of them runs
at a time (we use 1 core) uninterrupted for many packet sends, it saturates
the link, and so throughput doesn't degrade
- At N=<some threshold> the context switch overhead increase to the extent
of taking the CPU utilization to 100%. You may argue that at this point that
throughput should decrease when you add more processes, but if you think
about it, context switches happen at a fixed rate, and regardless of how
many processes you have, the rate of context switches stays the same. But it
also means that processes (or containers) have to wait longer before they
can get scheduled - hence the drop in jitter.

Sapan

Thank you.
> Regards.
>
>    -- Daniel
>
> Sapan Bhatia wrote:
> > Hello Daniel,
> >
> > Here's a summary of the results. Please feel free to contact me for any
> > questions or clarifications.
> > The experiments were run between two 3GHz Xeons with 4 megabytes of
> > cache.
> >
> > Sapan
> >
> > ----
> >
> > The evaluation strategy used is to compare the effective performance
> > of containers running with their own copy of the protocol stack with
> > that of processes on native Linux. Any significant deviation should
> > imply a major overhead in the implementation. Specifically, we run
> > the following steps:
> >
> > (i) We start with 1 container pushing out or routing data at full
> > throttle and progressively increase the number of containers,
> > measuring the overall throughput, CPU utilization and jitter. This is
> > not a good microbenchmark because the length of a time slice allows
> > containers to send out a few dozen packets in every burst - meaning
> > that packet sends are not interleaved, and there is no added
> > contention in the stack. But it's a good macrobenchmark, because this
> > is what we'd like to do in the context of PlanetLab (run many slices
> > at once). The only additional thing it brings into play is the
> > context-switch overhead.
> >
> > Results:
> > - There is no degradation in throughput compared to native Linux even
> > when we drop the packet size from 1024->256->64b. This is surprising
> > - so I double checked that the sessions were not bottlenecked on the
> > receiver (which uses a kernel-mode packet accounter or because of
> > Ethernet flow control, which was disabled).
> >
> > - In general, jitter degrades drastically beyond about 20 concurrent
> > containers or processes (i.e., this happens on native Linux as well).
> > The degradation happens sooner with containers and is slightly higher
> > (about 10-15%). At first, it seemed that this might be the 'visible'
> > performance impact of using containers, but it turned out to be an
> > effect of (poor) scheduling.
> >
> > Jitter degrades with concurrency because of the difference in packet
> > delays across and between context switches. As long as processes are
> > scheduled more-or-less in a round-robin order, it is proportional to
> > the concurrency. However, with heavy loads, some processes start with
> > a disadvantage because they get preempted frequently at launch time,
> > and this effect ripples through till things stablize. It takes
> > between 30 seconds and 7 minutes to stablize. This time is
> > independent of the implementation and is a function of the
> > concurrency (and HZ).
> >
> > Jitter is slightly worse for containers because the context-switch
> > overhead is higher - which is because the cache footprint is higher.
> >
> > This (unstable scheduling) effect should hopefully go away, as Marc
> > suggested, with a better scheduler (eg. the one used for PlanetLab).
> >
> > - CPU utilization varies considerably. We measure the concurrency at
> > which the CPU utilization hits the ceiling.
> >
> > Results: For native Linux - 19
> > . NetNS - 1 (surprising, I mailed Eric
> > about this and after exchanging a few messages, he was satisfied but
> > said that things would improve. Also note that this is his first
> release,
> > and to the best of my knowledge the first time somebody other than
> > him has used his code). OpenVZ - 7.
> >
> > The increase in CPU utilization from 18-19 and 6-7 happens in a giant
> > leap (from 45% or so to 100%). It also corresponds to a jump in the
> > no. of L2 cache misses (which shoot up at this point) suggesting that
> > the cache footprint of all the processes now challenge the L2 cache.
> >
> > =>This number is a measure of the cache footprint of the
> > implementation rather than a measure of the actual overhead.
> >
> > This is a preliminary set of experiments which I'll enhance and rerun
> > continually with
> > more complex setups.
> >
> >
> >
> >
> >
> > On 2/20/07, Daniel Lezcano <dlezcano at fr.ibm.com> wrote:
> >>
> >> Hi all,
> >>
> >> I heard some of you did some benchmarking on the network virtualization
> >> provided by OpenVZ and by the Eric's Biederman patchset. Is it possible
> >> to have more informations on the results ?
> >>
> >> Thanks in advance
> >>
> >>         -- Daniel
> >>
> >> _______________________________________________
> >> Devel mailing list
> >> Devel at lists.planet-lab.org
> >> https://lists.planet-lab.org/mailman/listinfo/devel
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.planet-lab.org/pipermail/devel/attachments/20070221/1d0ed7d2/attachment.html