[Dirvish] CPU bound or disk I/O bound?

Jason Boxman jasonb at edseek.com
Wed Feb 2 14:21:46 PST 2005


On Saturday 08 January 2005 15:26, Jason Boxman wrote:
> I'm pulling files off a fileserver with gigabit to a box running Dirvish
> with gigabit.  Since upgrading to the faster network cards, I still
> experience relatively slow transfer speeds in the same neighborhood as my
> previous 100Mbps setup.  I checked the network with Iperf and pushed some
> files around via FTP and I can obtain around 20MB/s.
>

After investigation it has been explained to me that 20MB/s isn't unreasonable 
for my configuration.  `rsync` spends most of its time looking at my files to 
see if they've changed.  There are so many and they're so small, it does that 
more than it transfers files.  I'd suspect I need disks with faster seek 
times.

On Saturday 22 January 2005 00:12, Bryan J. Smith wrote:
> On Fri, 2005-01-21 at 14:10, Jason Boxman wrote:
> > That's what I was afraid of, but I wanted to ask anyway.
> > How significantly pegged are just by having more than a single device in
> > a shared bus topography, given the other devices are essentially 99%
> > idle?
>
> Only one device can use the bus at a time, _period_.
>
> Your NIC could be tying up the bus for extended periods, even if no
> other cards are being used.
>
> You yourself stated that your NIC could only get 212Mbps (~26.5MBps).
> That's probably by taking ahold of the PCI bus the entire time, just
> bursting to memory.  In fact, when it comes to IP traffic, some
> programmed I/O (PIO -- i.e., now the CPU is involved in the I/O) does
> occur.
>
> Now factor in the reality that your disk controller must now access the
> same bus.  Your NIC bursts to memory, but then has to give up the bus
> for the memory to write back on the same bus to the disk controller.  As
> I said, 15MBps now sounds very reasonable.
>
> > For example, the P(iii) 600MHz has an Adaptec U2W SCSI card, but during
> > my tests it was essentially idle, except for loading the requested
> > binaries like `iperf`.  The boxes were both in single user mode.  Does
> > the mere presence of another PCI card immediately result in taking a
> > performance hit?  If so, just how bad of a hit?
>
> You're missing the point.  An I/O bus just moves stuff around.  If a
> card is not using that bus efficiently, then the entire bus suffers.
> That's why point-to-point busses, be they reduced parallel or dedicated
> serial, busses are becoming popular.  Especially as more and more
> hardware is "software driven."
>
> > I only ask because theoretically I ought to be able to push 132MB/s on
> > just a 32-bit 33MHz bus, no?
>
> You'll _never_ get that.  At the most, you'll get clost to 100MBps with
> a non-blocking I/O controller that just bursts through raw block I/O.
> IP traffic is massively inefficient, and the cheaper the MAC of your NIC
> card, the worse the utilization of your PCI bus (because the CPU is more
> and more involved with the IP traffic over that bus).
>
> >  And I am only pushing around 30MB/s on the bus itself?
>
> You're only getting an _effective_ 26.5MBps out of your NIC on the bus.
> It's very likely that your NIC is talking to your CPU, and that's
> actually using more bus.  I.e., the CPU is being used more than a more
> intelligent MAC to handle the IP traffic.  Anytime you require your CPU
> to help your I/O, you are using more peripheral interconnect bandwidth
> _exponentially_.
>
> It's like trying to do hardware MPEG capture versus software.
> 704x480x16-bit at 30Hz is well under 30MBps via software.  Yet you drop
> frames.  Why?  Because not only is the burst from the video capture card
> to memory going on, but then memory to CPU, CPU back to memory and then
> finally back over the bus to disk.  Whereas a hardware MPEG is only
> pushing around 3MBps -- from card to memory to disk.  Not only 1/10th
> the throughput, but no additional steps, really improving probably
> 30-40x.
>
> Furthermore, I'm sure the NIC is not filling its PCI bursts because the
> cheap MAC can't crunch all the IP traffic fast enough.  That results in
> inefficient use of its time on the bus.  Once you add in the fact that
> the NIC has to give up the bus to the disk, it's not surprising that the
> end result is 1/2 the performance.
>
> > (NIC I/O and then disk I/O.)  What stops me from reaching higher speeds?
>
> Nothing is stopping you, your NIC is likely using all the throughput of
> the bus.  It's just not resulting in an _effective_ result of
> _effective_ throughput.
>
> > Is the shared bus topography as bad as, for example, EIDE when you try
> > to run a slave and master on a channel, a widely known 'no-no'
> > practice?
>
> That's actually worse.  Because the master/slave relationship was _only_
> designed for EIDE PIO and _never_ for ATA DMA transfers.  PCI was
> actually designed for shared, PCI DMA transfers.
>
> > I'm just trying to quantify where the remaining 100MB/s on my bus went
> > that I can't seem to measure?
>
> You can't measure it because it's being wasted.  What you get in usable
> is not always what the bus is capable of.  But that throughput _is_
> being used -- either inefficiently or one card is holding the bus and
> not bursting as much data through it as the bus is capable of.
>
> > Which cards changed where?
>
> Swap the NICs between the two systems, see what happens.
>
> > Yeah, you've been mentioning that often and after playing with lots of
> > hardware this past year I finally understand what you're talking about
> > and why it's so important.  Even so, I can't afford that kind of
> > equipment. ;) I could probably pickup an AMD Althon MPX chipset based
> > board through if that would make a difference.  The current
> > ServerWorksIIILE board would then become the backup server, replacing the
> > P(iii) 600MHz box.
>
> No need.  The ServerWorks IIILE is close to what the MPX can do.
>
> The ServerWorks IV and, especially, the AMD8xxx HyperTransport
> peripheral tunnels/bridges really blow either away.


More information about the Dirvish mailing list