[mdlug] ESX3 IO Problems
Morris, Tim
tmorris at ugs.com
Wed Aug 8 15:51:39 EDT 2007
I don't have a script yet, I was just dd'ing up a bunch of different
size files and testing the transfer times (date && cp filea filea.00 &&
date, or time cp filea filea.00).
-- Tim Morris - UGS PLM Software
-----Original Message-----
From: mdlug-bounces at mdlug.org [mailto:mdlug-bounces at mdlug.org] On Behalf
Of Dan Pritts
Sent: Wednesday, August 08, 2007 11:12 AM
To: MDLUG's Main discussion list
Subject: Re: [mdlug] ESX3 IO Problems
Agreed - that really sucks. It might be within the realm of what I saw
happenning as i transferred vm's between systems.
If you send me your script I'll try it on lightly-loaded 2850 here.
On Tue, Aug 07, 2007 at 01:52:53PM -0500, Morris, Tim wrote:
> The problems I'm seeing are on the service console. I don't have any
> VMs created. I know the console is limited, but to 5MB/s? ESX 2.5.2
> performed much better in console I/O than that (on the same hardware).
> The problem is that I have 400GB of data in VMs to move to this
> system, so 5MB/s isn't going to cut it... I might have to revert to
> OpenSuse/Vmware Server if that is the case. Vmkfstools doesn't do much
> better. Is this Vmware's way of forcing you to buy VirtualCenter?
>
>
> -- Tim Morris - UGS PLM Software
>
> -----Original Message-----
> From: mdlug-bounces at mdlug.org [mailto:mdlug-bounces at mdlug.org] On
> Behalf Of Dan Pritts
> Sent: Tuesday, August 07, 2007 1:13 PM
> To: MDLUG's Main discussion list
> Subject: Re: [mdlug] ESX3 IO Problems
>
> I'm hardly a guru but you haven't said where you are doing the I/O,
> within the service console, or within guests, or what.
>
> I know that the service console is just not optimized to do disk I/O
> and you can expect it to be slow when using standard unix commands.
>
> I assume you've checked on the load using the vmware management
console.
> It can show you what each guest is doing i/o wise.
>
> beyond that i don't know that i can offer much advice, but there seem
> to be a lot of knowledgeable people on the forums at vmware.com.
>
> On Fri, Aug 03, 2007 at 10:18:39AM -0500, Morris, Tim wrote:
> > Are there any ESX gurus out there?
> >
> > I have a bad I/O problem with our ESX 3.0.2 build 52542 server that
> > I thought I would ask about. The same problem occurs when using
> > 3.0.1-32029 (unpatched).
> >
> > Host Computer:
> > Dell Poweredge 2850 BIOS A06
> > 2x 3.0 GHz Xeon Procs (HT disabled)
> > 4GB RAM
> > Dual Intel 8254NXX Gigabit Ethernet onboard PERC 4E/Di 2x 146GB
> > SCSI3
> > (Maxtor) drives internal (RAID 1)
> > 1 LD = ~139GB
> > PERC 4E/Dc
> > Powervault 220VS w/ 10x 146GB SCSI3 (Maxtor) drives (RAID 5 w/ 2
> > hotspares)
> > 1LD = 979MB
> > All components have been updated to the latest availble firmware
> > from Dell (even all 12 drives!)
> >
> > My problem is any type of file operation seems to hang every
50-200MB.
> > cp, ftp, scp, everything.
> > You can see it with an ftp hash especially. It hangs for a second,
> > then takes off again.
> >
> > I have put together some transfer stats for you to look at
(attached).
> >
> > Any suggestions on where to go from here?
> > Any Performance testing/diagnostic utils/tips?
> > Any more info needed?
> > Anyone run into this before?
> >
> > I could really use some help on this... Any suggestions would be
> > helpful.
> >
> > --------------------------------------------------
> > Tim Morris
> > Systems Engineer
> > Siemens Automation and Drives
> > UGS PLM Software
> >
>
> Content-Description: perf_test.txt
> >
> **********************************************************************
> **
> ***************
> > ** ESX3SRVR
> **
> > ********************************************************************
> > **
> > ***************** dd if=/dev/zero bs=1000 count=100000 of=small_file
> > dd if=/dev/zero bs=1000 count=500000 of=medium_file dd if=/dev/zero
> > bs=1000 count=1000000 of=large_file # ls -l
> > -rw-r--r-- 1 root root 1000000000 Aug 3 02:13 large_file
*
> (953.67MB)
> > -rw-r--r-- 1 root root 500000000 Aug 3 02:10 medium_file
*
> (476.84MB)
> > -rw-r--r-- 1 root root 100000000 Aug 3 02:09 small_file
*
> (95.37MB)
> >
> > CP RESULTS (performed in random order):
> > (date && cp -f small_file small_file.00 && date) Test 1: Start:
> > 02:43:41 End: 02:43:50 * (10.60MB/s) Test 2: Start: 02:44:07 End:
> > 02:44:12 * (19.07MB/s) Test 3: Start: 03:09:47 End: 03:10:08 *
> > (4.54MB/s)
> >
> > (date && cp -f medium_file medium_file.00 && date) Test 1: Start:
> > 02:44:30 End: 02:46:24 * (4.18MB/s) Test 2: Start: 02:52:56 End:
> > 02:54:49 * (4.22MB/s) Test 3: Start: 03:19:10 End: 03:21:04 *
> > (4.18MB/s)
> >
> > (date && cp -f large_file large_file.00 && date) Test 1: Start:
> > 02:47:20 End: 02:51:09 * (4.16MB/s) Test 2: Start: 03:02:17 End:
> > 03:06:12 * (4.06MB/s) Test 3: Start: 03:10:52 End: 03:14:40 *
> > (4.18MB/s)
> >
> > FTP (localhost) Results (performed in random order):
> > large_file
> > 1000000000 bytes received in 222 secs (4.4e+03 Kbytes/sec) *
> > (4.30MB/s) 1000000000 bytes received in 220 secs (4.4e+03
> > Kbytes/sec)
> > * (4.34MB/s) 1000000000 bytes received in 223 secs (4.4e+03
> > Kbytes/sec) * (4.28MB/s)
> >
> > medium_file
> > 500000000 bytes received in 111 secs (4.4e+03 Kbytes/sec) *
> > (4.30MB/s)
>
> > 500000000 bytes received in 106 secs (4.6e+03 Kbytes/sec) *
> > (4.50MB/s)
>
> > 500000000 bytes received in 109 secs (4.5e+03 Kbytes/sec) *
> > (4.37MB/s)
> >
> > small_file
> > 100000000 bytes received in 20.2 secs (4.8e+03 Kbytes/sec) *
> > (4.72MB/s) 100000000 bytes received in 5.46 secs (1.8e+04
> > Kbytes/sec)
> * (17.47MB/s) ! cached?
> > 100000000 bytes received in 13.6 secs (7.2e+03 Kbytes/sec) *
> > (7.01MB/s) 100000000 bytes received in 16.8 secs (5.8e+03
> > Kbytes/sec)
> > * (5.68MB/s)
> >
> >
> >
> **********************************************************************
> **
> ***************
> > ** MYSRV (My OpenSuse 10.2 box at home... 2.8GHz Celery, 1 IDE
Disk,
> 512MB RAM) **
> > ********************************************************************
> > **
> > ***************** dd if=/dev/zero bs=1000 count=100000 of=small_file
> > dd if=/dev/zero bs=1000 count=500000 of=medium_file dd if=/dev/zero
> > bs=1000 count=1000000 of=large_file # ls -l
> > -rw-r--r-- 1 root root 1000000000 Aug 2 22:02 large_file *
> (953.67MB)
> > -rw-r--r-- 1 root root 500000000 Aug 2 22:01 medium_file *
> (476.84MB)
> > -rw-r--r-- 1 root root 100000000 Aug 2 22:07 small_file *
> (95.37MB)
> >
> > CP RESULTS (performed in random order):
> > (date && cp small_file small_file.00 && date) Test 1: Start:
> > 22:09:29
> > End: 22:09:29 * (-) Test 2: Start: 23:00:57 End: 23:00:59 *
> > (47.69MB/s) Test 3: Start: 23:06:49 End: 23:06:50 * (95.37MB/s
> >
> > (date && cp medium_file medium_file.00 && date) Test 1: Start:
> > 22:09:47 End: 22:10:11 * (19.87MB/s) Test 2: Start: 23:01:12 End:
> > 23:01:44 * (14.90MB/s) Test 3: Start: 23:08:33 End: 23:09:03 *
> > (15.89MB/s)
> >
> > (date && cp large_file large_file.00 && date) Test 1: Start:
> > 22:10:29
> > End: 22:11:32 * (15.14MB/s) Test 2: Start: 23:03:25 End: 23:04:29 *
> > (14.90MB/s) Test 3: Start: 23:07:04 End: 23:08:09 * (14.67MB/s)
> >
> > FTP (localhost) Results (performed in random order):
> > large_file
> > 1000000000 bytes received in 00:50 * (18.90 MB/s) 1000000000 bytes
> > received in 00:48 * (19.61 MB/s) 1000000000 bytes received in 00:51
> > *
> > (18.58 MB/s)
> >
> > medium_file
> > 500000000 bytes received in 00:23 * (19.93 MB/s) 500000000 bytes
> > received in 00:25 * (19.06 MB/s) 500000000 bytes received in 00:22 *
> > (20.89 MB/s)
> >
> > small_file
> > 100000000 bytes received in 00:03 * (27.69 MB/s) 100000000 bytes
> > received in 00:03 * (27.17 MB/s) 100000000 bytes received in 00:03 *
> > (28.56 MB/s)
>
> > _______________________________________________
> > mdlug mailing list
> > mdlug at mdlug.org
> > http://mdlug.org/mailman/listinfo/mdlug
>
> danno
> --
> dan pritts
> danno at umich.edu
> 734-929-9770
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.476 / Virus Database: 269.11.8/940 - Release Date:
> 8/6/2007
> 4:53 PM
>
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug
danno
--
dan pritts
danno at umich.edu
734-929-9770
_______________________________________________
mdlug mailing list
mdlug at mdlug.org
http://mdlug.org/mailman/listinfo/mdlug
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.476 / Virus Database: 269.11.8/941 - Release Date: 8/7/2007
4:06 PM
More information about the mdlug
mailing list