[mdlug] ESX3 IO Problems

Tue Aug 7 13:12:46 EDT 2007

I'm hardly a guru but you haven't said where you are doing the I/O, 
within the service console, or within guests, or what.

I know that the service console is just not optimized to do disk
I/O and you can expect it to be slow when using standard unix
commands.

I assume you've checked on the load using the vmware management
console.  It can show you what each guest is doing i/o wise.

beyond that i don't know that i can offer much advice, but there
seem to be a lot of knowledgeable people on the forums at vmware.com.

On Fri, Aug 03, 2007 at 10:18:39AM -0500, Morris, Tim wrote:
> Are there any ESX gurus out there?
>  
> I have a bad I/O problem with our ESX 3.0.2 build 52542 server that I
> thought I would ask about. The same problem occurs when using
> 3.0.1-32029 (unpatched).
>  
> Host Computer: 
> Dell Poweredge 2850 BIOS A06
> 2x 3.0 GHz Xeon Procs (HT disabled)
> 4GB RAM
> Dual Intel 8254NXX Gigabit Ethernet onboard
> PERC 4E/Di
> 2x 146GB SCSI3 (Maxtor) drives internal (RAID 1) 
> 1 LD = ~139GB
> PERC 4E/Dc
> Powervault 220VS w/ 10x 146GB SCSI3 (Maxtor) drives (RAID 5 w/ 2
> hotspares) 
> 1LD = 979MB
> All components have been updated to the latest availble firmware from
> Dell (even all 12 drives!)
>  
> My problem is any type of file operation seems to hang every 50-200MB.
> cp, ftp, scp, everything.
> You can see it with an ftp hash especially. It hangs for a second, then
> takes off again. 
>  
> I have put together some transfer stats for you to look at (attached).
>  
> Any suggestions on where to go from here? 
> Any Performance testing/diagnostic utils/tips?
> Any more info needed?
> Anyone run into this before? 
>  
> I could really use some help on this...  Any suggestions would be
> helpful.
>  
> --------------------------------------------------
> Tim Morris
> Systems Engineer
> Siemens Automation and Drives
> UGS PLM Software
>  

Content-Description: perf_test.txt
> ***************************************************************************************
> **                               ESX3SRVR                                            **
> ***************************************************************************************
> dd if=/dev/zero bs=1000 count=100000 of=small_file
> dd if=/dev/zero bs=1000 count=500000 of=medium_file
> dd if=/dev/zero bs=1000 count=1000000 of=large_file
> # ls -l
> -rw-r--r--    1 root     root     1000000000 Aug  3 02:13 large_file * (953.67MB)
> -rw-r--r--    1 root     root     500000000 Aug  3 02:10 medium_file * (476.84MB)
> -rw-r--r--    1 root     root     100000000 Aug  3 02:09 small_file  * (95.37MB)
> 
> CP RESULTS (performed in random order):
> (date && cp -f small_file small_file.00 && date)
> Test 1: Start: 02:43:41 End: 02:43:50 * (10.60MB/s)
> Test 2: Start: 02:44:07 End: 02:44:12 * (19.07MB/s)
> Test 3: Start: 03:09:47 End: 03:10:08 * (4.54MB/s)
> 
> (date && cp -f medium_file medium_file.00 && date)
> Test 1: Start: 02:44:30 End: 02:46:24 * (4.18MB/s)
> Test 2: Start: 02:52:56 End: 02:54:49 * (4.22MB/s)
> Test 3: Start: 03:19:10 End: 03:21:04 * (4.18MB/s)
> 
> (date && cp -f large_file large_file.00 && date)
> Test 1: Start: 02:47:20 End: 02:51:09 * (4.16MB/s)
> Test 2: Start: 03:02:17 End: 03:06:12 * (4.06MB/s)
> Test 3: Start: 03:10:52 End: 03:14:40 * (4.18MB/s)
> 
> FTP (localhost) Results (performed in random order):
> large_file
> 1000000000 bytes received in 222 secs (4.4e+03 Kbytes/sec) * (4.30MB/s)
> 1000000000 bytes received in 220 secs (4.4e+03 Kbytes/sec) * (4.34MB/s)
> 1000000000 bytes received in 223 secs (4.4e+03 Kbytes/sec) * (4.28MB/s)
> 
> medium_file
> 500000000 bytes received in 111 secs (4.4e+03 Kbytes/sec) * (4.30MB/s)
> 500000000 bytes received in 106 secs (4.6e+03 Kbytes/sec) * (4.50MB/s)
> 500000000 bytes received in 109 secs (4.5e+03 Kbytes/sec) * (4.37MB/s)
> 
> small_file
> 100000000 bytes received in 20.2 secs (4.8e+03 Kbytes/sec) * (4.72MB/s)
> 100000000 bytes received in 5.46 secs (1.8e+04 Kbytes/sec) * (17.47MB/s)  ! cached?
> 100000000 bytes received in 13.6 secs (7.2e+03 Kbytes/sec) * (7.01MB/s)
> 100000000 bytes received in 16.8 secs (5.8e+03 Kbytes/sec) * (5.68MB/s)
> 
> 
> ***************************************************************************************
> **   MYSRV (My OpenSuse 10.2 box at home... 2.8GHz Celery, 1 IDE Disk, 512MB RAM)    **
> ***************************************************************************************
> dd if=/dev/zero bs=1000 count=100000 of=small_file
> dd if=/dev/zero bs=1000 count=500000 of=medium_file
> dd if=/dev/zero bs=1000 count=1000000 of=large_file
> # ls -l
> -rw-r--r-- 1 root    root 1000000000 Aug  2 22:02 large_file  * (953.67MB)
> -rw-r--r-- 1 root    root  500000000 Aug  2 22:01 medium_file * (476.84MB)
> -rw-r--r-- 1 root    root  100000000 Aug  2 22:07 small_file  * (95.37MB)
> 
> CP RESULTS (performed in random order):
> (date && cp small_file small_file.00 && date)
> Test 1: Start: 22:09:29 End: 22:09:29 * (-)
> Test 2: Start: 23:00:57 End: 23:00:59 * (47.69MB/s)
> Test 3: Start: 23:06:49 End: 23:06:50 * (95.37MB/s
> 
> (date && cp medium_file medium_file.00 && date)
> Test 1: Start: 22:09:47 End: 22:10:11 * (19.87MB/s)
> Test 2: Start: 23:01:12 End: 23:01:44 * (14.90MB/s)
> Test 3: Start: 23:08:33 End: 23:09:03 * (15.89MB/s)
> 
> (date && cp large_file large_file.00 && date)
> Test 1: Start: 22:10:29 End: 22:11:32 * (15.14MB/s)
> Test 2: Start: 23:03:25 End: 23:04:29 * (14.90MB/s)
> Test 3: Start: 23:07:04 End: 23:08:09 * (14.67MB/s)
> 
> FTP (localhost) Results (performed in random order):
> large_file
> 1000000000 bytes received in 00:50 * (18.90 MB/s)
> 1000000000 bytes received in 00:48 * (19.61 MB/s)
> 1000000000 bytes received in 00:51 * (18.58 MB/s)
> 
> medium_file
> 500000000 bytes received in 00:23 * (19.93 MB/s)
> 500000000 bytes received in 00:25 * (19.06 MB/s)
> 500000000 bytes received in 00:22 * (20.89 MB/s)
> 
> small_file
> 100000000 bytes received in 00:03 * (27.69 MB/s)
> 100000000 bytes received in 00:03 * (27.17 MB/s)
> 100000000 bytes received in 00:03 * (28.56 MB/s)

> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug

danno
--
dan pritts
danno at umich.edu
734-929-9770