[mdlug] script question

Thu Jan 9 20:15:45 EST 2014

 David. My slightly modified version is about 5 times faster over all and
using essentially the same system time.  Mine is more versatile as I can do
the rotated out zipped logs just by changing the grep to zgrep. first the
time trials then the modified version

Yours
$ time ./foo2.sh
Unique Hits Per Hour
12:00 AM: 0
01:00 AM: 0
02:00 AM: 0
03:00 AM: 42
04:00 AM: 263
05:00 AM: 248
06:00 AM: 246
07:00 AM: 306
08:00 AM: 414
09:00 AM: 369
10:00 AM: 447
11:00 AM: 402
12:00 PM: 425
01:00 PM: 490
02:00 PM: 452
03:00 PM: 454
04:00 PM: 435
05:00 PM: 480
06:00 PM: 419
07:00 PM: 265
08:00 PM: 0
09:00 PM: 0
10:00 PM: 0
11:00 PM: 0

real    0m28.311s
user    0m27.568s
sys     0m0.827s

Mine
$ time ./foo0.sh
Unique Hits - Hour
0           - Midnight
0           - One AM
0           - Two AM
42          - Three AM
263         - Four AM
248         - Five AM
246         - Six AM
306         - Seven AM
414         - Eight AM
369         - Nine AM
447         - Ten AM
402         - Eleven AM
425         - Noon
490         - One PM
452         - Two PM
454         - Three PM
435         - Four PM
480         - Five PM
419         - Six PM
263         - Seven PM
0           - Eight PM
0           - Nine PM
0           - Ten PM
0           - Eleven PM

real    0m5.305s
user    0m5.044s
sys     0m0.954s

My slightly modified script

#!/bin/bash

{
grep '2014:00:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Midnight/'
grep '2014:01:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - One AM/'
grep '2014:02:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Two AM/'
grep '2014:03:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Three AM/'
grep '2014:04:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Four AM/'
grep '2014:05:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Five AM/'
grep '2014:06:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Six AM/'
grep '2014:07:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Seven AM/'
grep '2014:08:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Eight AM/'
grep '2014:09:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Nine AM/'
grep '2014:10:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Ten AM/'
grep '2014:11:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Eleven AM/'
grep '2014:12:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Noon/'
grep '2014:13:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - One PM/'
grep '2014:14:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Two PM/'
grep '2014:15:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Three PM/'
grep '2014:16:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Four PM/'
grep '2014:17:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Five PM/'
grep '2014:18:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Six PM/'
grep '2014:19:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Seven PM/'
grep '2014:20:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Eight PM/'
grep '2014:21:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Nine PM/'
grep '2014:22:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Ten PM/'
grep '2014:23:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
|sort |uniq |wc -l |sed 's/$/ - Eleven PM/'
} |awk 'BEGIN {print "Unique Hits - Hour"}{printf "%-11s %-s %-s %-s\n",
$1, $2, $3, $4}

On Thu, Jan 9, 2014 at 11:01 AM, David Wood <dwood169 at gmail.com> wrote:

> Here is how I would do it.  I would actually keep it in 24 hour format.
> You could also remove the outfile and just have it echo to the screen.  The
> benefit of leaving the outfile is that you could add it as part of the
> logrotate and have this info for every day.  Its pretty much the same idea
> as you were doing but I put it in a loop and combined the grep/awk with
> just awk.
>
> #!/bin/bash
>
> #Some vars to make changes easier
> LOGFILE="/home/USER/var/DOMAIN/logs/transfer.log"
> OUTFILE="/tmp/uhitshr.$$"
>
> #Print the header for the file
> echo "Unique Hits Per Hour" > ${OUTFILE}
> for TIME in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23
> do
>     #Convert the 24h time to 12h
>     TIMEFORM=$(date -d ${TIME}:00 +%I":00 "%p)
>     COUNT=$(awk ' $4 ~ "'/....:${TIME}'" { print $1 }' ${LOGFILE} | sort -u
> | wc -l)
>     echo "${TIMEFORM}: ${COUNT}" >> ${OUTFILE}
> done
>
> more ${OUTFILE} && rm ${OUTFILE}
>
>
> OUTPUT looks like this:
> Unique Hits Per Hour
> 12:00 AM: 79
> 01:00 AM: 87
> ...
> 09:00 AM: 96
> 10:00 AM: 95
> 11:00 AM: 87
> 12:00 PM: 102
> 01:00 PM: 91
> ...
>
>
> On Thu, Jan 9, 2014 at 6:22 AM, Mat Enders <mat.enders at gmail.com> wrote:
>
> > I wrote this script to tell me the # of unique hits per hour to a
> website.
> >  So clients say look pingdom says my site was down for 14hrs we can no
> you
> > were getting hits.  Here is what I wrote, it also works as a one-liner
> (how
> > I originally wrote it) if you replace the ;'s with &&.  There has to be a
> > better way to right this than the way I did it.  All suggestions must use
> > bash only no perl.
> >
> > #!/bin/bash
> >
> > grep '2014:00:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Midnight/' > uhitshr;
> > grep '2014:01:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - One AM/' >> uhitshr;
> > grep '2014:02:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Two AM/' >> uhitshr;
> > grep '2014:03:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Three AM/' >> uhitshr;
> > grep '2014:04:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Four AM/' >> uhitshr;
> > grep '2014:05:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Five AM/' >> uhitshr;
> > grep '2014:06:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Six AM/' >> uhitshr;
> > grep '2014:07:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Seven AM/' >> uhitshr;
> > grep '2014:08:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Eight AM/' >> uhitshr;
> > grep '2014:09:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Nine AM/' >> uhitshr;
> > grep '2014:10:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Ten AM/' >> uhitshr;
> > grep '2014:11:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Eleven AM/' >> uhitshr;
> > grep '2014:12:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Noon/' >> uhitshr;
> > grep '2014:13:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - One PM/' >> uhitshr;
> > grep '2014:14:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Two PM/' >> uhitshr;
> > grep '2014:15:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Three PM/' >> uhitshr;
> > grep '2014:16:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Four PM/' >> uhitshr;
> > grep '2014:17:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Five PM/' >> uhitshr;
> > grep '2014:18:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Six PM/' >> uhitshr;
> > grep '2014:19:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Seven PM/' >> uhitshr;
> > grep '2014:20:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Eight PM/' >> uhitshr;
> > grep '2014:21:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Nine PM/' >> uhitshr;
> > grep '2014:22:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Ten PM/' >> uhitshr;
> > grep '2014:23:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Eleven PM/' >> uhitshr;
> > cat uhitshr |awk 'BEGIN {print "Unique Hits - Hour"}{printf "%-11s %-s
> %-s
> > %-s\n", $1, $2, $3, $4}' && rm uhitshr
> >
> >
> > --
> > Mathew E. Enders
> >
> > "Where once Samba and Apache sold Linux to the world they are now just
> part
> > of the plumbing.  But that's OK, plumbers make good money."
> > --Jeremy Allison
> > _______________________________________________
> > mdlug mailing list
> > mdlug at mdlug.org
> > http://mdlug.org/mailman/listinfo/mdlug
> >
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug
>

-- 
Mathew E. Enders

"Where once Samba and Apache sold Linux to the world they are now just part
of the plumbing.  But that's OK, plumbers make good money."
--Jeremy Allison