[mdlug] script question

Fri Jan 10 00:44:55 EST 2014

David,

Later in the day o much larger file to parse.  with the hard coded 2014
yours did speed up a little and the amount of data slowed mine down a
little but mine is still about three times as fast.

You:
real    0m23.305s
user    0m22.247s
sys     0m1.156s

Me:
real    0m7.340s
user    0m7.149s
sys     0m1.071s

On Thu, Jan 9, 2014 at 9:42 PM, David Wood <dwood169 at gmail.com> wrote:

> Yeah the part that really slowed down my version is the search in the awk.
> I did /..../time instead of 2014.  Im not sure how much faster it is with
> 2014 but I did notice a difference when I removed the hard coded year.  If
> you have time, would you mind rerunning it with the 2014 instead of the
> ....  Im curious for the time trials.
>
>
> On Thu, Jan 9, 2014 at 8:15 PM, Mat Enders <mat.enders at gmail.com> wrote:
>
> >  David. My slightly modified version is about 5 times faster over all and
> > using essentially the same system time.  Mine is more versatile as I can
> do
> > the rotated out zipped logs just by changing the grep to zgrep. first the
> > time trials then the modified version
> >
> > Yours
> > $ time ./foo2.sh
> > Unique Hits Per Hour
> > 12:00 AM: 0
> > 01:00 AM: 0
> > 02:00 AM: 0
> > 03:00 AM: 42
> > 04:00 AM: 263
> > 05:00 AM: 248
> > 06:00 AM: 246
> > 07:00 AM: 306
> > 08:00 AM: 414
> > 09:00 AM: 369
> > 10:00 AM: 447
> > 11:00 AM: 402
> > 12:00 PM: 425
> > 01:00 PM: 490
> > 02:00 PM: 452
> > 03:00 PM: 454
> > 04:00 PM: 435
> > 05:00 PM: 480
> > 06:00 PM: 419
> > 07:00 PM: 265
> > 08:00 PM: 0
> > 09:00 PM: 0
> > 10:00 PM: 0
> > 11:00 PM: 0
> >
> > real    0m28.311s
> > user    0m27.568s
> > sys     0m0.827s
> >
> > Mine
> > $ time ./foo0.sh
> > Unique Hits - Hour
> > 0           - Midnight
> > 0           - One AM
> > 0           - Two AM
> > 42          - Three AM
> > 263         - Four AM
> > 248         - Five AM
> > 246         - Six AM
> > 306         - Seven AM
> > 414         - Eight AM
> > 369         - Nine AM
> > 447         - Ten AM
> > 402         - Eleven AM
> > 425         - Noon
> > 490         - One PM
> > 452         - Two PM
> > 454         - Three PM
> > 435         - Four PM
> > 480         - Five PM
> > 419         - Six PM
> > 263         - Seven PM
> > 0           - Eight PM
> > 0           - Nine PM
> > 0           - Ten PM
> > 0           - Eleven PM
> >
> > real    0m5.305s
> > user    0m5.044s
> > sys     0m0.954s
> >
> > My slightly modified script
> >
> > #!/bin/bash
> >
> > {
> > grep '2014:00:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Midnight/'
> > grep '2014:01:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - One AM/'
> > grep '2014:02:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Two AM/'
> > grep '2014:03:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Three AM/'
> > grep '2014:04:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Four AM/'
> > grep '2014:05:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Five AM/'
> > grep '2014:06:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Six AM/'
> > grep '2014:07:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Seven AM/'
> > grep '2014:08:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Eight AM/'
> > grep '2014:09:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Nine AM/'
> > grep '2014:10:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Ten AM/'
> > grep '2014:11:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Eleven AM/'
> > grep '2014:12:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Noon/'
> > grep '2014:13:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - One PM/'
> > grep '2014:14:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Two PM/'
> > grep '2014:15:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Three PM/'
> > grep '2014:16:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Four PM/'
> > grep '2014:17:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Five PM/'
> > grep '2014:18:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Six PM/'
> > grep '2014:19:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Seven PM/'
> > grep '2014:20:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Eight PM/'
> > grep '2014:21:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Nine PM/'
> > grep '2014:22:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Ten PM/'
> > grep '2014:23:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> > |sort |uniq |wc -l |sed 's/$/ - Eleven PM/'
> > } |awk 'BEGIN {print "Unique Hits - Hour"}{printf "%-11s %-s %-s %-s\n",
> > $1, $2, $3, $4}
> >
> >
> > On Thu, Jan 9, 2014 at 11:01 AM, David Wood <dwood169 at gmail.com> wrote:
> >
> > > Here is how I would do it.  I would actually keep it in 24 hour format.
> > > You could also remove the outfile and just have it echo to the screen.
> >  The
> > > benefit of leaving the outfile is that you could add it as part of the
> > > logrotate and have this info for every day.  Its pretty much the same
> > idea
> > > as you were doing but I put it in a loop and combined the grep/awk with
> > > just awk.
> > >
> > > #!/bin/bash
> > >
> > > #Some vars to make changes easier
> > > LOGFILE="/home/USER/var/DOMAIN/logs/transfer.log"
> > > OUTFILE="/tmp/uhitshr.$$"
> > >
> > > #Print the header for the file
> > > echo "Unique Hits Per Hour" > ${OUTFILE}
> > > for TIME in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19
> > 20
> > > 21 22 23
> > > do
> > >     #Convert the 24h time to 12h
> > >     TIMEFORM=$(date -d ${TIME}:00 +%I":00 "%p)
> > >     COUNT=$(awk ' $4 ~ "'/....:${TIME}'" { print $1 }' ${LOGFILE} |
> sort
> > -u
> > > | wc -l)
> > >     echo "${TIMEFORM}: ${COUNT}" >> ${OUTFILE}
> > > done
> > >
> > > more ${OUTFILE} && rm ${OUTFILE}
> > >
> > >
> > > OUTPUT looks like this:
> > > Unique Hits Per Hour
> > > 12:00 AM: 79
> > > 01:00 AM: 87
> > > ...
> > > 09:00 AM: 96
> > > 10:00 AM: 95
> > > 11:00 AM: 87
> > > 12:00 PM: 102
> > > 01:00 PM: 91
> > > ...
> > >
> > >
> > > On Thu, Jan 9, 2014 at 6:22 AM, Mat Enders <mat.enders at gmail.com>
> wrote:
> > >
> > > > I wrote this script to tell me the # of unique hits per hour to a
> > > website.
> > > >  So clients say look pingdom says my site was down for 14hrs we can
> no
> > > you
> > > > were getting hits.  Here is what I wrote, it also works as a
> one-liner
> > > (how
> > > > I originally wrote it) if you replace the ;'s with &&.  There has to
> > be a
> > > > better way to right this than the way I did it.  All suggestions must
> > use
> > > > bash only no perl.
> > > >
> > > > #!/bin/bash
> > > >
> > > > grep '2014:00:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Midnight/' > uhitshr;
> > > > grep '2014:01:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - One AM/' >> uhitshr;
> > > > grep '2014:02:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Two AM/' >> uhitshr;
> > > > grep '2014:03:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Three AM/' >> uhitshr;
> > > > grep '2014:04:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Four AM/' >> uhitshr;
> > > > grep '2014:05:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Five AM/' >> uhitshr;
> > > > grep '2014:06:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Six AM/' >> uhitshr;
> > > > grep '2014:07:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Seven AM/' >> uhitshr;
> > > > grep '2014:08:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Eight AM/' >> uhitshr;
> > > > grep '2014:09:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Nine AM/' >> uhitshr;
> > > > grep '2014:10:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Ten AM/' >> uhitshr;
> > > > grep '2014:11:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Eleven AM/' >> uhitshr;
> > > > grep '2014:12:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Noon/' >> uhitshr;
> > > > grep '2014:13:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - One PM/' >> uhitshr;
> > > > grep '2014:14:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Two PM/' >> uhitshr;
> > > > grep '2014:15:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Three PM/' >> uhitshr;
> > > > grep '2014:16:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Four PM/' >> uhitshr;
> > > > grep '2014:17:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Five PM/' >> uhitshr;
> > > > grep '2014:18:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Six PM/' >> uhitshr;
> > > > grep '2014:19:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Seven PM/' >> uhitshr;
> > > > grep '2014:20:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Eight PM/' >> uhitshr;
> > > > grep '2014:21:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Nine PM/' >> uhitshr;
> > > > grep '2014:22:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Ten PM/' >> uhitshr;
> > > > grep '2014:23:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> > $1}'
> > > > |sort |uniq |wc -l |sed 's/$/ - Eleven PM/' >> uhitshr;
> > > > cat uhitshr |awk 'BEGIN {print "Unique Hits - Hour"}{printf "%-11s
> %-s
> > > %-s
> > > > %-s\n", $1, $2, $3, $4}' && rm uhitshr
> > > >
> > > >
> > > > --
> > > > Mathew E. Enders
> > > >
> > > > "Where once Samba and Apache sold Linux to the world they are now
> just
> > > part
> > > > of the plumbing.  But that's OK, plumbers make good money."
> > > > --Jeremy Allison
> > > > _______________________________________________
> > > > mdlug mailing list
> > > > mdlug at mdlug.org
> > > > http://mdlug.org/mailman/listinfo/mdlug
> > > >
> > > _______________________________________________
> > > mdlug mailing list
> > > mdlug at mdlug.org
> > > http://mdlug.org/mailman/listinfo/mdlug
> > >
> >
> >
> >
> > --
> > Mathew E. Enders
> >
> > "Where once Samba and Apache sold Linux to the world they are now just
> part
> > of the plumbing.  But that's OK, plumbers make good money."
> > --Jeremy Allison
> > _______________________________________________
> > mdlug mailing list
> > mdlug at mdlug.org
> > http://mdlug.org/mailman/listinfo/mdlug
> >
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug
>

-- 
Mathew E. Enders

"Where once Samba and Apache sold Linux to the world they are now just part
of the plumbing.  But that's OK, plumbers make good money."
--Jeremy Allison