[mdlug] script question

Thu Jan 9 21:42:14 EST 2014

Yeah the part that really slowed down my version is the search in the awk.
I did /..../time instead of 2014.  Im not sure how much faster it is with
2014 but I did notice a difference when I removed the hard coded year.  If
you have time, would you mind rerunning it with the 2014 instead of the
....  Im curious for the time trials.

On Thu, Jan 9, 2014 at 8:15 PM, Mat Enders <mat.enders at gmail.com> wrote:

>  David. My slightly modified version is about 5 times faster over all and
> using essentially the same system time.  Mine is more versatile as I can do
> the rotated out zipped logs just by changing the grep to zgrep. first the
> time trials then the modified version
>
> Yours
> $ time ./foo2.sh
> Unique Hits Per Hour
> 12:00 AM: 0
> 01:00 AM: 0
> 02:00 AM: 0
> 03:00 AM: 42
> 04:00 AM: 263
> 05:00 AM: 248
> 06:00 AM: 246
> 07:00 AM: 306
> 08:00 AM: 414
> 09:00 AM: 369
> 10:00 AM: 447
> 11:00 AM: 402
> 12:00 PM: 425
> 01:00 PM: 490
> 02:00 PM: 452
> 03:00 PM: 454
> 04:00 PM: 435
> 05:00 PM: 480
> 06:00 PM: 419
> 07:00 PM: 265
> 08:00 PM: 0
> 09:00 PM: 0
> 10:00 PM: 0
> 11:00 PM: 0
>
> real    0m28.311s
> user    0m27.568s
> sys     0m0.827s
>
> Mine
> $ time ./foo0.sh
> Unique Hits - Hour
> 0           - Midnight
> 0           - One AM
> 0           - Two AM
> 42          - Three AM
> 263         - Four AM
> 248         - Five AM
> 246         - Six AM
> 306         - Seven AM
> 414         - Eight AM
> 369         - Nine AM
> 447         - Ten AM
> 402         - Eleven AM
> 425         - Noon
> 490         - One PM
> 452         - Two PM
> 454         - Three PM
> 435         - Four PM
> 480         - Five PM
> 419         - Six PM
> 263         - Seven PM
> 0           - Eight PM
> 0           - Nine PM
> 0           - Ten PM
> 0           - Eleven PM
>
> real    0m5.305s
> user    0m5.044s
> sys     0m0.954s
>
> My slightly modified script
>
> #!/bin/bash
>
> {
> grep '2014:00:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Midnight/'
> grep '2014:01:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - One AM/'
> grep '2014:02:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Two AM/'
> grep '2014:03:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Three AM/'
> grep '2014:04:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Four AM/'
> grep '2014:05:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Five AM/'
> grep '2014:06:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Six AM/'
> grep '2014:07:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Seven AM/'
> grep '2014:08:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Eight AM/'
> grep '2014:09:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Nine AM/'
> grep '2014:10:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Ten AM/'
> grep '2014:11:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Eleven AM/'
> grep '2014:12:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Noon/'
> grep '2014:13:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - One PM/'
> grep '2014:14:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Two PM/'
> grep '2014:15:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Three PM/'
> grep '2014:16:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Four PM/'
> grep '2014:17:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Five PM/'
> grep '2014:18:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Six PM/'
> grep '2014:19:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Seven PM/'
> grep '2014:20:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Eight PM/'
> grep '2014:21:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Nine PM/'
> grep '2014:22:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Ten PM/'
> grep '2014:23:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print $1}'
> |sort |uniq |wc -l |sed 's/$/ - Eleven PM/'
> } |awk 'BEGIN {print "Unique Hits - Hour"}{printf "%-11s %-s %-s %-s\n",
> $1, $2, $3, $4}
>
>
> On Thu, Jan 9, 2014 at 11:01 AM, David Wood <dwood169 at gmail.com> wrote:
>
> > Here is how I would do it.  I would actually keep it in 24 hour format.
> > You could also remove the outfile and just have it echo to the screen.
>  The
> > benefit of leaving the outfile is that you could add it as part of the
> > logrotate and have this info for every day.  Its pretty much the same
> idea
> > as you were doing but I put it in a loop and combined the grep/awk with
> > just awk.
> >
> > #!/bin/bash
> >
> > #Some vars to make changes easier
> > LOGFILE="/home/USER/var/DOMAIN/logs/transfer.log"
> > OUTFILE="/tmp/uhitshr.$$"
> >
> > #Print the header for the file
> > echo "Unique Hits Per Hour" > ${OUTFILE}
> > for TIME in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19
> 20
> > 21 22 23
> > do
> >     #Convert the 24h time to 12h
> >     TIMEFORM=$(date -d ${TIME}:00 +%I":00 "%p)
> >     COUNT=$(awk ' $4 ~ "'/....:${TIME}'" { print $1 }' ${LOGFILE} | sort
> -u
> > | wc -l)
> >     echo "${TIMEFORM}: ${COUNT}" >> ${OUTFILE}
> > done
> >
> > more ${OUTFILE} && rm ${OUTFILE}
> >
> >
> > OUTPUT looks like this:
> > Unique Hits Per Hour
> > 12:00 AM: 79
> > 01:00 AM: 87
> > ...
> > 09:00 AM: 96
> > 10:00 AM: 95
> > 11:00 AM: 87
> > 12:00 PM: 102
> > 01:00 PM: 91
> > ...
> >
> >
> > On Thu, Jan 9, 2014 at 6:22 AM, Mat Enders <mat.enders at gmail.com> wrote:
> >
> > > I wrote this script to tell me the # of unique hits per hour to a
> > website.
> > >  So clients say look pingdom says my site was down for 14hrs we can no
> > you
> > > were getting hits.  Here is what I wrote, it also works as a one-liner
> > (how
> > > I originally wrote it) if you replace the ;'s with &&.  There has to
> be a
> > > better way to right this than the way I did it.  All suggestions must
> use
> > > bash only no perl.
> > >
> > > #!/bin/bash
> > >
> > > grep '2014:00:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Midnight/' > uhitshr;
> > > grep '2014:01:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - One AM/' >> uhitshr;
> > > grep '2014:02:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Two AM/' >> uhitshr;
> > > grep '2014:03:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Three AM/' >> uhitshr;
> > > grep '2014:04:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Four AM/' >> uhitshr;
> > > grep '2014:05:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Five AM/' >> uhitshr;
> > > grep '2014:06:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Six AM/' >> uhitshr;
> > > grep '2014:07:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Seven AM/' >> uhitshr;
> > > grep '2014:08:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Eight AM/' >> uhitshr;
> > > grep '2014:09:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Nine AM/' >> uhitshr;
> > > grep '2014:10:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Ten AM/' >> uhitshr;
> > > grep '2014:11:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Eleven AM/' >> uhitshr;
> > > grep '2014:12:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Noon/' >> uhitshr;
> > > grep '2014:13:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - One PM/' >> uhitshr;
> > > grep '2014:14:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Two PM/' >> uhitshr;
> > > grep '2014:15:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Three PM/' >> uhitshr;
> > > grep '2014:16:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Four PM/' >> uhitshr;
> > > grep '2014:17:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Five PM/' >> uhitshr;
> > > grep '2014:18:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Six PM/' >> uhitshr;
> > > grep '2014:19:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Seven PM/' >> uhitshr;
> > > grep '2014:20:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Eight PM/' >> uhitshr;
> > > grep '2014:21:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Nine PM/' >> uhitshr;
> > > grep '2014:22:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Ten PM/' >> uhitshr;
> > > grep '2014:23:' /home/USER/var/DOMAIN/logs/transfer.log |awk '{print
> $1}'
> > > |sort |uniq |wc -l |sed 's/$/ - Eleven PM/' >> uhitshr;
> > > cat uhitshr |awk 'BEGIN {print "Unique Hits - Hour"}{printf "%-11s %-s
> > %-s
> > > %-s\n", $1, $2, $3, $4}' && rm uhitshr
> > >
> > >
> > > --
> > > Mathew E. Enders
> > >
> > > "Where once Samba and Apache sold Linux to the world they are now just
> > part
> > > of the plumbing.  But that's OK, plumbers make good money."
> > > --Jeremy Allison
> > > _______________________________________________
> > > mdlug mailing list
> > > mdlug at mdlug.org
> > > http://mdlug.org/mailman/listinfo/mdlug
> > >
> > _______________________________________________
> > mdlug mailing list
> > mdlug at mdlug.org
> > http://mdlug.org/mailman/listinfo/mdlug
> >
>
>
>
> --
> Mathew E. Enders
>
> "Where once Samba and Apache sold Linux to the world they are now just part
> of the plumbing.  But that's OK, plumbers make good money."
> --Jeremy Allison
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug
>