[mdlug] Piping, redirection and shellscipts: 3/5/2025 7pm Eastern Standard time

[Steve Litt]

Hi Aaron,

If this were something I were running regularly, I'd us awk or Python.
Thing is, this was a one-off process, and collapsing the scraping of
the web screen was about 1/2 minute of work in Vim.

By the way, is there a lint utility for awk? A lint utility for awk
would be very, very handy, because as an awk program gets past 50
lines, very arcane things start to go wrong. Which is why unless I
expect it to be extremely simple, I just go straight to Python.

SteveT


Aaron Kulkis said on Wed, 5 Mar 2025 01:41:22 -0500

>Instead of using vim, use awk.
>
>On Sun, Mar 2, 2025 at 6:07 PM Steve Litt <slitt at troubleshooters.com>
>wrote:
>
>> Hi all,
>>
>> Where: GoLUG: https://meet.jit.si/golug
>> When: Wednesday, 3/5/2025 7pm sharp Eastern Standard time
>>       Arrive 15 minutes early for Microphone check & discussion
>> Who: Every attendee. This is a free-for-all discussion
>>
>> I've taken piping, redirection and shellscripts for granted the last
>> two decades. I'm probably not the only one. And it's probable that
>> each of us uses these things a little differently and could benefit
>> from everybody else's use of piping, redirection and shellscripts.
>>
>> Piping and redirection, possibly aided by a good text editor, bestow
>> on us a spectacular ad-hoc query system. Yesterday I was tasked with
>> getting every domain name I owned into my list at
>> 444domains.com/domains , which is created by a shellscript and Python
>> program that read a Yaml file and convert it to the web page. So I
>> had to do the following:
>>
>> * Screen scrape all my domain names from ionos, put in file
>>
>> * Screen scrape all my domain names from Spaceship.Com, put in
>> another file
>>
>> * Use Vim to remove everything but the domains themselves
>>
>> * Sort both files
>>
>> * Create a Python program to sort my Yaml file
>>
>> * Run diff between each file and the sorted domains of the Yaml file
>>     - The output with the left arrows need to be added to Yaml file
>>
>> * Re-sort the Yaml file
>>
>> * Run the shellscript to create the web page.
>>
>> This sounds straightforward, and if I were infallible it would be.
>> But I made lots of mistakes, including forgetting to sort one file,
>> producing lots of duplicates in the diff. For this reason, having the
>> system be ad-hoc, complete with sed, diff, grep, etc was wonderful.
>>
>> And with the knowledge learned, if I expect this to be an ongoing
>> thing, which I don't, I could put the whole process into a
>> shellscript to be run at will.
>>
>> The world has been trying for decades, with lackluster success, to
>> create "reusable code". To this day, piping, redirection,
>> shellscripts, and a good curation of fast, tested and known good
>> processes like grep, sed, cut, sort, etc yield by far the quickest
>> and easiest way to reuse code. Nowadays you can even go GUI by using
>> the zenity command.
>>
>> Mine is just one example. I bet every one of you has a unique example
>> of using piping, redirection and shellscripts to do something that
>> would have been a massive program in Perl, Python, etc. The GoLUG
>> meeting Wednesday night is where you can talk about your unique
>> examples, and learn from others.
>>
>> Also up for discussion are our favorite shells (bash, /bin/sh, dash,
>> ksh, csh, etc), and the role of shellscript checkers like shellcheck.
>> And the age old question: When is something so big that it should be
>> done in C, Python, Rust, Lazarus, etc.
>>
>> This link (thanks Kyle) https://www.youtube.com/watch?v=tc4ROCJYbm0
>> has given me the inspiration to try harder to do it with piping,
>> redirection, shellscripts and utilities before busting out my Python
>> interpreter, and if I do go Python, do it as a "do one thing" process
>> inputting from stdin and outputting to stdout.
>>
>> Hope to see you there.
>>
>> SteveT
>>
>> Steve Litt
>> GoLUG Publicity Coordinator
>> _______________________________________________
>> mdlug mailing list
>> mdlug at mdlug.org
>> http://mdlug.org/mailman/listinfo/mdlug  
>_______________________________________________
>mdlug mailing list
>mdlug at mdlug.org
>http://mdlug.org/mailman/listinfo/mdlug