[mdlug] Piping, redirection and shellscipts: 3/5/2025 7pm Eastern Standard time

[Aaron Kulkis]

Instead of using vim, use awk.

On Sun, Mar 2, 2025 at 6:07 PM Steve Litt <slitt at troubleshooters.com> wrote:

> Hi all,
>
> Where: GoLUG: https://meet.jit.si/golug
> When: Wednesday, 3/5/2025 7pm sharp Eastern Standard time
>       Arrive 15 minutes early for Microphone check & discussion
> Who: Every attendee. This is a free-for-all discussion
>
> I've taken piping, redirection and shellscripts for granted the last
> two decades. I'm probably not the only one. And it's probable that
> each of us uses these things a little differently and could benefit
> from everybody else's use of piping, redirection and shellscripts.
>
> Piping and redirection, possibly aided by a good text editor, bestow on
> us a spectacular ad-hoc query system. Yesterday I was tasked with
> getting every domain name I owned into my list at
> 444domains.com/domains , which is created by a shellscript and Python
> program that read a Yaml file and convert it to the web page. So I had
> to do the following:
>
> * Screen scrape all my domain names from ionos, put in file
>
> * Screen scrape all my domain names from Spaceship.Com, put in another
>   file
>
> * Use Vim to remove everything but the domains themselves
>
> * Sort both files
>
> * Create a Python program to sort my Yaml file
>
> * Run diff between each file and the sorted domains of the Yaml file
>     - The output with the left arrows need to be added to Yaml file
>
> * Re-sort the Yaml file
>
> * Run the shellscript to create the web page.
>
> This sounds straightforward, and if I were infallible it would be. But
> I made lots of mistakes, including forgetting to sort one file,
> producing lots of duplicates in the diff. For this reason, having the
> system be ad-hoc, complete with sed, diff, grep, etc was wonderful.
>
> And with the knowledge learned, if I expect this to be an ongoing
> thing, which I don't, I could put the whole process into a shellscript
> to be run at will.
>
> The world has been trying for decades, with lackluster success, to
> create "reusable code". To this day, piping, redirection, shellscripts,
> and a good curation of fast, tested and known good processes like grep,
> sed, cut, sort, etc yield by far the quickest and easiest way to reuse
> code. Nowadays you can even go GUI by using the zenity command.
>
> Mine is just one example. I bet every one of you has a unique example
> of using piping, redirection and shellscripts to do something that
> would have been a massive program in Perl, Python, etc. The GoLUG
> meeting Wednesday night is where you can talk about your unique
> examples, and learn from others.
>
> Also up for discussion are our favorite shells (bash, /bin/sh, dash,
> ksh, csh, etc), and the role of shellscript checkers like shellcheck.
> And the age old question: When is something so big that it should be
> done in C, Python, Rust, Lazarus, etc.
>
> This link (thanks Kyle) https://www.youtube.com/watch?v=tc4ROCJYbm0
> has given me the inspiration to try harder to do it with piping,
> redirection, shellscripts and utilities before busting out my Python
> interpreter, and if I do go Python, do it as a "do one thing" process
> inputting from stdin and outputting to stdout.
>
> Hope to see you there.
>
> SteveT
>
> Steve Litt
> GoLUG Publicity Coordinator
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug