[mdlug] Version control: tools and training

[Dave McMillan]

     Well, my management got their hands on some sales brochures (cue 
Dilbert comedic-horror reaction here), and latched onto a toolset called 
"VersionDog <https://issconnectivity.com/change-management/>", which 
certainly *sounds* good, but I'm pushing back on the rush to buy until 
we can get some hands-on demos.

     Let me see if I can break down my various files into a 
comprehensive category set:

1.    Robot languages (various, but almost all accessible to some degree 
as straight ASCII text)

2.    PLCs (all proprietary, all the time!  VersionDog's biggest selling 
point at the moment is that it handles many of the most popular PLC 
proprietary languages.  Still unsure how I'll have to handle some of the 
more obscure processors....)

3.    SQL Databases.  These contain a large mix of information, some 
common across all projects, some project-specific, some locally 
machine-specific, and (sadly) all of them open to changes in the field 
(for example, certain spare parts are not simply swappable, but have to 
have certain software offsets associated to their unique serial #s).  
VersionDog's support has been unclear as to how well they handle SQL, so far

4.    Windows executables (all in Delphi).  I don't know who decided 
that writing a hard-realtime control engine in Windows XP (now 7) was a 
good idea, but I'm stuck with this setup at least for the immediate future

5.    Delphi-esque "scripts" that run atop the realtime engine (#4).  
Basically a (very) trimmed-down implementation of Delphi

        The software we're responsible for (already has a large 
installed code base that has to be imported as our "version 1") breaks 
down broadly into:

1.    "Core" deployable libraries -- should be 100% common across *all* 
projects, updates need to be 100% backwards-compatible

2.    "Project" branches -- some particular projects require specialized 
modifications that can't be bundled with the Core, or may have to wait 
to be retrofitted later

3.    "Testing" branches.  This is where is really starts to get hairy.  
On the same machine, or group of machines, I may have multiple Testing 
branches of various modules (Core or Project level), each for a 
different improvement or bugfix, each being written and tested by 
different people, and often making different mods to the same files.  I 
can't *count* the number of times BugFix B got generally deployed after 
testing, only to discover that it had un-written changes to the same 
modules for BugFix A, regardless of how careful we were trying to be.  
And then figuring out what exactly changed, and when, and how to restore 
BFA without breaking BFB (while simultaneously working on C, D, and E in 
parallel)... let's just say I'm *really* sick of doing this entirely by 
hand, over and over again.  A really robust way of managing these 
branches and merging them, with tracking, is near the top of my 
Christmas list.

4.    "Machine" branches -- this gets a little vague, but there's 
certain customizations that have to be made per individual machine, even 
across machines that are, at the *design* level, identical.  And since 
these customizations can happen as parts of Core or Project level 
modules...  I do what I can to keep them segregated, but I can't force 
it across the board.  I do everything I can to keep these customizations 
restricted to my "config" files as opposed to my "executable" files, but 
not all of the processors we use give us that luxury (plus it makes some 
sorts of data handling very cumbersome -- like parsing XML in a 
programming language that has only a few very primitive string-handling 
functions).

5.    "Field" changes.  I can write universally-deployable Core code all 
day long, but on the production floor, the exact location a robot picks 
up a tool or applies a process often *has* to be unique on a 
robot-by-robot basis -- no matter how hard the construction team tries, 
there's always some small physical differences between work cells that 
have to be compensated for by hand, in software.  Again, I try to 
segregate the modules that are going to contain any of these changes, 
but that still leaves me with modules that *have* to contain a mix of 
"global" code and "field" code -- I not only have to track these Field 
changes over time on a machine-by-machine basis, but also prevent my 
change-monitor tools from red-flagging these "harmless" changes (so far, 
about the best I've been able to do is put a comment-line divider in the 
file, with anything above the line being "hard" code, and anything below 
the line being "field adjustable").

     At the end of the day, though, I think my biggest problem is still 
going to be getting the entire team *using* whatever tools we use, and 
trained in all using them the same way.  There's a LOT of old habits to 
break, and unending time pressure from On High to "don't bother doing it 
Right, do it Right Now."  Whatever tools we end up using need to be as 
easy and painless to learn and use as possible, or else people will just 
start "forgetting" to use any of it -- I can't impose any requirements 
from above, this "cultural shift" is going to have to be a bottom-up 
thing.  It's not that many of my team will be *unwilling* -- rather, 
they're already so driven to the wall that anything that makes their 
jobs even slightly harder, and isn't THE MACHINE IS ON FIRE critical, is 
going to simply get constantly preempted by emergency fire-fighting 
until it eventually never gets done, and any training gets forgotten 
from lack of use (and then management comes down on *us* for not being 
able to simply copy&paste from project to project).  Anything I try to 
get the team to use *really* needs to have an immediate and visible 
benefit to making their lives easier, or at minimum be workload-neutral.

On 2/21/2017 8:34 AM, Dave McMillan wrote:
>
>     This... is going to be rather off-topic, but this is the most 
> likely group I know of to ask about this.
>
>     I work for a Detroit-area industrial-automation company that's 
> never really been a software development house, but now finds itself 
> becoming one unexpectedly.  So I've been tasked with finding a way to 
> bring us into the 21st century. :-\
>
>     So, I have a small group of "developers" that are essentially 
> self-taught, and have spent their careers doing mostly one-off 
> software projects and managing their own versions and backups 
> independently (if at all), often mostly inside their heads.  Plus a 
> larger number of "debuggers" that have less programming skill, but get 
> much more time on the active machines, who will be the people who do 
> most of the bug-finding and reporting, and installing/testing updates 
> and bugfixes pushed out by the developers.  And customers that want 
> levels of traceability and documentation that, frankly, we've *never* 
> done.
>
>     Just to ice the cake, this being industrial automation means that 
> a lot of our source code is tied up in proprietary languages that are 
> recorded in proprietary binary formats and not generally accessible by 
> 3rd-party tools like (for example) git.  So I'm in the position of 
> possibly being forced to maintain an entire *ecosystem* of different 
> version-control systems with brand-specific branches for different 
> pieces of equipment.
>
>     And did I mention that while *some* software has to be 
> centrally-controlled, other parts of the *same* software have to be 
> field-editable on a machine-by-machine basis?  And I have to track 
> *all* of that.
>
>      So, right now, my biggest concern is less the *tools,* than the 
> *training.*  Whatever tool (or tool set) we end up choosing (and there 
> I'm at the mercy of the beancounters), I think my *bigger* problem is 
> the total lack of any in-house *culture* for version control, 
> multi-person developer/debug teams, and obsessive 
> tracking/documentation/etc.  And I, frankly, am just as clueless. So, 
> I'll put it to people who (hopefully) know more about this kind of 
> thing than I do:  where do I *start*?  Are there training programs, or 
> certification courses, or even just a *really good* O'Reilly book that 
> I can use to get myself off square one?  And does anyone know of any 
> sort of training curriculum for bringing this sort of cultural shift 
> into an existing "herd of cats" development team?
>
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug