[mdlug] Version control: tools and training

[Dave McMillan]

     ....whew.  I think I'm going to have to read this 3-4 times to get 
a handle on it.  For one thing, I think I need a new dictionary.  :)  
There's definitely several terms of art with which I am unfamiliar.  Is 
there a "Version Control for Idiots (or management)" type of book?  
Something that lays out the terminology, and has a fit-for-management 
explanation of the whys and hows?  At some point soon, I'm going to need 
to boil this down to baby talk in order to get the bean counters to be 
willing to spend money on this.

     (side note:  I should have mentioned, cloud services are verboten.  
Some customers will have actual security-clearance issues, and even the 
ones that don't are generally utter hardcases about information 
control.  So even in-house, for our own dev teams, I can't use cloud 
services like GitHub, and most customer sites will block any remote 
access.  So tracebility in the field will have to be sneaker-net'd back 
home.  And whatever I set up will also have to win over the support of 
our corporate IT shop, which is a classical "office computers running 
Windows and Office only" world)

     SDCL -- that, I think, is part of our problem.  We not only don't 
have anything like that, we as a company don't even have a *concept* of 
it.  At the management level, software is just like hardware -- beat on 
it until it works, then ship it to the customer.  Next project, start 
from scratch.  I *came up* through that system, and have all the bad 
habits from it.  It *worked*, but only because of two factors:  first, 
there was a lot of "canned" software that was nearly plug&play, covering 
90%+ of all our use cases, and was essentially invisible to us.  And 
second, the "off standard" cases were so rare that the whole company 
only needed one or two people total (yours truly) to handle them when 
they popped up, and were so small-scale and so different from each other 
every time that making any sort of broadly deployable "package" was 
almost pointless.

     What's happening *now,* though, is that we're moving into new 
customer industries, with much different requirements, and *no* 
pre-existing "canned" software.  As a result, we have to *create* all 
that "canned" software from scratch, and no one in management (yet) 
grasps just how many tens of thousands of invisible man-hours were 
supporting all the stuff we used to do for our classical customers.  
Like I said, I came up through the old system, self-taught along the 
way, and while I can *see* the problem and have a vague idea of how it 
needs to be addressed, I might as well be a freshman in his first 
comp-sci course where this effort is concerned -- and I'm the top 
developer in the company!

     As for the deployment rate... well, since this industry application 
is very young, the rate at which we push out changes is actually quite 
rapid, until a particular project is in production -- at that point, 
that project is fairly well "frozen" unless/until a serious bugfix or 
enhancement becomes necessary. But the development pace on our "Core" 
software is still moving quite rapidly -- it hasn't reached the level of 
maturity where changes would be minor and only occasional.  One 
complicating factor here is that I can't build a test system for each 
project -- automation projects are too intertwined with unique hardware 
on a project-by-project basis, and simulation can only take you so far.  
So any "Core" alteration or bugfix has to be hand-tested on actual 
production systems, if it's ever back-deployed.  That, or I have to 
start managing an ever-increasing number of per-project branches.


On 2/21/2017 8:36 PM, Derek DeJonghe wrote:
> Hi there,
>
> Long time listener first time caller, so I'll explain a bit about my
> background and why this topic has caught my attention. I work for a small
> cloud consulting company out of Ann Arbor that specializes in Software as a
> Service (SaaS) enablement and DevOps (as loaded as that term is). My
> portion of the company defines and deploys infrastructure and system
> configuration as code, and helps development teams with Continuous
> Integration and Continuous Delivery (CI/CD). We follow a well defined
> software development lifecycle (SDLC). As a consultant I often find myself
> embedded in organizations that need to adapt their practices and delivery
> to survive. We help enable companies to increase their packaging and
> release rate from quarterly multiple times a day.
>
> In today's industry, I've seen a large shift to Git over all other source
> control. Why is that? As Bob pointed out, it's distributed, it's also
> extremely lightweight, and fast to switch between branches. Git stores
> content as metadata and SVN just stores files, this means that branches in
> SVN are pretty much just folders. Git's contents are cryptographically
> hashed for integrity and can be verified, where network or disk issues can
> impact SVN integrity. Other comparisons such as TFS (only really a
> contender because of windows integration... but TFS 2015 server introduced
> Git so..) have kind of just fell to the wayside.
>
> Either way once you've chosen a source control flavor, you need to
> introduce a branching strategy or your team is likely to fall back into the
> same processes that they already use just with their code backed up and
> automatically merged. I strongly recommend GitFlow to all of my clients.
> GitFlow focuses on feature branches and allows developers to branch off of
> a main line, do work, and merge back in when finished. This helps to limit
> "Dead code" in your project, stuff that's half implemented and not called
> so on and so forth. One would branch off of and merge back into a "main
> line branch" called develop, on the develop branch code would be tested for
> integration in a real environment if passes, it flows up through different
> branches or environments with different tests along the way, if it fails it
> gets kicked back to devs and is not a release candidate. Usually before
> code is merged into the main line it's tested and peer reviewed.
> https://www.atlassian.com/git/tutorials/comparing-workflows#gitflow-workflow
>
> Tagging and versioning, you'll need to tag your repository with version
> numbers. My suggestion is semantic versioning, it's widely used and
> excepted standard. http://semver.org/ Ideally every time code is merged
> into develop your code would be tagged with a version. Once it's in the
> main line it it's either a release candidate or not however it will not
> change as it goes up through the other main line branches and therefore
> only needs to be tagged upon first entry. Once it reaches the master branch
> it's a release, anything on master should be deployable code.
>
> Testing, we use CI servers like Jenkins (my prefered), Bamboo, CircleCI,
> Travis, etc they go on and on, to build and test code. Before a branch can
> be merged in it should pass at least unit testing, then is reviewed by a
> peer. A CI server can drastically reduce the time to market with features
> and bugfixes because all that "It's compiling" time where devs sword fight
> on roller chairs and then run tests on the result is pushed off to a server
> that does this for a living then pushes back a pass or fail result. When
> it's a pass it moves on to be merged, once merged it may go through another
> set of tests and produces a versioned artifact.
>
> The continually produced artifact is the CD (continuous delivery) portion,
> that is continuously delivering artifacts. For this you'll need another
> type of repository. Do not put artifacts in your source control... they're
> not source. You need a blob storage for that. There are servers out there
> that are really nice and feature rich such as artifactory (
> https://www.jfrog.com/artifactory/), however, I do not think they're
> necessary. A simple blob storage will do because your artifact is already
> versioned. A well defined security policy around something like an Amazon
> S3 bucket will do, it has 11 9's of durability so unless you tell it to
> it's not going to lose your artifact.
>
> This is my preferred approach to the source control and artifact management
> side of the Software Development Lifecycle (SDLC).
>
> In Industrial Automation (PLC n such?) I assume you do not have the luxury
> of a high release rate, which means that your software needs to be
> extremely well tested before delivery. Depending on the type of industry if
> you're software isn't extremely well tested it could cause quite a lot of
> damage which means a rigorous process like this could be extremely helpful.
> CI/CD systems and a branching strategy like this can drastically increase
> productivity and time to market. For books look into GitFlow if it sound
> right for you, a large portion of the software industry is using it and
> there are books out there, lots of them small free ebooks. If you read into
> gitflow and find it's too strict or complex there are variants on it that
> are easier to grasp and adhere to.
>
> Derek DeJonghe
>
> On Tue, Feb 21, 2017 at 5:04 PM, gib at juno.com <gib at juno.com> wrote:
>
>> Sounds like a presentation . . .
>>
>> ---------- Original Message ----------
>> From: "Dr. Robert Meier" <list1c30fe42 at bellsouth.net>
>> To: MDLUG's Main discussion list <mdlug at mdlug.org>
>> Subject: Re: [mdlug] Version control: tools and training
>> Date: Tue, 21 Feb 2017 12:13:54 -0500
>>
>> Dave,
>>
>> Version control is a wide field.
>> This is a first summary.
>> Further details  will follow.
>> Your response beforehand will help direct details.
>>
>> There are many proprietary tools available for industrial software
>> management.
>> My experience has been mostly with IBM Rational (in the 90s
>> called Smart) and I can recommend it for its ease-of-configuration,
>> line-level resolution, support of plug-ins/custom scripts.
>>
>> There are many open source tools available for industrial software
>> management.
>> My experience has been mostly with svn, git, and cvs.
>> I can recommend all of the above especially:
>>          svn - centrally mastered with support for field-editing
>>          git - distributed master with support (via svn bridge)
>>                  for central control
>>          cvs - centrally mastered, scriptable, with svn interface
>>
>> There are two basic divisions for version control and nearly all
>> tools support both, but usually are heavily biased toward one side.
>> The better tools (including those named above) will interface
>> (e.g. via bridge, virtual agent, ...) seamlessly with their
>> opposite number.
>>
>> 1. (svn, rational)
>>          central master - One host has the master copy of all files.
>>          editing - files are "checked out" for editing
>>                  usu. each file can be simultaneously checked out by
>>                          only one user
>>                  better tools (rational, svn) support "merging"
>>                          when two editors conflict
>>          branches - identify SEQUENCES of fileset changes
>>          labels - identify STATES of fileset changes
>>          security - central reading and writing restrictions
>> 2. (git)
>>          distributed master - Each file master may be on different host.
>>          editing - files are copied (individually or en-masse) for edit
>>          security - separate from revision (since file master is mutable)
>>                  usu. rules applied to pulls toward "master" host
>>          temporary repo - contain SEQUENCES of fileset changes
>>          archive repo - contain STATES of fileset changes
>>
>> Training is available online for both approaches.
>>
>> Hopefully helpful,
>> --
>> Bob
>>
>>
>>
>> On 02/21/2017 08:34 AM, Dave McMillan wrote:
>>>       This... is going to be rather off-topic, but this is the most
>>> likely group I know of to ask about this.
>>>
>>>       I work for a Detroit-area industrial-automation company that's
>>> never really been a software development house, but now finds itself
>>> becoming one unexpectedly.  So I've been tasked with finding a way to
>>> bring us into the 21st century. :-\
>>>
>>>       So, I have a small group of "developers" that are essentially
>>> self-taught, and have spent their careers doing mostly one-off software
>>> projects and managing their own versions and backups independently (if
>>> at all), often mostly inside their heads.  Plus a larger number of
>>> "debuggers" that have less programming skill, but get much more time on
>>> the active machines, who will be the people who do most of the
>>> bug-finding and reporting, and installing/testing updates and bugfixes
>>> pushed out by the developers.  And customers that want levels of
>>> traceability and documentation that, frankly, we've *never* done.
>>>
>>>       Just to ice the cake, this being industrial automation means that a
>>> lot of our source code is tied up in proprietary languages that are
>>> recorded in proprietary binary formats and not generally accessible by
>>> 3rd-party tools like (for example) git.  So I'm in the position of
>>> possibly being forced to maintain an entire *ecosystem* of different
>>> version-control systems with brand-specific branches for different
>>> pieces of equipment.
>>>
>>>       And did I mention that while *some* software has to be
>>> centrally-controlled, other parts of the *same* software have to be
>>> field-editable on a machine-by-machine basis?  And I have to track *all*
>>> of that.
>>>
>>>        So, right now, my biggest concern is less the *tools,* than the
>>> *training.*  Whatever tool (or tool set) we end up choosing (and there
>>> I'm at the mercy of the beancounters), I think my *bigger* problem is
>>> the total lack of any in-house *culture* for version control,
>>> multi-person developer/debug teams, and obsessive
>>> tracking/documentation/etc.  And I, frankly, am just as clueless. So,
>>> I'll put it to people who (hopefully) know more about this kind of thing
>>> than I do:  where do I *start*?  Are there training programs, or
>>> certification courses, or even just a *really good* O'Reilly book that I
>>> can use to get myself off square one?  And does anyone know of any sort
>>> of training curriculum for bringing this sort of cultural shift into an
>>> existing "herd of cats" development team?
>>>
>>> _______________________________________________
>>> mdlug mailing list
>>> mdlug at mdlug.org
>>> http://mdlug.org/mailman/listinfo/mdlug
>>>
>> _______________________________________________
>> mdlug mailing list
>> mdlug at mdlug.org
>> http://mdlug.org/mailman/listinfo/mdlug
>>
>> ____________________________________________________________
>> How To Remove Eye Bags & Lip Lines Fast (Watch)
>> Womans Weekly
>> http://thirdpartyoffers.juno.com/TGL3131/58acb9bdcf41f39bd01dast02vuc
>> _______________________________________________
>> mdlug mailing list
>> mdlug at mdlug.org
>> http://mdlug.org/mailman/listinfo/mdlug
>>
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug