Continuous delivery and legal aspect

[u/lppedd]

Hey folks, not sure if this is the best sub for the question, but I'll try anyway. I'm asking here to gather some feedback before asking internally.

I've been developing software for a decade by now, but admittedly I've never really investigated the legal aspect that kicks in every time a piece of software goes out to the public. I mean, I'm aware of a subset of steps (certificate of originality, export control, OSS, etc.) because I'm part of that process, but I've never gone beyond that because of time and responsibilities.

We've recently switched from a 4-months long release cycle for product A, to a more agile alternative for product B, where we (devs) planned to deliver multiple updates per month. Not really continuous delivery, but you get it. This was discussed for a long time, we came up with the proper versioning scheme, and all parties seemed to agree.

However, now we are blocked by processes again: we must let the PO know about an update at least a couple weeks in advance, so that all involved people (no idea who they are apart from a couple of them) begin working on their tasks. I see a lot of stuff going on in Jira (I don't use it, I track on GitHub), like dozens of work items, epics and whatnot. We then get an estimated code freeze and text exit dates.

This obviously screws up the "continuous" aspect, as realistically we are able to ship once a month on the good months. So I'm wondering, how do companies manage to release multiple updates per month? Or even multiple updates per day? Do they have a legal department always running at 100% to keep up with development?

Or is the process I'm witnessing something entirely arbitrary that they simply don't want to change? Should I even attempt to get this corrected?

[u/flavius-as]

It's easy.

As a developer you stop looking at the real users as "the delivery".

Instead, you designate environments to deliver to. These are environments where then the PO and legal take over.

Code freeze? That's for those environments. It's not for development. You freeze adding new features to about-to-be world-public environments, and instead spin up a new production-like environment.

Capability needed: CICD maturity.

So: you deliver continuously. And strategically change the place to which you deliver continuously. PO owns the trigger for doing that.

Developers? Never blocked. Continuously delivering to a place.

[u/hippydipster → u/flavius-as]

If it takes them 2 weeks to review while developers move on to delivering other stuff, then A) the feedback from the review can end up in multiple tracks of development as devs have to fix those issues and deliver them to the right environment without including everything else they've added in the meantime, and B) they might create delivered environments faster than the legal review process can keep up with.

[u/flavius-as → u/hippydipster]

True.

This only creates pressure on the PO and to adapt and shift left.

[u/RustOnTheEdge → u/flavius-as]

Yes easy. Also freaking pointless as you will continue to build without any real validation. The shortened feedback loop is non existent.

[u/lppedd → u/RustOnTheEdge]

Yeah I have to agree here. The idea is good but it all falls apart the moment they don't adapt to it, which is pretty much a sure thing. I think the change has to start one level above the PO.

[u/flavius-as → u/lppedd]

It's not pointless because it shifts responsibility where it belongs.

But yes, you should have real customers (opt in for testing your pre releases) access those fronzen environments.

But that's not your realm any more. It's the realm of executives and business developers.

[u/kbn_ → u/flavius-as]

This is the way. And in the limit, it is going to mean his legal department will be running 100% on this one question, because development velocity will be much much higher. That in turn will create other process pressures to either allow hotfixes and minor things to bypass the process, or to integrate PO further upstream allowing them to work more incrementally as features and fixes materialize.

All good things obviously, and can only happen once engineering is no longer just eating the bottleneck.

[u/flavius-as → u/kbn_]

That PO should be the one defining the acceptance criteria and then accepting each and every ticket delivered.

But yeah.

As a technical person, we deliver the technical tools which then create the pressure for people to change.

[u/kbn_ → u/flavius-as]

Agreed but depending on packaging and deployment, it might be difficult today for PO to do that as the code lands. That should absolutely be the North Star though! The way to get there incrementally is to do exactly what you said: make the delivery to PO continuous even while delivery to the customer is periodic and waterfall. It moves the obvious bottleneck to PO, and they’ll be the ones clamoring for a more integrated and incremental validation process.

TLDR: fully aligned

[u/lppedd → u/flavius-as]

That would mean creating a way for the PO (or someone else) to push out what's in the proposed staging environment. We currently don't have that.

But to be entirely clear, we are talking about publishing to a marketplace. That's something I could automate in 30 minutes, if I didn't have to go through all the stuff I described.

My point is I can automate whatever I want, but if the other involved parties don't have any automation, it simply doesn't work.

[u/flavius-as → u/lppedd]

I don't understand the problem.

You can make for yourself all technical tools.

You can provide to others all technical tools.

You can, for instance, grant permission to the PO to click on the green button in Jenkins. Whenever they want. Not your problem. You just provide the URL or whatever information necessary for them access that frozen environment and deploy it to the market place. Not your problem.

[u/lppedd → u/flavius-as]

That's a good point. I don't want to commit to implement such a process without them telling me "we're ok using it" tho. I already got burned by similar things in the past.

[u/flavius-as → u/lppedd]

If they're scared of too many buttons, it's easy to whip up an internal site with a single bit green button which flips a bit.

But these are details.

For success, your task is to recognize their fears and concerns and provide them with the tools necessary to overcome their own concerns.

[u/ZukowskiHardware]

Continuous delivery to me is shipping multiple times a day if you want.  Your system just sounds like complicated waterfall. 

[u/CardiologistStock685 → u/ZukowskiHardware]

it sounds like the story of when Skype team started to adapt with agile process.

[u/failsafe-author]

I’ve never worked on a product that required the legal department to be included on every release, and I’ve worked in some pretty heavily regulated fields (banking and real money gaming, for example).

Legal does need to be involved led on the product side, and there are considerations, but if product tells us to build something, we trust they’ve done through the legal steps necessary.

[u/diablo1128 → u/[deleted]]

I worked on medical devices for 15 years, think dialysis machines, and legal was always there when we wanted to release to the field. They had to coordinate with the FDA to get releases approved before we could push them out in to the field.

Legal didn't care about internal releases, but anything going to the field had legal involved.

[u/Flashy-Whereas-3234]

From a purely software standpoint, I would try and detach the "deployment" aspect from the "release" aspect, whatever that release process may be.

The web example is feature flags; we develop new features behind flags (this can include invisible improvements too, we've upgraded frameworks this way) and we merge incremental changes back to master asap. We deploy daily (or faster), but nobody sees these changes because they're behind a flag.

When development is complete we hand the flag over to Product, and when Product have done all their comms and marketing, they flip it live. Some features sit in production for weeks waiting for marketing because they love to space out content and have something to talk about, but none of that is a development problem - we're done and out of the way.

This would be more difficult if your product is a binary executable for customers to run locally; can you still ship it and just have flags lurking in the code? Can you use compiler flags to have code omitted until it gets the rubber stamp? Or can you have two types of releases; code updates and flag updates, so making something "live" isn't actually shipping code it's just flipping a magical bool.

We love flags because it means all the code has been out there for ages, just nobody is using it. The trunk is the trunk, we keep developing even if a feature never sees the light of day due to politics (has happened!)

[u/damianh → u/Flashy-Whereas-3234]

Totally this. What's happening in Jira is a load of product / sales / support / customer motion stuff. Decoupling deployments from releases (or "launches") via feature flagging is a core CI/CD maturity practice, along with trunk based dev, branch by abstraction etc. Requires a load of discipline though.

[u/mastermindchilly → u/Flashy-Whereas-3234]

I like the approach of thinking of a release as a set of feature flag values. It gives you the capability of testing your code’s resiliency against past, present, and future release configurations.

[u/MendaciousFerret → u/Flashy-Whereas-3234]

This, just FF it, deploy and let Product release (which they probably will never do because it's not perfect...)

[u/waywardworker]

When I worked on export controled stuff the first release was run through legal, we had a dedicated export control team. This was involved, lots of details and fun.

Every subsequent release did broadly the same thing so sat in the same export category (they are fairly wide), we didn't even notify them.

I suggest having a chat to all these different teams and figuring out what they are up to and need. Most of this work also doesn't need the actual product, if you need a sign off before release they should be able to do it while you are still developing the feature.

[u/lppedd → u/waywardworker]

Thank you! That makes sense and it's why I'm so perplexed by all this complexity and fixed deadlines. As I replied to another commenter, I feel like this goes under the "we don't want to change" category.

[u/Hot-Profession4091]

What’s actually going on here is one part of the organization wants to ship faster and another part of the organization isn’t on board yet.

[u/Jmc_da_boss]

This is a thing in many places, the key point to me in how you deal with this is to frame the question as capabilities. "Can I deploy to prod in 10 minutes at a technical level?"

The fact the business may or may not be stuck in the 1800s is not relevant to the capabilities question.

Once you have the capabilities in place, THEN you start making noise about the slowness of other processes outside of your control.

[u/lppedd → u/Jmc_da_boss]

Yes we can. Our pipeline can do that, we just need the ok, which obviously is missing.

I feel like there are way too many people involved in the process, not sure if you get what I mean. Too many people that need to be synced.

[u/ColdPorridge → u/lppedd]

The question is have them is what decisions do they need to make/why are they involved? Is it because they’re unsure the release meets quality standards? If so, there may be automation you can provide to help them make that decision, with the goal of eventually having as many of them agree to fully automated sign offs as possible.

[u/serial_crusher]

This is going to heavily depend on what kind of product you’re shipping, and how much regulation exists.

But I’d be willing to bet that a lot of the delays you’re seeing are internal bureaucracy and inefficient processes more so than actual legal requirements. You should start working with stakeholders to identify where the bottlenecks are and address them.

[u/New_Firefighter1683]

I don't understand.

Shouldn't there be a team doing this?

All the things you mentioned that you're a part of are part of CI. Sounds like someone already built those into the pipeline.

must let the PO know about an update at least a couple weeks in advance, so that all involved people (no idea who they are apart from a couple of them) begin working on their tasks

Shouldn't the PO be dictating this instead of the other way around? "This is ready for development with these requirements", not "we developed all these requirements, did we just make shit up?"

I mean.. you CAN still roadmap things out a couple weeks, no?

[u/Ibuprofen-Headgear]

I guess it depends on your industry / product to some degree. Most of the stuff I’ve worked on hasn’t had to “go through legal”. I’m working on something now that does require a specific compliance, which adds a step or 2 pre-release, but nothing crazy (I worked on a similar project in the past, but again, that hardly added any time). In your case, I suspect a lot of it is in various people either not trusting certain other people or processes and/or not wanting to give up their perceived level of control.

[u/lppedd → u/Ibuprofen-Headgear]

Yes, I believe it's the control aspect. The process has not changed from what we had with product A. The Jira workflows are the same, and I suspect it's the key point here: they don't want to have to come up with another workflow that would differ from what they're used to.

I may have to bring this up with my skip manager in my geography.

[u/Antoak]

The idea is that as updates become frequent, they become smaller, which means increased simplicity for both you and legal.

Honestly most changes are boilerplate, you probably just need a spreadsheet of what kinds of changes require a higher level of scrutiny.

[u/lokaaarrr]

Is this a cloud or on-premise product? Ive done lots of fedramp+ (and other regulatory schemes) cloud service stuff with no legal.

If it’s not cloud, then real continuous is not a great plan. You really need a foolproof fast rollback ability.

[u/lppedd → u/lokaaarrr]

It's products delivered to marketplaces. Think about the VS Code marketplace. The feedback cycle is as short as it can get for an on premise product. I'd like to publish as soon as I get a fix or a feature in the trunk. We basically do this already internally, for ourselves, but the blocker is the paperwork, if we can call it that way.

[u/lokaaarrr → u/lppedd]

That's always going to be harder, just a lot more that can go wrong.

Do you have a "beta channel" that people can opt into? That might be a starting point. You could get legal to have some addendum they need to agree to in order to get access to it.

[u/lppedd → u/lokaaarrr]

Yes, the marketplace offers a pre-release channel. We originally planned to use it, but then hit this problem and were left with only a release a month, in the stable channel, just because there is no point in doing pre-releases.

[u/cez801]

Assuming you are not in very specific industries such as avionics / healthcare devices - then the chances are that the process from your product manager and / or legal team is probably not rooted in reality. What I mean is that people will often say ‘oh but the regulations say xyz’ and that creates the process . I know this because I worked in healthcare software doing patient record systems and later FDA approved devices for home use. And in working with the FDA - for example - agile is 100% allowed.. but our legal team kept saying ‘we can’t do that because FDA’ The issue was that there were steps we needed to do, but not in exactly the way that our legal team thought was required. We got to the point of doing daily updates - although we did not push them to end users every day ( mainly because our users would get tired of the changes ).

The approach we took was:

• daily releases.
• the daily release is shippable.
• we don’t patch backwards. Only forewords.
• then the customer facing team and legal team would just decide what day to do a release. - and they could release that days versions

So, how do other companies do it? Depends on industry and style ( on prem is different from SaaS, which is difference from licenses cloud ).

But in general, for engineering, work out how to get to the point of a daily release. Don’t branch to patch. If someone wants to release it - they take that version.

[u/Orca-]

End to end Continuous delivery doesn’t make sense when you’re shipping hardware that can’t be updated in the field, software that needs extensive verification cycles, etc.

It works great for the web where hypothetically each refresh of the page could be a new application, but it doesn’t work everywhere as you see here.

When you get two chances a year to do the full fat extensive testing and validation needed to release high availability software, things move at a different cadence.

The only way to change it is to reduce the acceptance criteria, and sometimes that is not acceptable.

Push your CD to the part you control, but be prepared to branch appropriately. If a critical bug needs to be fixed, legal/testing/etc. will want it based off the build you gave them, not what you’ve been working on for the last month.

[u/QuantityInfinite8820]

Legal has nothing to do with it unless it’s a regulated registry like banking