Build Artifacts with Andrew

Tag: devops

  • Can AI Generate Functional Terraform?

    Nope.

    The end. Thanks for reading my post. Reading time: 1 minute.

    I posted the other day about this topic, and I am intrigued by the possibilities. I’m certainly interested in the ways that you can use it for infrastructure, and the article in that post offers a somewhat-different use case for AI-generated Terraform: cloud migrations and multi-cloud solutions. But I’d by lying if I said I wasn’t very skeptical of the code that it writes.

    With all that on my mind, I appreciate the analysis in this article: “Can AI Generate Functional Terraform?” by Rak Siva.

    I’d add mainly that GenAI is currently about as useful as a very junior developer, and that’s probably because they’re both doing the same thing: Google the results and copy-paste without really understanding.

    Then again, if you’ll indulge a quickly-emerged cliché: none of us saw any of this coming just five years ago.

  • AWS Launches Trust Center

    Compliance just got a tiny bit easier in AWS-land. AWS announced that they’re launching their new AWS Trust Center, an all-in-one hub for AWS’s security-related documentation.

    I certainly haven’t read through the whole site, but just eyeballing what they’ve got:

    • Security
    • Compliance
    • Data protection and privacy
    • Operational visibility
    • Report an incident
    • Agreement and terms

    I doubt they’ve even released any new documentation, but it’s a nice step forward to put all this stuff in one place.

  • DevOps 101: Continuous Improvement and Learning

    Courtesy: Wikimedia Commons

    No development team is perfect. Even the highest-performing teams can improve on what they’re doing, or else they probably need tweaks from time to time as knowledge, attitudes, situations, and responsibilities evolve.

    Continuous improvement is a fundamental principle of DevOps, promoting a culture of ongoing learning, feedback, and enhancement. The ideal of continuous improvement involves iteratively identifying areas for improvement, implementing changes, and measuring the impact to drive further refinements.

    Look, I get it: this seems like super-basic stuff. And it is! These things feel obvious, they’re pretty easy to get right, and you can easily forget about Continuous Improvement when a team is functioning well. But I’ve seen more than one team fail to implement a simple end-of-sprint retrospective, and the results are not good.

    Read more!

  • Easier Cloud-to-Cloud Migrations?

    Cloud with a lock. Courtesy of Wikimedia Commons.

    An Empty (Theoretical) Promise

    It’s long been a promise of Infrastructure as Code tools like Terraform that you could theoretically create platform-independent IaC and deploy freely into any cloud environment. I doubt anyone ever really meant that literally, but the reality is that your cloud infrastructure is inevitably going to be tied quite closely to your provider. If you’re using an aws_vpc resource, it’s pretty unlikely that you could easily turn that into its equivalent in another provider.

    And yet, several of the organizations I’ve worked with have been reluctant to tie themselves closely with one cloud provider or another. The business reality is that the vendor lock-in is a huge source of anxiety: if AWS suddenly and drastically raised their prices, or if they for some reason became unavailable, lots and lots of businesses would be in a big pickle!

    The amount of work required to manually transfer an existing system from one provider to another would be nearly as much as creating the system in the first place.

    GenAI as the Solution?

    I ran across this article about StackGen’s Cloud Migration product. The article isn’t long, so go read it.

    Instead of requiring DevOps teams to map out resources manually, the system uses read-only API access to scan existing cloud environments. It automatically identifies resources, maps dependencies, and – perhaps most importantly – maintains security policies during the transition.

    StackGen isn’t new to using generative AI for infrastructure problems, but they have an interesting approach here:

    1. Use read-only APIs to identify resources, including those not already in IaC.
    2. Use generative AI to map those resources, including security policies, compliance policies, and resource dependencies.
    3. Convert those mapped resources into deployment-ready IaC for the destination environment.

    Using a process like this to migrate from provider to provider is interesting, but the one use case that really gets me thinking is the ability to deploy into a multi-cloud environment.

    I’ll be keeping my eyes on this one.

  • Cyberattack brings down Newspaper Publisher

    Lee Enterprises, one of the largest publishers of newspapers in the United States, has had outages caused by a cyberattack. There have been no details on the nature of the attack, but the St. Louis Post-Dispatch has been affected.

    Read more on Tech Crunch.

  • DevOps 101: Cross-Functional Teams

    Crayons (courtesy of Wikimedia Commons)
    Courtesy Wikimedia Commons

    Cross-functional teams play a vital role in a DevOps culture as they bring together individuals with diverse skills and expertise from different areas of software development and operations.

    By embracing cross-functional teams, organizations can foster collaboration, improve communication, streamline processes, and create an environment conducive to innovation and continuous improvement. In a DevOps culture, where speed, agility, and quality are paramount, cross-functional teams play a crucial role in breaking down barriers, improving collaboration, and delivering high-value software efficiently.

    Read more!

  • Checksums are SHAping up to be complicated!

    I have plenty more of my beginner DevOps materials (see DevOps 101: Hello There!), but I also want to post about problems I’ve run into. So, this is something I’ve been mulling over a bit lately.

    The Issue

    My team is coordinating with another team on passing files from one AWS S3 bucket to another, and we need to verify that the file at the destination is the same as the file at the source.

    Normally, you would probably just rely on S3; it’s reliable, and it does its own checksum validation on its copy operations. However, the work is part of a permanent archive, and verifying file integrity is the core-est of core requirements. Not only do we need to verify the integrity every time we move the file, but we also need to occasionally spot-check our archives while they’re at rest.

    Our customer has traditionally used one of the SHA algorithms for file validation. That’s not a problem per se, but calculating a SHA on a very large file (100+ GB is not that unusual) is slow and expensive! We’d rather avoid it as much as possible.

    Potential Solution: S3’s “Checksum”

    One solution would be if AWS would handle calculating the checksum for us as part of its own file integrity checking. I think it might fit our requirements if we could get an AWS-calculated checksum of the full object that we could then do our spot-checking on later.

    As it turns out, AWS automatically provides this as a sort of by-product of the S3’s copy object feature. When it calculates the checksum that it uses for a copy, it stores that information and makes it available for its own later use and for the user.

    However, AWS doesn’t offer what they call full-object checksums if you’re using SHA. They only offer composite checksums. The distinction, as the documentation puts it, is:

    Full object checksums: A full object checksum is calculated based on all of the content of a multipart upload, covering all data from the first byte of the first part to the last byte of the last part.

    Composite checksums: A composite checksum is calculated based on the individual checksums of each part in a multipart upload. Instead of computing a checksum based on all of the data content, this approach aggregates the part-level checksums (from the first part to the last) to produce a single, combined checksum for the complete object.

    The main problem, as you may have noticed, is that if you’re using multi-part uploads, then the composite checksum of the entire object is going to depend on your chunks being exactly the same size every time a file is moved. When you’re moving between different services, that can be super brittle: a change in one system’s chunk size would affect the ultimate checksum in completely unpredictable ways.

    Full-Object Checksums and Linearization

    The reason why you can’t do a full-object checksum using SHA is because SHA is basically a huge polynomial equation; change one bit in the original, and the hash, by design, is changed completely. SHA can’t be linearized, meaning you can’t calculate different parts independently and then re-combine them.

    This brings us to Cyclic Redundancy Check (CRC) algorithms. These are also error-detection algorithms, but they’re calculated more or less like the remainder of a giant division problem. And, importantly, if you take the remainders from n division problems, add them together, take the remainder again, you get the remainder of the sum. So, in a super-simple example, if you want the remainder from 150 % 4, you could do it this way:

    • 150 % 4 = ??
    • 100 % 4 = 0 and 50 % 4 = 2
    • (0 + 2) % 4 = 2, therefore 150 % 4 = 2

    It’s a roundabout way of calculating such a small modulus, but if you have a thousand file chunks, then you can find the CRC in essentially the same way.

    So, that’s why AWS only offers full-object checksums for CRC algorithms: they don’t want to have to compute the whole SHA any more than we do.

    What does that mean for us?

    I obviously think using CRCs and full-object checksums to replace our manual calculations would save a lot of compute (and therefore both time and money).

    It’s still an open question whether or not switching to CRCs will satisfy our requirements. There also might be weird legacy issues that could crop up after relying on one specific algorithm for years.

    Let me know if anyone has thoughts on this issue or, especially, if I’ve gotten things wrong.

  • DevOps 101: Keeping it Agile and Lean

    DevOps didn’t come from nowhere, and it doesn’t operate in a vacuum. Any time you’re trying to apply DevOps principles, there are already going to be existing ways of doing things either in the workplace or, at the very least, in the minds of the team as they come together. You might build a team of thirty professionals, and they would have thirty different opinions on the “right” way to do things.

    Relatedly, it’s hard to define DevOps, because it’s one of those terms that’s used in slightly different ways by different people. There’s a fair amount of debate about what is and isn’t DevOps.

    There’s a whole discussion about whether or not it should be “DevOps” or “DevSecOps” and whether or not the two terms refer to the same thing. In a future post, I’ll explain why I prefer the term DevOps, even though I think everyone would agree that the “Sec” (Security) portion of the culture is extremely important.

    Read more!

  • DevOps 101: Culture and Mindset

    Old-style rotary phone. Courtesy of Wikimedia Commons.
    Courtesy Wikimedia Commons

    I’m sure everyone wants to start by diving into pipelines or traces or something. I certainly wanted that as a learner: get me straight to the technical stuff! But DevOps is more than just a set of tools or technologies; it’s a mindset, a way of thinking, a commitment to bridging the gap between development and operations (and security — more on that later) to enable faster and more reliable software delivery.DevOps culture encourages:

    • Breaking down silos and fostering a sense of shared ownership and accountability.
    • Open communication, trust, and mutual respect among team members.
    • Blameless culture, where failures are seen as learning opportunities rather than occasions for blame.

    Read more!