top 50 comments

sorted by: hot top controversial new old
[–] 181 points 2 years ago (3 children)

If capitalism insists on those higher up getting exorbitantly more money than those doing the work, then we have to hold them to the other thing they claim they believe in: that those higher up also deserve all the blame.

It's a novel concept, I know. Leave the Nobels by the doormat, please.

  • source
  • hideshow 6 child comments
  • [–] 14 points 2 years ago (2 children)

    Was there a process in place to prevent the deployment that caused this?

    No: blame the higher up

    Yes: blame the dev that didn’t follow process

    Of course there are other intricacies, like if they did follow a process and perform testing, and this still occurred, but in general…

  • source
  • parent
  • hideshow 4 child comments
  • [–] 128 points 2 years ago (2 children)

    Git Blame exists for a reason, and that's to find the engineer who pushed the bad commit so everyone can work together to fix it.

    Blame the Project manager/Middle manager/C-Level exec/Unaware CEO/Greedy Shareholders who allowed for a CI/CD process that doesn't allow ample time to test and validate changes.

    Software needs a union. This shit is getting out of control.

  • source
  • hideshow 4 child comments
  • [–] 14 points 2 years ago (3 children)

    Or it needs to be a profession.

    Licensed professional engineers are expected to push back on requests that endanger the public and face legal liability if they don't. Software has hit the point where failure is causing the economic damage of a bridge collapsing.

  • source
  • parent
  • hideshow 5 child comments
  • load more comments (1 reply)
  • [–] 65 points 2 years ago (1 child)

    "George Kurtz, the CEO of CrowdStrike, used to be a CTO at McAfee, back in 2010 when McAfee had a similar global outage. "

  • source
  • hideshow 2 child comments
  • [–] 37 points 2 years ago (3 children)

    I do wonder how frequent it is that an individual developer will raise an important issue and be told by management it's not an issue.

    I know of at least one time when that's happened to me. And other times where it's just common knowledge that the central bureaucracy is so viscous that there's no chance of getting such-and-such important thing addressed within the next 15 years is unlikely. And so no one even bothers to raise the issue.

  • source
  • hideshow 5 child comments
  • [–] 19 points 2 years ago (1 child)

    Reminds me of Microsoft's response when one of their employees kept trying to get them to fix the vulnerability that ultimately led to the Solar Winds hack.

    https://www.propublica.org/article/microsoft-solarwinds-golden-saml-data-breach-russian-hackers

  • source
  • parent
  • hideshow 2 child comments
  • load more comments (1 reply)
    [–] 29 points 2 years ago (3 children)

    If you don't test an update before you push it out, you fucked up. Simple as that. The person or persons who decided to send that update out untested, absolutely fucked up. They not only pushed it out untested, they didn't even roll it out in offset times from one region to the next or anything. They just went full ham. Absolutely an idiot move.

  • source
  • hideshow 6 child comments
  • [–] 21 points 2 years ago* (1 child)

    The bigger issue is the utterly deranged way in which they push definitions out. They've figured out a way to change kernel drivers without actually putting it through any kind of Microsoft testing process. Utterly absurd way of doing it. I understand why they're doing it that way but the better solution would have been to come up with an actual proper solution with Microsoft, rather than this work around that seems rather like a hack.

  • source
  • parent
  • hideshow 2 child comments
  • [–] [S] 14 points 2 years ago (1 child)

    We still don’t know exactly what happened, but we do know that some part of their process failed catastrophically and their customers should all be ready to dump them.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 5 points 2 years ago

    I'm quite happy to dump them right now. I still don't really understand why we need their product there are other solutions that seem to work better and don't kill the entire OS if they have a problem.

  • source
  • parent
  • [–] 23 points 2 years ago (2 children)

    Wild theory: could it have been malicious compliance? Maybe the dev got a written notice to do it that way from some incompetent manager.

  • source
  • hideshow 3 child comments
  • [–] 3 points 2 years ago

    While that’s always possible, it’s much more likely that pressures to release quickly and cheaply made someone take a shortcut. It likely happens all the time with no consequences so is “expected” in the name of efficiency, but this time the truck ran over grandma.

  • source
  • parent
  • load more comments (1 reply)
    [–] 23 points 2 years ago

    If I'm responsible for the outcome of the business, I want a fair share of the profits of the business.

  • source
  • [–] 11 points 2 years ago* (last edited 2 years ago)

    I get that it's not the point of the article or really an argument being made but this annoys me:

    We could blame United or Delta that decided to run EDR software on a machine that was supposed to display flight details at a check-in counter. Sure, it makes sense to run EDR on a mission-critical machine, but on a dumb display of information?

    I mean yea that's like running EDR on your HVAC controllers. Oh no, what's a hacker going to do, turn off the AC? Try asking Target about that one.

    You've got displays showing live data and I haven't seen an army of staff running USB drives to every TV when a flight gets delayed. Those displays have at least some connection into your network, and an unlocked door doesn't care who it lets in. Sure you can firewall off those machines to only what they need, unless your firewall has a 0-day that lets them bypass it, or the system they pull data from does. Or maybe they just hijack all the displays to show porn for a laugh, or falsified gate and time info to cause chaos for the staff.

    Security works in layers because, as clearly shown in this incident, individual systems and people are fallible. "It's not like I need to secure this" is the attitude that leads to things like our joke of an IoT ecosystem. And to why things like CrowdStrike are even made in the first place.

  • source
  • [–] 8 points 2 years ago (1 child)

    As a counterpoint to this articles counterpoint, yes, engineers should still be held responsible, as well as management and the systems that support negligent engineering decisions.

    When they bring up structural engineers and anesthesiologists getting "blame" for a failure, when catastrophic failures occur, it's never blaming a single person but investigating the root cause of failures. Software engineers should be held to standards and the managers above them pressuring unsafe and rapid changes should also be held responsible.

    Education for engineers include classes like ethics and at least at my school, graduating engineers take oaths to uphold integrity, standards, and obligations to humanity. For a long time, software engineering has been used for integral human and societal tools and systems, if a fuck up costs human lives, then the entire field needs to be reevaluated and held to that standard and responsibility.

  • source
  • hideshow 1 child comment
  • load more comments (1 reply)
    [–] 7 points 2 years ago (1 child)

    CTOs that outsourced to a software they couldn't and didn't auidit are to blame first. Not having a testing pipeline for updates is to blame. Windows having a verification system loophole is too blame. Crowd strike not testing this patch are too blame. Them building a system to circumvent inspect by MS is their fault.

    Now with each org there is probably some distribution of blame too, but the execs in charge are first and for most in charge...

    Honestly this is probably enough serious damages in some cases that I suspect ever org to have pay some liability for the harms their negligence caused. If our system is just that is, and if it is not than we have a duty to correct that as well

  • source
  • hideshow 2 child comments
  • [–] 7 points 2 years ago

    I've said it before and I'll say it again.

    Corporate culture is a malicious bad actor.

    Corporate culture, from management books to magazine ads to magic quadrants is all about profits over people, short term over stability, and massaging statistics over building a trustworthy reputation.

    All of it is fully orchestrated from the top down to make the richest folks richer right now at the expense of everything else. All of it. From open floor plans to unlimited PTO to perverting every decent plan whether it be agile or ITIL or whatever, every idea it lays its hands on turns into a shell of itself with only one goal.

    Until we fix that problem, the enshittification, the golden parachutes, and the passing around of horrible execs who prove time and time again they should not be in charge of anything will continue as part of the game where we sacrifice human beings on the Altar of Record Quarterly Profits.

  • source
  • parent
  • [+] 5 points 2 years ago
    [removed by mod]
  • source
  • load more comments
    view more: next ›