CrowdStrike Falcon shutting down systems last Frday

[TC33]

Someone on the news said every affected system will need to be physically rebooted, which apparently will take years. Is that true?

It may take years for that 1 system that no one really knew about to be rebooted

Pretty sure any large or even small scale company people rely on will all be fixed by Monday/Tuesday. Most already are.
It was a big outage and big impact, but the remediation was pretty straight forward.

Any company still in trouble by tomorrow needs to seriously look at their Disaster Recovery processes.

 

[TC33]

Crowdstrike software bug is a pure example of not doing proper testing of software before a release. No bugs should ever be introduced into the market due to a lack of testing. Yes, software companies get what they deserve when they offshore software development for cheap labor in foreign countries. It is not the best and brightest that drives these profit-seeking stock option bonus CEOs; it is cheap and easy labor.

Unfortunately its not always up to the engineer and tester whether code is properly tested and released.
Sometimes it just needs to be released and it needs releasing tomorrow........

There is probably a bunch of poor Developer/Testers out there now losing sleep over the thought of their future. Just because they were put under ridiculous amounts of pressure to deliver.

 

[jaylach]

@Colin_T : Actually Microsoft offers a tool to 'hide' updates that are not wanted. Also, granted that it is third party, but WU10man allows for pretty extensive update control in Windows 10 and 11. I have it on both and updates don't install until I say they should which gives me time to 'hide' any update I don't want. Wu10man is no longer being developed but still available.

Wu10man:

GitHub - WereDev/Wu10Man: Enable/Disable Windows 10 Automatic Updates

Enable/Disable Windows 10 Automatic Updates. Contribute to WereDev/Wu10Man development by creating an account on GitHub.

github.com

MS tool to hide updates:
http://download.microsoft.com/download/F/2/2/F22D5FDB-59CD-4275-8C95-1BE17BF70B21/wushowhide.diagcab
This is not a link to a web page but a direct link to the download.

 

[Fishfunn]

Unfortunately its not always up to the engineer and tester whether code is properly tested and released.
Sometimes it just needs to be released and it needs releasing tomorrow........

There is probably a bunch of poor Developer/Testers out there now losing sleep over the thought of their future. Just because they were put under ridiculous amounts of pressure to deliver.

I always user-test my software modifications, no matter what the schedule is. I always made time if the schedule was too tight. I worked many extra hours at home if I needed to. That is the main reason I welcome being retired.

 

[raylove]

It's strange how this issue was somewhat selective. I work in a chain where each branch runs identical Windows 10 computers (we were all upgraded 3yrs ago), yet about half got the blue screen and half suffered no issues. They are all left in standby overnight.
Even more strange is that most branches have a main computer and a second slave computer that operates through the main one i.e. if the main goes down the secondary one won't work, but the main one can operate alone. On Friday our main unit worked fine but the secondary unit had the blue screen. I don't understand how one was affected but not the other when they are intrinsically linked.

 

[GaryE]

On a side note, thank goodness whatever fool named that company (Crowdstrike Falcon???) doesn't make up common names for tetras. I'm off to feed my shoalwalloper peregrines...

 

[Colin_T]

@Colin_T : Actually Microsoft offers a tool to 'hide' updates that are not wanted. Also, granted that it is third party, but WU10man allows for pretty extensive update control in Windows 10 and 11. I have it on both and updates don't install until I say they should which gives me time to 'hide' any update I don't want. Wu10man is no longer being developed but still available.

When I turn off auto updates, it turns itself back on straight after or shortly after. And it always turns itself on if I leave the pc idle for a couple of minutes. It does my head in and the next pc is gonna be something that doesn't use windows. However, I don't like apple either so I think I'm stuffed.

 

[jaylach]

When I turn off auto updates, it turns itself back on straight after or shortly after. And it always turns itself on if I leave the pc idle for a couple of minutes. It does my head in and the next pc is gonna be something that doesn't use windows. However, I don't like apple either so I think I'm stuffed.

Using Wu10man I haven't had any issue with updates going back to default but I don't actually turn them off just set to notify.

If you don't want Windows or Apple take a look at Linux Mint. I have it on an older laptop. The install media is bootable and live so you can check it out before actually installing.

Home - Linux Mint

Linux Mint is an elegant, easy to use, up to date and comfortable desktop operating system.

www.linuxmint.com

 

[jaylach]

I have a question that I have seen no mention through all this. Where were the system backups to restore the systems???

I mean I'm just an individual yet I have multiple images and a clone backup. Don't corporations do system backups? Shoot, toss in bootable recovery media and restore an image or clone... Is this something that is too simple to be used by corporations? Unless I'm missing something basic this should not have been nearly as big of a deal as it was.

Of course it could be that restoring a backup would just result in the faulty Crowdstrike update being installed again borking the system again but, after the faulty update was stopped a simple system recovery should bring the system back to fully operational as to when the backup was done. Granted that some companies set system backups to update the backup when there is any change to the system drive but this is a really dumb practice and should never be done as any issue brought to the system drive would automatically be carried over to the backup defeating the whole idea of having a backup. This is the same principle that has me doing manual clone backups of my system drive instead of just running a Mirror R.A.I.D. array. With a Mirror array any change to the system drive is also automatically applied to the other drive in the array. Doing manual clones give me a working system drive regardless of what happens to my normal system drive. Just makes sense to me... LOL! Mayhaps that is why I have 5 awards from Microsoft...

 

[Rhysp87]

I have a question that I have seen no mention through all this. Where were the system backups to restore the systems???

I mean I'm just an individual yet I have multiple images and a clone backup. Don't corporations do system backups? Shoot, toss in bootable recovery media and restore an image or clone... Is this something that is too simple to be used by corporations? Unless I'm missing something basic this should not have been nearly as big of a deal as it was.

Of course it could be that restoring a backup would just result in the faulty Crowdstrike update being installed again borking the system again but, after the faulty update was stopped a simple system recovery should bring the system back to fully operational as to when the backup was done. Granted that some companies set system backups to update the backup when there is any change to the system drive but this is a really dumb practice and should never be done as any issue brought to the system drive would automatically be carried over to the backup defeating the whole idea of having a backup. This is the same principle that has me doing manual clone backups of my system drive instead of just running a Mirror R.A.I.D. array. With a Mirror array any change to the system drive is also automatically applied to the other drive in the array. Doing manual clones give me a working system drive regardless of what happens to my normal system drive. Just makes sense to me... LOL! Mayhaps that is why I have 5 awards from Microsoft...

Servers and critical endpoints should have backups, yes, but the process of restoring all those backups would take time if you’d lost multiple devices.
Thankfully the company I work for wasn’t affected but the issue if we had been (which I suspect is the case for many places) is that the fix was easy but generally required you to be at the affected device. Accessing affected endpoint devices could be incredibly painful if you’ve got thousands of them. Even more so now that people work from home and devices may be spread across the country.
Ultimately the problem wouldn’t have been data loss or access to a backup, the problem was the logistics of getting those backups applied.