Last Tuesday, loads of Linux users—many running packages released as early as this year—started reporting their devices were failing to boot. Instead, they received a cryptic error message that included the phrase: “Something has gone seriously wrong.”
The cause: an update Microsoft issued as part of its monthly patch release. It was intended to close a 2-year-old vulnerability in GRUB, an open source boot loader used to start up many Linux devices. The vulnerability, with a severity rating of 8.6 out of 10, made it possible for hackers to bypass secure boot, the industry standard for ensuring that devices running Windows or other operating systems don’t load malicious firmware or software during the bootup process. CVE-2022-2601 was discovered in 2022, but for unclear reasons, Microsoft patched it only last Tuesday.
…
The reports indicate that multiple distributions, including Debian, Ubuntu, Linux Mint, Zorin OS, Puppy Linux, are all affected. Microsoft has yet to acknowledge the error publicly, explain how it wasn’t detected during testing, or provide technical guidance to those affected. Company representatives didn’t respond to an email seeking answers.
Hey Microsoft: Windows is yours, GRUB is mine. I don’t give a shit if GRUB is vulnerable, I’ll fix that myself if I choose to.
Mind your own fucking business. The most you should ever do is let me know about it, not try to patch things you aren’t responsible for…
The update was meant to fix a situation where an attacker would somehow get grub onto a machine that was SINGLE booting windows and use grub to tamper with secureboot. this fix was meant to only apply in single boot situations where it should be entirely unexpected to see grub. as they said, something went seriously wrong.
if only there was some way people could test updates before rolling them out to everyone
this here is the real issue.
Preach brother
CVE-2022-2601 was discovered in 2022, but for unclear reasons, Microsoft patched it only last Tuesday.
I respect their journalistic integrity for not speculating, but it was definitely because the NSA was exploiting it.
Ehhh that’s likely enough, but Microsoft is also just shit at fixing things
That’s what they want you to believe.
No, they really are. No doubt they do plenty of stuff at the behest of the NSA, but they are also a deeply disfunctional company with conflicts between departments and bare minimum funding for security, since it’s seen as a cost centre
I hate to break it to you but why would the NSA need a security hole in secure boot. They already have all your data from Windows plus Microsoft has the decryption keys.
Because some users are putting that data on Linux. So they want Linux to be killed.
They can’t change grub. But they sure as hell can convince micro$org to search for and nuke it.
Of course no idea if this happened. Just answering why they would might want to.
So all afected people were potential targets?
Potential targets? Sir, thats everybody.
No, collateral damage.
No, intelligence exploits will sometimes affect the majority of computers on a continent
“secure” boot, the industry standard for ensuring that devices don’t run software other than Windows during the bootup process
FTFY
Nah, you can enroll your own keys and set it up so you can be reasonably certain that your boot image hasn’t been altered, validating its integrity against the potential threat of bootkits. I do this with my Gentoo install.
So, no booting into Windows until this is fixed then? Fine by me. Hell, might actually make me uninstall it completely and free some disk space…
Well… It’s the opposite… People affected by this issue could not boot Linux…
Right, but you have to boot into Windows first to even get the update in the first place…
But if you don’t boot Windows first you’ll not be affected by this issue. So my statement is correct
That’s… What the person you replied to said in the first place.
That’s… What the person you replied to said in the first place.
Nice meme, bro!
No, they’re literally saying the same thing
Woah, interesting.
Is that like a legal option because it looks like it doesn’t ask you to provide an image or whatever? Not that I mind either way, just curious if this is prone to be deleted soon or not.What’s the upside of having it in a VM?
Edit: nevermind the legality, found a disclaimer at the bottom of the page.
The upside is you can treat it as just another program with a big flat file that serves as it’s hard disk. You can move a VM between computers, they’re universal. Hell you can move it to a data center and hardly notice a difference. You can make a snapshot, try something out, and if it borks, roll it back to a previous snapshot. You can copy the VM any number of times.
Basically it decouples operating systems from hardware so you can treat a computer like software.
Oh cool! I’ll need to look into that, thanks! Wonder if there’s a way to convert an existing Windows parition into this somehow, installed software and all, because that would be perfect…
Not that I know of, though imaging a physical Windows install to a VM is very possible. I just kinda like the docker solution because it’s fairly lightweight, but if you want a more robust solution, a VM is the way to go. There’s still limitations on both solutions like gaming not really being a thing unless you get deep in the weeds with things like VFIO and Looking Glass.
There is also the option to use two GPUs (or one and not use it for Linux). https://github.com/mr2527/pop_OS-win10-KVM-setup
Yah, that’s the VFIO method I was referring to.
Ah, okey! I saw the guide and thought it looked interesting at first. But then my wallet saw two GPUs and promptly said no.
Why would anyone want to game in a VM?
More than one gamer on one machine? https://github.com/games-on-whales/wolf https://github.com/nestriness/nestri
Secure Boot is bullshit anyway
It is fine if you only accept signatures from yourself. However, that’s a lot of work as you need to sign everything.
Good luck replacing the PKI on your system’s Secure Boot firmware. Most platforms probably don’t support it and have no documentation
How is it a lot of work? There’s generally one sig you have to add on installing a new OS. Sometimes, rarely, one for a new kernel module. It’s not like you sign every single package you boot.
Still takes work. You also need to disable all other keys if you want it to matter in terms of security.
What are you talking about with “disabling all other keys”? You don’t need to do this at all. You’re seriously making a mountain out of a molehill.
Why wouldn’t you disable other keys? If anyone can boot anything why use secure boot?
I think you’re misunderstanding the purpose of Secure Boot. It’s not designed, nor very good at, preventing physical access. It’s designed to verify the authenticity of the code you are booting each time, most generally to prevent remote attacks. Think of it more like how HTTPS works. The reason you commonly have to install new keys when installing Linux is because there are separate ones for the bootloader, the OS, and kernel modules. GRUBs is generally already in the database. The OS can be hit and miss, Canonical generally has theirs included for example. Then there’s the kernel modules. If they were built and included in binary form, they’re usually signed with the same key as the OS. But if they’re built locally, say when you install NVIDIA driver’s, then they’re signed with a local key, which has to be enrolled. So it’s similar to a self-signed HTTPS certificate. A lot of routers use those, and browser’s will throw a big warning you have to click through. It’s the same with Secure Boot. For example, if a virus tries to build a malicious kernel module, it will throw the same enrollment screen, which would let you know something’s up if you didn’t initiate it. There also has to be a password, that you set in userspace, and then re-enter on the enrollment screen, confirming that it’s a requested action.
Disabling other keys won’t prevent someone from simply entering the bios and disabling Secure Boot first if they have physical access, which would let them boot anything. If you want to prevent that, then the methods you would generally use is setting a system password in the BIOS it asks for each boot, or disabling other boot options (or the boot menu depending on the computer) and setting a BIOS password. However, if you’re trying to prevent people from booting other OSes as a way to protect your files from being accessed, well someone could just take the drive out with physical access. The best practice there is to encrypt the drive with something like BitLocker, FileVault or LUKS/dm-crypt (basis of many distros full-disk encrypt features).
Edit: You could also have Secure Boot enabled, delete every other key and set a BIOS password if you wanted too I guess. I haven’t tried, nor read of anyone trying too.
I get to dual boot at work (I run mint btw) and the only reason I ever boot into windows every week or three is to make sure it doesn’t get so out of date that it gets booted from the network.
I guess it’s time to stop that shit! Having windows available is not worth the risk of messing up my work machine. Hell I’m tempted to nuke that windows partition and double the size of my /home partition!
Though I will give Microsoft credit that m365 stuff, including video calls in Teams, work great using the web versions in Firefox. That’s even with the security and privacy stuff cranked up. I only white listed those sites for cookies and local storage for convenience.
Years ago I finally nuked my Windows dual boot after one of their updates broke it. I still remember my laptop booting into Windows and being so confused. Haven’t missed it once.
Whaaaat, you’re having a good experience with teams in Firefox? I’ve run into all kinds of problems with teams under Firefox in linux, particularly with codecs and not being able to receive video. It works better under edge in linux, but unsurprisingly, the best teams experience is under the native client in Windows.
Yeah, honestly it’s worked fine without any fiddling around. If it makes a difference, I tend to do things like let mint use non-free components if necessary, and I know I do have the “play drm stuff” option turned on I’m Firefox, even though the privacy and security stuff is all strict.
It’s just a Dell laptop with a discrete nvidia gpu in addition to the embedded Intel one. I think it works fine though with either the open drivers or the closed nvidia ones, but I don’t know if it touches that gpu.
This sort of ridiculousness is why I got two seperate drives (needed the extra space anyways) and choose which one to boot from the mobo EFI menu.
Yep, I don’t even fuck with grub since that has fucked me over in the past too, I just go into the fucking bios and select it manually lmao
I’m confused - why is Microsoft trying to - or expected to, by the article authors - patch a vulnerability in GRUB?
It was supposed to patch Secure Boot, not demolish GRUB.
That’s why it’s a problem.
I was interested too. It seems Microsoft has released a patch that blacklists vulnerable grub versions from being able to be secure booted even if they are signed properly:
https://msrc.microsoft.com/update-guide/en-US/advisory/CVE-2022-2601
The link was at the top of the article.
Maybe this update somehow affects your UEFI firmware, and it installs a list in there?
It was a Windows vulnerability that allowed an exploit box GRUB
it was a vulnerability in Grub tho, i understand the Microsoft hate but not to the extant of lying.
Nothing in a third party software suite should be able to defeat Microsoft’s security. So yeah, it was a problem Microsoft needed to fix in Microsoft software. If there’s something grub also needed to attend to, that’s a different matter as far as Microsoft’s concerned.
Because they don’t want ignorant end users to blame them if the ancient, unpatched version of GRUB that’s at issue is used as part of an exploit attacking Windows boxes.
windows update can and will always find your dual boot eventually and break it
I got around that by having two EFI partitions, grub linux partition is loaded always at boot and it chainloads to the Windows EFI boot partition if I choose Windows. Windows does not know another partiton exists.
Yet. They will come for you, too, eventually.
Probably, but so far so good after 7 years, only thing is I have to enrol the MOK? key in the Secure boot after major kernel updates. Which amount to typing a password at the MOK boot screen to enroll the key
So glad I recently removed Windows from my former dual boot system completely. Was sick of getting errors during Linux boot up after running Windows for that one piece of software I couldn’t get to work in Wine or Bottles. The culprit I assumed was Windows updates, which I attempted to disable through the registry on several occasions. It would work for a short period and then Microsoft, in all their wisdom, would just reenable updates because clearly they know better than I what I want my system to do. The last time it happened was the final straw for me when I wanted to boot into Windows briefly only to be left waiting half an hour for Windows to apply updates on shutdown. Pissed me off so much I killed the power mid-update, booted up a live partition tool and wiped Windows off my system completely (updating the grub to remove dual boot). That’s when I discovered that not properly shutting down Windows would mark my other drives dirty and make them read only. To fix this I ended up having to insert Windows installation media and pretend like I wanted to reinstall Windows 10 again. Once it got to the stage when it was about to write to the drive I cancelled the installation and rebooted back into Linux. Voilà! Could write to my drives again. To hell with Windows. I’d rather live without that one piece of software and have my system do what I want it to do rather than it second guess me and disregard my instructions. This whole automatic update thing really boiled my piss. At least with Linux I can choose to apply updates when it’s convenient for me to do so.
I have two pieces of software I cannot live without, to the point that I would rewrite them for Linux if it came to that. Running Windows as a VM using Virtual Box has been a nice experience so far. (Given that both software are not CPU nor GPU heavy and could run on a tree if need be.)
I installed windows 11 in kvm based vm and gave it 80GB of space on ssd. I have booted into it abot 5 to 6 times in last year or so. I hate that I have to keep it, but its nice to have when some shitty websites demand that they work only on windows. (I mean wtf, its a f*ing website)
I can relate. Last time that happened, I gave up or trying to find out how that works and just used another computer that was already connected to the TV.
What two pieces of software, if you don’t mind sharing?
I ask because a relative who is a software developer could somehow barely finally leave windows, because of WinSCP, which is, afaik, a GUI for secure copy commands. Why rsync or sftp commands cannot be enough for a software developer without WinSCP was beyond me. But perhaps there is something I don’t know about each of these pieces of software.
BookLibConnect and AaxAudioConverter. I use them do download my Audible purchases. They are both written using WPF (or some other Windows API only GUI lib) and thus cannot be run on Linux. I might rewrite them using the newest C# cross platform library, but that library does not compile native on Linux, only on Windows… (Unless you use the community maintained version).
I did try to find replacement for both for them but their ease of use and the conversion tool for axx to m4b made it preferable to just install Windows in a VM.
As for WinSCP, it is a SFTP/FPT client that is really nice and I did miss it initially as well. But Nautilus file manager has both SFTP and FTP support built into it. And if you want a dedicated client, I can recommend Terminus (but I am not a heavy user, rclone in terminal does most of my heavy lifting).
Great answer thanks for sharing!
Also has ssh available in nautilus, comes in handy sometimes
Today I learned
Has SecureBoot ever accomplished anything vaguely resembling security?
Securing proprietary hardware against peeps installing alt OSes
Yeah, it made installing Linux more difficult, so it actually lowered computer security by pushing you to use windows
Yes, it made people realize we don’t need Secure Boot and it’s just a pit of vulnerabilities.
If it’s a Linux problem why Microsoft has to patch it?
It’s like if someone gives you a ride to the hospital and the doctor treats him instead of you
I’m not sure I follow that analogy, if you get a ride to a hospital you don’t expect it to lock off all other destinations. What happens in the hospital is irrelevant.
From reading the article, this is more like if you walk into a hotel and they burn down your house so you have no choice but to stay. I suppose in theory you could argue in very bad faith that this is a problem with the house since it’s the house that burned, but in reality the problem is the fact they’re the ones who started the fire.
We didn’t start the fire, it was always burning since the World’s been turning
Because people cannot block darn windows updates. Its a real malware only allowed by law
Microsoft: you can have security updates
Users: good
Microsoft: just keep in mind they will make major changes and will totally change the desktop and settings.
Users: wait what Microsoft Edge opens
It’s a problem in the Secure Boot chain, every system is affected by any vulnerability in any past, present or future bootloader that that system currently trusts. Even if it’s an OS you aren’t using, an attacker could “just” install that vulnerable bootloader.
That said, MS had also been patching their own CVE-2023-24932 / CVE-2024-38058, and disabled the fix for that in this update due to widespread issues with it. I don’t think anyone knows what they’re doing anymore.
booting into windows?
Y’all, help a dummy out. I dual boot windows and Fedora. I only keep windows around for a very few college classes that require for screenwriting software. I have not booted into windows in months. I have a screenwriting class coming up in a week.
How worried should I be? I am not great with computers, I run fedora mostly because I support the philosophy of Linux, less for the techy stuff. Please advice, Linux people. I’m scurred.
Does that screenwriting software require a lot of performance? You might opt to install Windows into a virtual machine, as described here: https://www.windowscentral.com/how-setup-windows-10-virtual-machine-linux
Essentially you’re using some software to emulate a computer inside your computer that can run any operating system you want. It doesn’t need to touch your actual operating system installation, you can treat it as just another program. For your use case that sounds appropriate; you occasionally need to run specific software that has low system requirements. This way you can do that without risking Microsoft borking your Linux machine any time it feels like it.
I’d imagine it requires about as much as a word processor, since that’s basically what it is. A word processor with a specialized template and some nifty autofill options. Again, dummy here. If I’m running a virtual machine, can I create a file in it that is saved to my actual machine, or would I need to, like, email it to myself using the virtual windows os?
The latter thing you mentioned would work, but you can set up some shared storage between the VM and your machine. Here is some more info: https://www.makeuseof.com/how-to-create-virtualbox-shared-folder-access/
This describes a Windows host and a Linux VM, I’m sure you’ll be able to figure out the other way around. :)
Don’t use Virtualbox as it is a Hype 2 hypervisor not a hype 1. You want actually KVM via Libvirt. Libvirt also has the advantage of not requiring any proprietary software. (Just make sure to install the virtual drivers)
That’s all fine and dandy but OP said they’re not very technical. Conceptually Virtualbox is a lot simpler to deal with. There’s a lot of advantages (philosophical and practical) to be had with a KVM or QEMU setup for sure, but if you want a simple to understand click-it-together setup then Virtualbox is better. If OP wants to graduate to a better setup then I hope they go for a good FOSS solution eventually but going straight for the deep end is rarely a good idea if you want people to understand what they’re doing.
Virtual manager is pretty straight forward. You click on create and follow the wizard. You can even have VMs running in the background
Sorry idk specifically how to avoid the update, but the linked ArsTechnica article has some advice
Someone here advised & I’d agree: use a Windows VM, for things you haven’t found the Linux version of yet.
Windows’s plan to screenshot everything will include your private artistic work too, so you’ll be doing yourself a favor
When I was still dual-booting Windows and Linux, I found that “raw disk” mode virtual machines worked wonders. I used VirtualBox, so you’d want a guide somewhat like this: https://superuser.com/questions/495025/use-physical-harddisk-in-virtual-box - other VM solutions are available, which don’t require you to accept an agreement with Oracle.
Essentially, rather than setting aside a file on disk as your VM’s disk, you can set aside a whole existing disk. That can be a disk that already has Windows installed on it, it doesn’t erase what you have. Then you can start Windows in a VM and let it do its updates - since it can’t see the bootloader from within the VM, it can’t fuck it up. You can run any software that doesn’t have particularly high graphics requirement, too.
I was also able to just “restart in Windows” if I wanted full performance for a game or something like that, but since Linux has gotten very good indeed at running games, that became less and less necessary until one day I just erased my Windows partition to recover the space.
And probably disable quick boot as I’m guessing the kernel is going to get pissed when you suddenly switch between virtualization and native
I’ve never run a virtual machine, because I’ve always had, frankly, really shitty laptops. Like… Cheapest of the cheap without being a Chromebook. Only decent computer I’ve ever bought got broken within a month. :(
Can I run VMs on really low end specs? The screenwriting software is the only thing I need it for, and I’m assuming it’s pretty much the same as running a word processor.
Provided your CPU has virtualization features (described here) then the performance overhead for virtualization is negligible. So very probably you’ll be fine.
The memory requirements for virtualization is not negligible.
That depends, if you’re going to run a barebones W10 install with what amounts to a word processor I think 2GB should be enough. If you can run Chrome you can run a VM. 4GB if you’re feeling generous, that’s a fair compromise as compared to the disadvantages of dual booting.
I remember trying to push the limits with a Windows 10 VM, and 2GB was the bare minimum;
however, Windows loves to abuse virtual memory (basically using the main storage drive instead of RAM) and if that drive is a HDD the PC is little more than an IoT space heater.A relative of mine has a Windows 10 PC with 4GB of memory and it takes ~ 5 minutes to start Chrome after booting it up; it does have a lot of miscellaneous bloatware on it, though.
However bad that sounds, you’re probably best off disabling all updates in windows. O&O shutup10 has a setting for that. Download it to a pendrive with Linux, and boot windows with network unplugged.
I will do this. Thank you!
Does your device have 16gb of ram? If so install Windows in Virtual manager with the guest addons. It will allow copy and paste along with lots of other features while keeping Windows in its own area.
It was 8 gigs. Someone else suggested boxes over a VM, would 8 gigs be enough for either of those?
No, 8 GB is not enough for virtualization of Windows guests
If you have trouble make a rEFInd USB stick and boot that
Do you know which bootloader you have? There are two popular ones in use currently, one called systemd boot, the other is called grub. From reading this post only grub seems to be affected. I don’t really know which one fedora defaults to at the moment, and it likely depends on what happened during the installation process as well.
If they’re using Fedora, then it is highly likely that they are using GRUB as you have to very much go out of your way to utilize systemd-boot on Fedora the last time I checked.
What do you use? Maybe there is a Linux alternative to that so you don’t have to bother with a VM.
They require a program called Final Draft. I looked around but couldn’t find an alternative
Try running it in Bottles. A lot of programs work there without many issues.
Use Bottles Flatpak
Bottles uses WINE which is way more performant than a VM.
Thank you for the advise! YouTube tutorials, here I come!
Technically it uses Soda + Proton but same diff
How is that different?
Proton works better and is generally more performant
Yes Proton is different from Soda, but how is “technically Soda” actually what Bottles runs? I thought bottles runs WINE, which runs Soda as the runtime
Which are both custom versions of Wine with extra patches? They aren’t something like Luxortorpeda where it replaces the Windows game engine with a Linux one.
Soda is the default Runtime, Proton (and outside of steam you should use Proton-GE) is the Steam one with way more compatibility for Games
Yes, I am aware of what it uses, but thanks for over-explaining. I was commenting on that person’s implication that Soda and Proton, aren’t infact, just variations of Wine.
Out of curiosity, have you tried Fade In?
I looked into it, but I can’t afford it out of pocket. The school pays for final draft, but won’t cover anything else :/ If I could, that would definitely be my go-to
It’s free tho? Except for some minor limitations:
“The free downloadable demonstration version of Fade In includes all key functionality except for online realtime collaboration, and will place a watermark on any printed/PDF output.”
And there are ways around those
Oh shit, this I did not know. I just Googled the price and I guess it only showed the paid version. Sweet! Thank you! If this works, I can officially uninstall Windows! That’s literally the last thing holding me to it. :D
My pleasure. I will mention, that unless the author changed the program since last I used it, it also has a small popup every ten minutes or so, asking if you’d like to buy it. Remarkably, I didn’t find this terribly annoying, and forgot all about it until writing this comment - so don’t let that be a hindrance!
So I just emailed my professor, and he says that I can use fade in if the formatting is the same as final draft, and I buy the license so there’s no watermark. Which sucks, but fuggit it it lets me keep using Linux. Do you know if the formatting is the same? This is only my second ever screenwriting class.
The watermark is only applied if something is printed directly from Fade In: export and print somewhere else and there should be no watermark. As for the formatting, I don’t recall - but I do know, that everything is configurable; so you can make the formatting the same, if it differs
Which screenwriting software? Have you tried running it under WINE?
And do you HAVE to use that one in particular? Or can you use something like Trelby, Manuskript, or Scrite?
The school pays for final draft, and I am poor. But someone else just showed me fade in was free and works with Linux, so I’m gonna try that out!
Ah. I did love final draft when my school paid for it. I’ve never used fade in, but the three I mentioned are all free, too. I’m not sure what version of final draft you’re using, but it doesn’t really matter for this, as its support under WINE is pretty lacking. Good luck!!
Can you install windows in a VM instead? VirtualBox is easy to set up.
Don’t use Virtualbox as native libvirt will be faster and doesn’t involve any licensing.
Depends if you care more about performance or ease of use. Based on the fact that OP hadn’t considered VM as a solution, I assume they aren’t super familiar with hypervisors.
Virtualbox is a pain. Virtual manager is much easier and natively supported. You just click new and then follow the wizard
That’s not at all the case in my experience. Sure virtual box modules can be harder to install, but libvirt has so many issues that the average user has no idea about. I’ve had networking issues, display issues, and so on. At one point it read the display scaling information and scaled down the VM display instead of scaling it up. Furthermore RedHat don’t even support virt manager anymore. They want you to use Cockpit. Honestly the all around best virtualization solution is probably VMWare or something like Gnome boxes or QuickEmu.
I would of agreed with you historically but these days I say libvirtd all the way.
Still having these issues very recently.
It looks like some GRUB versions are fixed, e.g. possibly in Ubuntu from 22.10. Dunno if Fedora has the fixed version. I’m facing the same with my Mint/Windows dual boot; considering not booting windows till I’m ready to upgrade Mint to 22.
If you do get problems, it also looks like you can get around it by turning off secure boot until things are sorted.
If you’re not an experienced Linux meddler I wouldn’t recommend changing your bootloader from the default given by your distribution, but I guess if this is widespread most distros will upgrade their bootlodladers soon to deal with it.
I just tried installing this patch tonight on my windows drive - not because I use windows, just to… you know… keep it updated and secure I guess.
It literally won’t even install. It just fails out every time. Whatever. Microsoft releases so many bad patches lately. WTH are they even doing over there? Windows used to be king and they’ve been screwing it up since 8 came out.
Microsoft fired its entire QA team 10 years ago, and shifted the responsibility for testing onto developers. They also got rid of their dedicated hardware lab where software would be tested on many different hardware combinations.
I have worked in two companies that made the same move of firing QA, and in both the quality of the released software took a marked dive. (In neither company did senior management admit that what everyone warned them would be a mistake was a mistake. Instead they blamed developers.)
These days Microsoft’s testing team is whichever users receive each update first. They rely on users and telemetry to do what should be the job of dedicated testers.
This is hardly a new thing for MS. One of the first emails I remember getting when I got to college back in 2003 was from campus IT begging people not to install the latest XP update because it reenabled a vulnerability to existing malware.
Microsoft fired its entire QA team 10 years ago, and shifted the responsibility for testing onto developers. They also got rid of their dedicated hardware lab where software would be tested on many different hardware combinations.
That…makes SO much sense and explains a lot! Thanks for mentioning it.
