A Practical Approach to Finding Secure SMB Partners 

Matt Lee, Security and Compliance Senior Director at Pax8, opened his presentation at the Nashville Innovate conference by explaining how he came to be where he is today.

He recalled a moment from early in his career that was like a punch in the face for him. His MSP employer acquired another small MSP, which shortly after experienced a cyberattack by REvil/Sodinokibi ransomware gang, leading to a three-million-dollar loss.

That loss included several lawsuits, multiple settlements, visits with the FBI, and ultimately the MSP’s CEO was on the affidavit for the FSB arrest of the threat actor who perpetrated the incident.

In this article, Matt shares some of his insights on how MSPs are not all created equal and what you can do to evaluate them.

If you think about it, the largest economy in the world is the United States, and it grows at a rate of about 2.6% per year in GDP. China is the second, and up until last year was growing at about 5.3%, then dropped into the 3% range, but they will catch up again and likely pass us under this current model.

Currently the third largest economy in the world is, you guessed it—cybercrime— and it’s growing at an annual rate of 15.5% at best guess. This money is coming out of your pockets, the pockets of SMBs, and American citizens’ pockets. That’s what drives me to do what I do.

The Target Breach 

Everybody knows about the November 2013 Target breach. How did it happen? What was the mechanism used by the attackers? Where did the first loss come in? 

It’s HVAC, but what was the HVAC company? It was Fazio Mechanical Services, but nobody gives a hoot about Fazio.  

At the end of the day, it’s all about your name. That’s what everyone remembers, it’s what’s in the news and what’s on the authorization key. It’s all anyone is going to see and remember. Nobody cares about Fazio.  

To put this in perspective, do you think this has come to us suddenly? I mean, most of you have been in Infosec for most of your career, right? Do we think this just came out?  

The ‘Information is Beautiful’ website tracks the world’s largest data breaches. There is a vast difference between what we see looking at 30,000 records from 2009–2012 and what it looks like today.  

2019-2012	2023-2024

It’s quite literally ever-present. You could do the math on this but when you start adding it up, we’re pretty much down to every citizen in the entire world. Obviously not counting duplicates and other things like that, but ultimately this has become a huge problem.  

Who do we think is best suited to fix the problem? It’s us—and SMBs are bearing most of the losses. I’ve been through it myself working on CompTIA’s emergency response team, providing guidance for managed service providers going through massive breaches. 

What I experienced in 2019 before REvil started egressing data was technically not a breach, it was an incident. Back then, there was no double-extortion, so we didn’t touch the data directly in the way they accessed it.

According to the small business data center:  

• 43% of America’s GDP comes from all these small companies. 

• 46% of Americans, roughly 62 million, are employed by small businesses  

• Most small businesses have no employees.  

• SMBs provide work to almost half of the American workforce.  

Ultimately, if we think about it, small business drives the economy, with most of these companies employing between 10 and 19 employees.  

How Do Small Businesses Manage Security?

Most don’t deal with it at all, while a few may use an antivirus/endpoint protection or even work with an MSP.  

I work with Pax8, a cloud commerce marketplace that works with more than 35,000 MSPs that service 500,000 small and midsize businesses with 10 million end-users in 53 countries. Those 35,000 MSPs range in size from very small to very large. 

One of my biggest challenges when working with enterprises or larger players is that MSP doesn’t equal MSP. When you think of an MSP serving Spirit, Boeing, or organizations of that size, they’re linear. For example:  

• You know what their service is, and it has a specific SLA.
• There’s a number to it, and it’s a provided thing we can track.

All things considered, it’s very businesslike.

When you think about the MSP I’m referring to—even though it uses the same three letters—all things are not equal. It’s been a problem for Congress and CISA, or CISA specifically, but when we talk about this understanding of MSP, one MSP doesn’t equal other MSPs. The people making up these MSPs quite literally might have been selling wine at the local wine store and then said, I’m doing this thing. I’m going to be an MSP.

My most popular LinkedIn post had roughly 200,000 views and consisted of a sticky note that said “MSP.” That’s all it takes. You can go get a business license and destroy someone’s world.  So, when we think about this, you should understand that the service provider maturity you may be used to, even though it may not be perfectly mature, it may not be the same as what we’re talking about in this MSP space.

I gave a talk at GrrCON with a friend last year called “MSPs Suck, But the World Without Them is Worse.” During the talk, I broke it down that most SMBs with under 500,000 in sales make up about 14,000,000 of the businesses below, and they spend between 6 and 7 percent of their top line revenue on tech, literally all on tech. Not on information security.

Literally, the laptop that Sally is going to use next week is in the $30,000 number. Let’s let that sink in. Who wants to do that business? Why do MSPs even do that? Does anyone know?

The ScreenConnect Vulnerability 

In economies of scale, a lot of these MSPs provide automation capabilities. They’ve gotten very good at getting their RHEM or reactive hours per endpoint per month, down. They could get it down, so they knew how many trucks they must roll.  They got very good over the last five or six years using these embedded tools.

One of those is ScreenConnect. It’s predominantly used in the SMB sector, but as I’ve said before, MSP doesn’t equal MSP. I do a lot of data research. I like to track how well companies and different places in the world do on patching, how well they close the hole that could be used to destroy you. That’s the question I try to ask.

ScreenConnect is owned by ConnectWise and is a remote access/management tool that lets us remote into a device but also provides back-end capabilities to run very highly privileged commands on the system. Typically, MSPs exclude those ScreenConnect instances from their antivirus, from their SIEM, and from the things that would help them determine malfeasance because they don’t want it to be an obstacle to whatever they’re trying to do.

MSPs don’t sign code, they run scripts that they’ve just written at their desk with no validation or change control processes. One of the things I like to do is track a provider’s speed at patching, when I have the time. People make it easy by having version numbers in headers, exposing infrastructure to scanning, etc.

The ScreenConnect vulnerability was trivial. They realized that if you put a forward slash on setup.aspx, it would take you to a setup page. You could then make yourself an admin user, which then gave you access to that system where you could do whatever the heck you want. That’s the nature of this vulnerability. It’s called Slash and Grab.

I looked at 1000 servers running ScreenConnect in Shodan. Shodan isn’t perfect, and it’s going to be a little bit old. I wanted to curl those and keep track of them, targeting a specific version that they were running. Of those 1000 servers in that set, 455 were reliably online on a consistent basis, give or take. 

Three days into the ScreenConnect vulnerability, I found that out of the 455 servers, 59 were patched, but 13% were all a perfect 10: easily searchable, accessible, and simply indexable—millions of machines ready to run ransomware on.

Now this is primarily an SMB tool—some patched it instantly because it is a trivial patch. It has a high priority and a lot of access and capabilities.

I can almost promise that if I landed on one of these machines, I would win. In the evening of the same day, there were 71 patched devices out of 455 live, so he’s at 15.6% at this point.

For those tracking the story, ConnectWise did something unprecedented. Their board of advisors allowed them to give away the software.  

If you read the bottom line, it says ConnectWise has removed the licensing restriction so that its partners no longer under maintenance can upgrade to the latest version of ScreenConnect.  

They gave this away, which sucked because the revenue is gone. But at the same time, they made a better decision as a company. Following this, upgrades began to increase, but not by a lot. So if the threat team realized they didn’t have the right license and now it’s free, why wouldn’t they patch?

How Do We Better Assess SMB Partners?

I found these comments on an MSP subreddit from October 2023.

I’m not trying to be pejorative; honestly, I’ve been there. I’m also not trying to throw shade because I’ve lived in that world. So, the argument here is how do we better assess our SMB clients?  

Most of us send out questionnaires, and it’s not that we’re not assessing this or don’t already have active programs for supply chain risk. But we might be using the wrong approach.
 
Of course, I’ve completed these forms–the 300 Question Questionnaire, The Stig, and others. But often I’m forced to check the boxes because the questions aren’t clear and certainly inadequate to speak to the shared responsibility model most have with the MSP. I’ve even seen where SMBs have filled them out without even asking if the MSP is doing something. This means you’re getting bad data from many of the SMBs completing them, in the sense that people just respond to the checklist and say: “Yep, yep, yep, we’re good,” and typically we don’t ask for proof to understand it. Some of you may have more mature practices than this. In my mind, the answer is to ask the MSP since most SMBs hire some form of managed services provider, whether that’s across the board for smaller clients or a horizontal service for larger clients. But if you ask the MSP questions about themselves and their client, you’ll come away with a better understanding of cybersecurity posture, capabilities, and understanding.

I don’t always ask questions, but when I do it’s likely to be open-ended. Many of these questionnaires start with “do you have this? Do you have MFA?”

Look at the insurance questions from travelers in 2022. Do you have MFA? A binary response to a non-binary question, on what? MFA on what? Where? Which context? On APIs? On what?

We’re starting to get into those aspects. We need to stop asking yes-and-no type questions that are simple to understand. We should be asking open-ended questions and not very many of them.

CIS Controls

Are you a CIS practitioner, or do you follow CIS? I’m working with several projects around CIS, and they are all working to extend some of the CIS capabilities into how we sell and buy products. Pax8 has released a MITRE-esque list that ties products and capabilities to their taxonomical components in CIS. Vendors typically don’t understand what the products they’re selling do.

Gartner typically drives how we categorize things and what we use.

CIS is a very pragmatic framework. It has 153 safeguards. They prescribe simple things like “stove hot, don’t touch stove.” They’re basic, pragmatic, and guiding things, whereas NIST CSF, SP 800-171, and SP 800-53 have a bit less of that. Much more of this whole “what” thing.  CIS is the “how” to NIST’s “what” and that you can take these and crosswalk them to everything. That’s how I live my life with CIS, and it’s a great way to look at the maturity of these SMB partners.

Control 1 – Asset Management

If I got this questionnaire, I would be horrified. Simply ask an open-ended question of your SMB’s MSP:

“How do you track (Cradle to Grave) assets (End User devices, Portable, Mobile, Servers, IoT, Network Devices, Etc.) for your own Technicians and Assets in use in your own MSP?” 

Then wait. Just let them explain. You are looking to hear how they begin with onboarding of the device, tracking in some way (foil stickers, serial numbers, etc.) You want to hear concepts such as from the beginning, reissuance, and destruction how an asset is managed.

Immature answers will only be of systems, or tools, not of process or procedure management, and certainly not holistic of all device classes.

Next, ask how they perform the same function for their client, who is YOUR vendor!

“How do you track (Cradle to Grave) assets (End User devices, Portable, Mobile, Servers, IoT, Network Devices, Etc.) for your clients?” 

This too can be a varying mix of “we only do their workstations” to holistic and mature answers that use asset management and tracking tools, ASM, and other processes to track all asset types for a client partner and determine each’s risk to the organization.

Control 4 – Configuration Management

Now that you believe they have a good handle on the “what” of asset management, it is time to ask how they configure those devices! Approximately 1/3 of all safeguards require configuration management processes and procedures in control 4.1. And with such a broad set of assets to manage, how does this get done in an efficient and efficacious way? Is it done? Or are assets left vulnerable and non-compliant? 

Ask: 

“How do you manage the vast pool of assets owned by your own MSP that need to be configured in a scalable way? Explain how you take measures to ensure change management is being followed and validate that the desired state is being achieved via documented processes and procedures.”

And ask the MSP about their client, your potential vendor:

“How do you apply these same processes and procedures to your clients’ asset configurations? How do you manage the Change Management Process under a Shared Responsibility Model?” 

Responses incorporating processes and procedures that lean toward using robust Change Management to actionable, repeatable, and automated toolsets will likely have a higher overall efficacy and adherence to configuration baselines.

Responses that talk about manual processes, lack of documentation, and no mention of validation (systemic or manual) likely have no adherence to baselines and will pose a significant risk to your organization.

Control 5 – Identity Management

So, they know what an asset is, and they know how to configure it. We are really getting somewhere now. Next, let’s focus on the “who” we serve and protect. Yep, time to ask about Identity. How does someone get handed a credential? How do we verify them each time, what can the credential access? How is it taken away? Do we give vendors creds? How do we handle passwords? How….

It is far easier to destroy your system with valid, overprivileged credentials than to find at that moment you have not patched quickly enough. While you control the credentials provided to the vendor to access your systems, the theft of them from those vendors due to poor identity practices of your SMB vendor will still have the context of that vendor—reducing that risk benefits all.

Ask: 

“How do you handle the onboarding and offboarding of a new employee? I am really look for answers around Identity, Authentication, Authorization, Accounting, and specifically how you handle credentials for employees and your clients’ environments.” 

Look for responses that include management of identities from cradle to grave, including job role change, and show an overall healthy understanding of the importance of identity validation, and all components of the AAA compendium. Centralization of identity and access to single SSO providers in the response would lend towards a healthy maturity.

Now to the client Identities:

“How do you influence good behaviors around identity management as above for your client? How do they handle third party credentials?” 

You really want to hear that they store third party credentials in a password manager, with a strong security posture, and super bonus points if they use a centralized SSO (IdP) and tie access to the password manager with a healthy conditional access policy to open that vault.

Control 7 – Vulnerability Management

• Assets, check 

• Configs, check 

• Identity, check 

We are starting to find a great set of partners! MSP, Vendor, and Us! 

Now ask: 

“How do you manage the cadence of vulnerability hunting, and patching / remediation, for any assets you manage, especially those that deliver services to your end clients?” 

You are looking for a healthy understanding of patching and vulnerabilities in general. Bonus points if the MSP takes a risk-based approach, where the most impactful systems, vulnerabilities, and activity, dictate a faster patch cycle than normal.

Additionally, it is awesome to hear about patching around network, IoT, Operating Systems for Mobile, End-User Desktop/Laptops, Servers, and Network Equipment as this will show maturity.

Ok, this is a big one:

“Tell me about your client’s patch schedule and roughly what percentage of systems patch in any given patch cycle?” and “How do you handle patching any on premise or self-hosted Line of Business Applications for your client?” and finally, “What percentage of the asset inventory for your client is patching managed by your organization?” 

The challenge many MSPs have is getting client buy-in and self-governance, and as an outsider, MSPs are fighting an uphill battle to get reasonable patch windows, risk-based patching, including emergency patch approval, and compliance by the end user humans. If your MSP has a patching process in place that has these, you can make the logic leap that better governance is in place internal to your prospective partner organizations. 

The easiest way in, besides identity, is lack of adequate patching, as this has bitten the organization of MANY of the readers of this article. Look back to Proxy Shell, Move It, Log4J, Pwnkit…and you will find many victims that simply did not have the operational ability to patch before the POC and weaponization began.

Control 17 – Incident Response 

If you have made it this far, congratulations, you are likely ready to hire your SMB Vendor, and, by proxy, their MSP partner who has admin access to their systems, and by proxy, potentially your systems. 

Now let’s try to find out how an MSP and their SMB Partner treats incident response. Identify and Protect are the two functions Left of Boom. Right of Boom you have Detect, Respond, and Recover. By my basic math, that means 3/5ths of the NIST CSF pillars are after an event occurs. We are Detecting, Containing, Eradicating, and Recovering. How well someone (MSP and your Vendor, their client) does this will often impact the outcome as much or greater than the Identify and Protect layers. 

Ask an MSP this simple question: 

“Describe your internal process when you have a cybersecurity event or incident involving your own systems or employees?” 

You want to feel they actually have one, and that they have practiced it. Tabletops should be mentioned, and after-action review should be mentioned. They should know who they must contact, have assigned roles and responsibilities, and failover personnel. 

Now ask them to: 

“Describe (if present) your current established incident response plan with your client our prospective vendor.” 

You will want to show that personnel from each side have been defined, each side has and knows their portion of an incident response operation. Bonus points if they tabletop with the client personnel on at least an annual basis. Client insurance, counsel, and required regulatory or contractual contact are documented and encompassed in the plan.

The TLDR

Too long, didn’t read:  Stop asking the long questionnaire they’ll lie on and ask open-ended questions where they can only tell you the truth. And, by God, if you see the ChatGPT elements, you’ll know they lied.