What Everyone Should Know About Cloud Permissions

Matt Carle (00:00): If you want to share those slides. [inaudible 00:00:04]. Oh, I'm Matt Carl. I do product things here at Sonrai, which means I talk to lots of folks like yourselves, listen to the problems that you have, look at the technology and solutions that we have, and figure how to mesh those together to give you absolutely amazing solutions to the problems that you experienced in your cloud. Nigel, if you click next, we're here for everything you should know about cloud permissions. Skip that. Okay. What are we going to talk about today? I'm going to cover the state of cloud permissions just to give you guys an overview of sort of what we see, so when we have fortunate opportunity to speak with you and your organizations about your clouds, about security in your clouds about permissions and identities in your cloud, what are we seeing collectively across that landscape? (00:48): Then I'm going to hand off to Nigel, who's really going to take you through what are the most sensitive permissions in cloud? How do we look at permissions in aggregate? How do we determine what is sensitive? And then really what can we do to develop a plan to tackle really the excessive privilege that we see in a lot of these cloud estates? Nigel, if you click forward. One more, sir. (01:11): So the attack surface is huge. This is an aggregate, so what we've done here is taken a number of customers that we work with. I think there's about 30 involved in this. And this is everything from small to large, a bit more towards the medium to large side. I'm not going to read the core numbers here, but just to sort of give you an idea, 35,000 plus identities. When you're listening to Nigel, I talk and we mentioned identity. (01:38): It can be a confusing term even today in 2024 when we think identity and cloud, we think human, absolutely. We also think every non-human that exists in that world, which could be an instance profile attaching a role to a workload in EC2, it could be a service account running in GCP or it could be a managed identity in Azure. It's really anything that can leverage permissions to take action in that cloud. Lots of humans, way, way more non-humans involved in this problem. What you're seeing here, really the three numbers in the center really hone in on what we see as the problem when it comes to permissions in cloud. Over 20,000 of these 35,000 excessively permissioned, in particular against sensitive permissions in the cloud, and Nigel will really hone in on what sensitive means. But to give you an idea, there's about 46,000 permissions across these three major cloud environments that we tend to focus on. (02:35): 46,000 is a lot of things to look at. We talk about sensitive as being things that are used every day. They make your cloud, what your cloud is. They make your cloud run, but the hope is that you know exactly who's running these permissions. There are a lot of identities in these Clouds with these most sensitive permissions, completely unused 26,000. Actually, the numbers generally somewhere between 60 to 80% of identities we see are completely unused. Problem? Absolutely. (03:03): We're going to talk about that in a few minutes when we talk about zombies. That's what we refer to them here at Sonrai. And then services that are unused. Why would services matter? Your permissions grant you access to services. Those services do things when you ask them to do things like create pre-signed URLs to expose data to the outside world or create internet gateways to allow things into my cloud. (03:24): Services are fundamentally what the cloud runs on top of. Ensuring that the services that you use are the only services that are there ,and that thus all the permissions related to all of the unused services are turned off is key. We say 152 here. If you're a smaller cloud, that number's probably north of 300 in any of the given cloud environments. It's a pretty big problem. How did we get here? It's always a question. Some people ask it. A lot of people know. I think a wise man once said back in 2004, move fast and break things. The cloud's an amazing place. It allows you to move very quickly. It allows you to respond much, much quicker than we used to be able to, but the reality is time often is one of the biggest enemies to least privilege in the cloud. Things get created, they evolve. (04:15): They do different things than they did yesterday and we forget to take away the things that they used to do. So they have a tendency, especially the non-people to collect permissions and generate a broader and broader scope in the cloud in which they can take action. I'll give you guys an idea. Another great example here is dev teams. We ask them to move quick. We ask them to produce capabilities. We ask them to release product very, very fast. I do that. I'm in product. I ask my dev teams to produce product. Give you an example. If I'm asking developers to do things that involve something like EC2, EC2 alone in AWS has 643 permissions. (04:53): So 643 permissions, over 400 of which are right based permissions, things that change that cloud. I'm now asking this development team to figure out exactly what they need and I'm asking them to figure out what they need today and then I'm asking them to change it tomorrow, and then I'm asking to change it again in six weeks and this is how things start to quickly get out of hand in these cloud environments. (05:13): And the other side of it is the cloud ops team's trying to push back. How do we push back against the dev team that's tasked with producing capabilities for our companies to be successful? We ask, we push, we push, we push, but cloud ops teams are not going to delete things. They're not going to turn things off. They don't know what they're going to break, and the communication in these larger clouds can be very difficult. Understanding what can I turn off, what can be taken away? What can we reassess? Often the communication breaks down and the cloud starts to experience this type of exposure. Next, Nigel. (05:48): How many identities are human? I kind of covered this a second ago, but I do like to talk about this. Often we hear we've got this tackled. We use a great PIM/PAM solution and SSO to ensure that our identities can't do things they shouldn't. This number 19%, average, this can be very, very, very disparate. This can actually be 5% to 95% of the identities in your world are non-humans. Most people aren't awake in the middle of the night making their cloud run, which means there are other identities responding to transactions, are responding to queries from your applications. Those are your non-humans. Those are your workloads. Those are your Lambda functions. You name it. This problem, and these by the way, are the ones we forget about, these non-humans, everything you see to the right are really where we see the biggest disparity from an excessive privilege, as well as an unused landscape. Next slide, Nigel. (06:43): Zombies. My favorite topic. Personally, I consider... When we came up with the term zombies, it was really to focus in on those identities in cloud, that they're free to wander around, they exist, they live and breathe, but they don't do anything. They're usually an artifact. They're an artifact of a development process. They're an artifact sometimes of people. As you can see on the right here, this is not generally a huge people problem. I think Rich just mentioned don't use IAM users. I agree 100%. IAM users generally something you should stay away from, particularly in AWS. But if you do have them, which they can exist, you use them for integrations with other platforms, nothing makes for quite like a bad day when your legacy terraforming IAM role, somebody gets a hold of the access key, wakes it up and it starts creating infrastructure in your world to do things that they want it to do versus what you want it to do. (07:38): These zombies really are one of the biggest problems we see. They can account for 60 to 80% of the identities in that cloud environment. They don't do anything and they really are a promise for a problem in the future. I can assume through them in AWS, I can get my hands on them in GCP as a service account that's not used anymore and start taking action on behalf of myself when I shouldn't. These really are something that should be cleaned up, but when you present a cloud ops team with 89,000 of these and ask them to delete them, it becomes a bit of a problem. People get scared about deleting that kind of volume of things. So how do we tackle how do we clean it up? How do we remove these permissions so that we ensure that these zombies can't actually harm us in the future? Next slide, Nigel. (08:25): So where do we start? Here at Sonrai, we're lucky we have Nigel and his team. So that's where we start. We asked Nigel and his team to look across all of these clouds across all 46,000 of these permissions, and start to develop a plan for how can we scale and attack this problem? We chose MITRE, we looked at MITRE and we thought this is a great framework that we can associate with. It's our plan. It's how we look at all the permissions across that 46,000 and really truly arrive at those 3,000 that are really quite concerning and that we should actually wrap our heads around, and our arms around, ensuring that only the identities that use them are the ones that we want to and the others are not. I'm going to now hand off to Nigel. He's going to run through really what are sensitive permissions and what are some strategies that you can use to actually tackle these, wrap your arms around them and take control of them in your cloud. Nigel Sood (09:18): Yeah, so thank you, Matt. Yeah, I'll talk a little bit about what sensitive permissions are, how we've kind of come to that definition. We'll run through some examples and then we'll talk about what you can do in your own cloud environments in order to secure those. And really it is an aim to kind of take that massive set of 46,000 different permissions across AWS, Azure and GCP and kind of digest it into a really small set that we can do something about very easily. (09:42): So what are sensitive permissions? Well, they're the entitlements that can be used to enable, facilitate, hide, unintended access to resources and data. It's the things that directly enable a bad actor to achieve one or more of their aims in the cloud. Like Matt mentioned, we kind of tie this to MITRE. And this may seem like a super broad definition, and it absolutely is. So we'll break it down a little bit further and try and expand on this. Generally speaking, the sensitive permissions, at least in our definition, they're the things that let bad actors get in and stay in. So that's our initial access and persistence. It's the things that grant greater access. This includes upward, as in privilege escalation, and outward as in lateral movement. It's the things that let bad actors evade defenses, disabling logging, tampering, the security features, that sort of stuff. (10:31): Things with exfiltration paths, things that let data out, and the other high-risk MITRE tactics, which is kind of like a catch-all for stuff like reconnaissance, credential access, arbitrary code execution, this sort of thing. And what's the use of this? Well, like Matt said, we were able to take that 46,000 permissions across all three major cloud providers and distill it down into 3,000 sensitive permissions. If you go identity by identity and try and get each one to least privilege considering all the hundreds of permissions that those thousands of identities have, you're going to end up underwater. But if we can focus on these really sensitive permissions first, and then work towards continuing that process of least privilege with the less sensitive permissions later on, that's a really good way to go from a high-risk scenario and lower that risk score as you are continuing to make progress. (11:19): If we can identify the most dangerous permissions and remove them from all the identities that don't need them right off the bat, we've already gone a long ways towards reducing overall risk in the cloud. And like we were talking about, we map these to MITRE. This helps us in a few different ways. One is it helps us stay true to our intent. It allows us to very, very easily point out exactly how a given permission is able to be abused by an attacker, and it also helps us keep track of the overall risks in each cloud and each service within each cloud. So these are the examples that we're going to go through a little bit later. We've got some from AWS, GCP and Azure across some of these different MITRE tactics. But I'm going to pass it back over to Matt really quickly. Now that we have a really good definition for our sensitive permissions, we can talk a bit more about some specific stats for those. Matt Carle (12:07): Excellent. Thanks, Nigel. Granted versus used. So in what Nigel just described, I think it certainly is the key of 46,000 things. We have about 3000. In that decreased scope, we find scale. You can absolutely attempt to worry about 46,000 things. It is pretty much an impossible task. Trying to go to 39,000 different identities today and give them the policy that's most appropriate for what they do, just to do it again tomorrow to some fraction of those which could be upwards of 5%, 2%, don't know, still a lot, and do it again and again and again and again. As those workloads change what they do, how they behave, new product is released. In that sensitive definition, we focus in on what are the most problem not problem concerning permissions if they were to be run by something that should not run them. And we look at how those are granted versus how those are used. (13:04): So when we say granted, meaning in AWS, I've attached a policy document in a role or to a role that grants those permissions to that role, that is entirely different than that role has ever actually invoked EC to create internet gateway for instance. When we actually take that whole landscape of 46,000 and we distill it down to the most sensitive permissions, Nigel, one step, we actually reduced the landscape quite significantly. 51% we drop away. So only 51% ever actually have these permissions granted to them. That's still a lot by the way, to have half of your identities in your cloud be capable of doing one or more of the most sensitive things in that world? Not a great day. But if we actually look at the data in these clouds, which we've done across many, many, many of these clouds, we actually get a wonderful reduction here. (13:56): Nigel, what is it? It's something like 4%? 4% of these identities across your humans and non-humans actually use the most sensitive actions, meaning they actually create ingress or egress, actually destroy data stores. They create identity or create policy for identity. Only 4% across most customers that we see actually use these most sensitive permissions. And arguably, it's the in this 4%, it's what makes your cloud generally run day in and day out. If we can constrain down to this 4%, we ensure that the cloud runs, we ensure that the cloud does exactly what we intended it to do, and we remove 96% of those identities from this picture. They're still allowed to do what they do. They should continue on listing buckets. They should continue on generating reports, but they should not for any reason at all, wake up tomorrow and decide that they need an egress to the internet when they've never ever done it before. Back to you Nigel. Nigel Sood (14:56): Sure. So we've talked a little bit about sensitive permission usage. Let's dive into some specific examples so you can get a bit of a better understanding of what types of permissions are sensitive and why. So again, we'll work through some different MITRE tactics, some examples from different clouds, and then we'll go on to talk about what you can do with those afterwards. So the first one I'd like to talk about is from AWS Lambda, creating function URL configs, updating function URL configs. I won't get too deep into the weeds about how these work, but the risk here is opening stuff up to the world. With AWS Lambda functions, which are serverless ways of executing code, some common ways to invoke those is by hitting an HTTP endpoint. A function URL. And around these URLs you can put up mechanisms to help secure them. (15:44): And you do that using these two permissions that we're looking at right here: CreateFunctionURLConfig and UpdateFunctionURLConfig. And a couple of the things you can do is control stuff like cross-origin requests. You can add restrictions such that only people who have authenticated with an IAM identity in AWS are allowed to invoke this function. And so what an attacker could do with these permissions is create a configuration for an existing function URL that says, "I know in the past you used to require an IAM identity in order to invoke this function, but we're going to change that so that just anybody who has access to this URL, now they can start executing this Lambda function." (16:22): And what this does is it opens it up to the world. Anybody with access to that URL can now ping it when otherwise they wouldn't be able to before. And even if an attacker or somebody doing something malicious within your cloud and configuring these function URL configs, were to lose access to that identity tomorrow, that URL can continue to be abused. So really this is creating an entry point into your cloud that somebody could use to get code running using your existing Lambda functions with whatever parameters they want to provide and get whatever data they want back. (16:51): It's taking those requirements for IAM authentication and removing them, creating that entry point. This is one example for initial access. There are so many others and there are so many other things even just within AWS Lambda that can do stuff similar to this, but we'll keep moving on. Our next example comes from GCP, and this is a good example of lateral movement. So the permission here is compute.instances.osAdminLogin. And what this does is it takes somebody with a GCP identity, so somebody who has access to this permission, and it allows them to go from that IAM identity with a very high level of visibility into what available resources exist and what their configurations are. (17:34): But if you have a GCP IAM identity, you don't necessarily know what's going on inside each compute. With this permission, you can pivot from that IAM identity to a local machine user and specifically an admin. So you're talking system route level access or something close to it. They can pivot to within that system and then continue their exploration from the inside, collecting sensitive data, running arbitrary code as desired. This is kind of textbook lateral movement, going from one level of access to another within the cloud. And you really aren't going to need a lot of people within your cloud necessarily having admin permissions on all of your computes that are running all of your workloads. So this is one example from GCP. We'll move on to an example from Azure and kind of further down the attack chain towards defense evasion. (18:19): So I'll talk a little bit about what Azure policies are. Azure policies are kind of different from the concept of policies in AWS, which are more IAM-related. Azure policies have a lot to do with compliance and auditing. When you create policies, you can attach them to resources using policy assignments to track specific compliance requirements. For example, if I want to secure my VNets, I might attach a policy to them saying, "I don't want these VNets open to the public internet." And the remediations on those policies can do a whole bunch of different things. They can alert people on your SecOps teams. They can even provide automated remediations in some cases using automation to actually bring those resources back into compliance. And what this permission does right here, what we're considering sensitive, the policy assignments exemption action, is that an attacker with this permission can take resources that have policies assigned to them and say, "I don't want this policy to apply to this resource anymore." (19:13): So back to our VNet example, if an attacker wanted to open up a VNet to the world, they could use this permission, the policy assignments exempt action to exempt that VNet from any policy requirements that are already attached to it. And this can be done at the subscription level, at the org level, whatever, and essentially stop whatever remediations would happen from taking effect. So if this would have sent an alert to somebody on your ops team beforehand, now it won't. If this would have automatically removed whatever resources are opening it up, now, it won't. This is a way for them to avoid any sort of security mechanisms that are already in place through Azure policies and making it so that any resources that an attacker tries to kind of keep available and vulnerable, it allows them to do that without being detected. (20:03): So this is one more example from Azure talking about defense evasion, and again, with all of these services, with GCP compute, with AWS Lambda with policies and Azure, there are so many examples of sensitive permissions within these, but these are just a few. So over the last year, myself and a small team here at Sonrai, we've been digging through permissions across all three major cloud providers trying to identify what all the sensitive permissions were in them, and we encountered a lot of problems along the way. And I want to share a few of those right now, just so that if any of you with your own organizations try a similar task, hopefully you can avoid some of the headaches that we encountered along the way. So these were some of the pitfalls that we encountered. (20:45): The first is the rate of introduction of new permissions. All three major cloud providers are constantly expanding their set of offerings and with these new features and capabilities come new IAM permissions that can be used to restrict access. These are changed near daily. It's never a static set. Keeping up to date for the stream of new permissions, at least with the level that we were looking into them here at Sonrai was a full-time job. (21:11): Just in the last five months, AWS released over 800 new permissions to support new features that they were pushing out. And in our expertise, we deem that over 50 of those just in the last five months that were newly released are sensitive and would provide an attacker sensitive levels of access within your cloud. So you can't just go through an exercise of trying to figure out what's sensitive and then saying, "Yep, we've done. Yep, we're done. We've identified all the sensitive things." You need to stay up to date on all the new stuff that's coming out because this set of available permissions is never static. (21:46): The next thing is sneaky sensitive permissions. These are the things that appear benign and very service or domain-specific, and in 95% of cases that may be true, but they also contain less used but very powerful capabilities. And our example here for a sneaky sensitive permission is from Service Catalog. Executing a provision product service action. (22:08): It's a good example of a sneaky sensitive permission because ostensibly it allows users to perform specific tightly controlled actions for resources provisioned through Service Catalog. The example that they give in the docs here from AWS is turning on or off a provisioned ECG instance. It might seem like having the ability to control things granularly like this is great for compliance, so why are we looking at it as something that's potentially sensitive? Well, service actions through Service Catalog work under the hood by calling SSM documents. And one of the AWS managed SSM documents made available by default to Service Catalog is the ability to update CloudFormation stacks. And when you set up these actions, whoever's setting up the action can give it a role where SSM will perform the actions using that role. So somebody calling this update CloudFormationStackAction isn't even necessarily updating the CloudFormation stack with their own identity. They're using a different one. (23:02): So somebody calling this action with an IAM identity that only has access to Service Catalog. Well, now if you call this, you've kind of pivoted to being able to update a CloudFormation stack with a completely different role that has the capability to produce a bunch of resources. If you create an EC2 instance and add a user data parameter to run a startup script, you can use that to exfiltrate the credentials of the EC2 instance, and now all of a sudden you've got a brand new identity that you have access to. There's a whole bunch of crazy scenarios even just within Service Catalog like this. (23:32): And the moral of the story is basically it's not a permissions normal usage that dictates whether it's sensitive or not. It's the worst case scenario. What an attacker could do if they were really determined. And determining the worst case scenario for these permissions is a hard problem solve. Matt Carle (23:47): Nigel, quick question. Last week we were working with a customer and I think it was AWS Connect. I think it was AWS. Did we not find another one of these? Were we not investigating a scenario in that particular service that was similar? Nigel Sood (23:58): Yeah. So AWS Connect is like Amazon's contact center infrastructure. And we encountered something that was basically like this. There was a permission, I believe it was called UpdateContactFlow or something similar, where when a user calls or when a customer calls a contact center, there's a whole workflow that they get booked through. Maybe they dial a number to get routed to agents dealing with particular topics, something like that. Maybe you play a prompt when they first call to say thank you for calling customer service. (24:30): And there's an update contact flow permission that allows somebody to go in and change what that flow does. Maybe you change the words of your starting prompt or maybe you add a new option to the menu, or maybe you want to route people to different agents at a slightly different percentage distribution. And all of that stuff seems very, very domain-specific until you dig really deep into what you can do with this permission and the type of data that you can supply to it. (24:55): And one of the things that you can actually do with that permission is invoke a Lambda function. And when you invoke the Lambda function, it doesn't do it using the role of the individual who set up that workflow. It does it using the role that was assigned to the connect instance. So all of a sudden you can set up the system where when somebody calls your connect instance or one of your contact centers with a specific phone number. They can go and execute a Lambda function and get the results back using a role that they wouldn't even necessarily have access to because they're just dialing a phone number. (25:23): So there's some really crazy things that you can do with this, and there's examples all over the place. There's some really cool ones in SageMaker as well, creating pre-signed URLs to gain access to AWS identities through notebook instances, stuff like that. Matt Carle (25:36): Excellent, thank you. Nigel Sood (25:38): Yeah, sneaky sensitive permissions. You got to watch out for the worst case scenario, what you can do with those. (25:45): And the last thing is permission dependencies. So you've got a good grasp on what permissions are sensitive and you want a straight back access to them. The first step in doing that is actually identifying who needs access to sensitive permissions. AWS, we found 172 different permissions that actually depend on other permissions. And not just permissions from the same service, but specifically on sensitive permissions from another service. And if you want to block access to one of these and then try and perform an action that depends on it, figuring out what went wrong can be really time-consuming. Here we've got our sensitive permission logs, but resource policy and then I try and do something like create a clean room's membership that depends on that permission. And the error message that we get is very, very ambiguous. (26:27): It says something about corrupted log resource policies or something about approaching size limits, and that's not the issue here at all. The issue is that I've restricted access to information that I've called sensitive is not giving me the right information. Usually they're really good when it's direct, but when you're dealing with permission dependencies, it can get difficult. (26:44): So we've talked a lot about sensitive permissions, we talked about some of the issues that we've encountered. What can we do right away to start reducing the permissions sprawl within your cloud organizations? So we've got three recommendations for you today. The first, anybody who's dealt with IAM permission control in AWS has already heard this before. Digital wildcard permissions. These are overly permissive and even when you think you've got a good grasp on what those wildcards expand to, oftentimes there's stuff within there that you don't necessarily know exactly what they do and maybe they are sensitive. (27:18): And that's not even the only issue. An additional aspect of this is the fact that like we said, in the last five months, AWS released 800 new permissions. If one of those 800 new permissions were sensitive permission that matched one of the wildcard statements that you've already put into your existing IAM policies, all of a sudden that's a new sensitive permission that all of your identities now have access to, that they didn't before. Matt Carle (27:41): I want to ask you a question. If you were to guess, so I read these slides the other day and I actually decided to go look at some of our metrics internally. If you were to guess how many identities in a general cloud have access to star permissions i.e, they have a policy with a star permission and that policy is actually granted to them, not just policies with start permission, active on users, would you give me a number? Nigel Sood (28:02): I think I would probably have to guess somewhere around maybe 10 to 15%. I think that's pretty bad, but I think maybe 10 to 15%. Matt Carle (28:08): 25%. 25%. Now don't get me wrong, every cloud's different, but I'm generalizing here looking at some metrics. We have found 25% of identities will carry some form of a star permission. Not calling anybody out. You might want to go look though, because that means you're subject to every permission in AWS today and every permission in the future. I also decided to take it one step further. I thought it was a fantastic opportunity. Of those 25%, how many stars were the reason for sensitive permissions to be associated with these identities? Give me a guess. Nigel Sood (28:45): I'll guess probably around 10%. Matt Carle (28:48): 10%. 10% of identities in AWS estates that we've looked at, that we have metrics on, have access to sensitive permissions, creating internet gateways, destroying data archives, creating policy, creating egress paths through pre-shined URLs, and they didn't explicitly get it from a named permission. They got it because of a star permission that's been assigned to them. That's a wild number by the way. That applies to all of these clouds that have starbase. Azure's just as bad. Anyway, sorry to interrupt. Keep going. Nigel Sood (29:17): Yeah, like you said, right at the end there, this applies to AWS. This applies to Azure, it applies to GCP everywhere. If you've got these used within your organization, no matter what cloud you're in, definitely go take a look and see if you can get rid of them. But yeah, this is one thing. The next thing is if you've got your set of sensitive permissions, you can start restricting access to them. In AWS and GCP are some really great tools for this. You can in AWS, use CMPs to restrict access at the identity level. You can use SCPs to do it at the organization level, and you can use permission boundaries to put a limit on what individual identities can do. You can do really similar things in GCP with deny policies and with principal access boundary policies. Those are relatively new within the last year. (30:00): Azure looks kind of like it has the tools to be able to do this, but it's actually quite tricky. They do have deny assignments, so the ability to deny permissions, but those are locked behind Azure Blueprints and you can't customize them. You might look at roles and go, "Oh, I can specify things and not actions and not data actions," but that just restricts grants within those roles. If you have another role that grants access to permissions, you still get the permissions. It's actually really difficult to specifically deny individual permissions in Azure. Here at summary, we have found a way that you could do this through role control, but it's a little bit outside the scope of this presentation, so I'm going to leave it at that for now. (30:34): And the last thing, the last recommendation we've got for you is use your own roles. All three major cloud providers provide predefined roles that you can use for identities that are designed to do specific tax. Our recommendation here is not to not use those predefined roles, but it's if you're going to use them, clone them, and then use those clones that you're assigning to your identities. And the reason for this is that these predefined roles are changed all the time. They get new permissions all the time when new features come out. And what this does is it allows you the opportunity to evaluate the new permissions within those changed roles before granting them to identities within your cloud. If your identities can already do everything they need to do with the services as you've configured them, expanded access when those predefined roles, change may not be necessary. (31:17): And that prevents the scope of access for these identities from creeping upward and upward and upward as the clouds provide new features. And a couple of quick resources just to help you in getting started with some of this if you want to implement some of those recommendations, definitely check out the Sonrai blog. (31:35): A lot of the data that myself and the team that I work with have found over the last year, get published through this blog. We have monthly sensitive permission recaps where we talk about all the new permissions that have come out this month and which ones are sensitive and which ones you need to be aware of, some risk reduction strategies using SCPs and other cloud native tools, as well as deep dives on specific technologies. Talking about this one earlier this year that we put out talking about all the sensitive permissions that we found in AI services in AWS specifically. So definitely check that out. And some of today's other sessions as well, talking about taking these ideas and then implementing them using cloud-native solutions. So definitely check out these presentations from Cole and from Jeff, but yeah, we'll bring it over to Q&A. Karen Levy (32:22): Wow, great presentation. Tons of information. We added some new stats part way through. I didn't even hear those stats, so thanks for adding those in, Matt. Matt Carle (32:30): Sorry. Karen Levy (32:30): One of my favorite things is these sneaky sensitive permissions that we've uncovered through all of our research. And one of the questions we have, is there a playbook or framework that we could use to track these systematically? A lot of questions are around what tools do we use? How do we evaluate these permissions? I think people are trying to recreate your expertise. Nigel, what advice do you have beyond what you've shared? Nigel Sood (32:57): Yeah, so you definitely need a mechanism for tracking all the permissions that are coming in, and the ones that you've evaluated. And in terms of tracking what's available, GCP has a really good tool that you can use. They have an IAM policy change log that basically tells you from week to week what's new and what's been removed and what exists and what predefined roles. AWS and Azure, it's a little bit more difficult, but they do have endpoints that you can hit that basically returns all of the different permissions that are available at the point in time when you hit that endpoint. So if you build a little bit of automation around it, you can run that every so often and figure out what's new and what's different, and then evaluate all the things that are constantly being added. In terms of actually taking an individual permission and evaluating it to determine whether it's sensitive or not, you do need to put in the work and you are going to need some people with some expertise in what those cloud services are. (33:50): It's not as easy as taking the permission, plugging it into ChatGPT or into a spreadsheet and looking at the name and description and going, "Yeah, this is probably bad." You do need to look particularly at the documentation, oftentimes at the API docs for the actions that depend on those permissions, and then even playing around with it yourself, figuring out exactly what arguments you can specify and what effect that has on the resources that you are hitting. And so in our case, typically when we receive a new permission, our first thing that we'll do is we'll head to the AWS docs, we'll head to the GCP docs, see exactly what's available to hit using the API endpoints, and then we'll go into an organization that we've set up a GCP project or an AWS account, and we'll play around with it and see exactly if we can break anything using it. It's a super fun experience, but it can be a little bit time-consuming. Karen Levy (34:36): It does sound like you have some fun with that. Matt, do you have anything to add to that? Matt Carle (34:42): No, that's a pretty great answer. I mean, it's a task. It is a task. It is not easy. There's some great tools out there to help. Nigel's pointed out a few. Definitely later on today, some of the sessions you'll see some other ways that you can certainly wrap your arms around these. Nigel Sood (34:58): Absolutely. Karen Levy (35:00): Yeah, it's a task and then it's the volume times. The work involved, it's a lot. Nigel's a busy person here. This question gets a bit technical. It's directed at you, Matt. Anyone can answer, but Cole says, great point about star permissions. Any advice you can share on dealing with ephemeral infrastructure? For example, I need EC to start or stop instant permission, but I won't know the iron of the instances. Is that where security engineers should be looking at ABAC tagging to prevent star policies? Here's ABAC again. It came up on Sandy's presentation when we kicked off today. Matt Carle (35:39): ABAC's a lovely capability for those that have not spent much time looking at ABAC in AWS, specifically, there are some amazing controls that allow for constraining actions to be taken based on attribution that's either on the resource itself or in other ways. That would definitely... I mean an SCP itself sitting at the top is definitely going to be a sledgehammer. We've heard that on previous conversations, but you can tune them and you can tune them to do amazing things. And I think ABAC is a wonderful topic specifically when it comes to, I want to target resources that look like this or that are tagged like this. It is definitely a mechanism to pull those needles out of that haystack. Karen Levy (36:20): Great. Just filtering through all these questions here. A question from Rick. What methods are available for doing guardrails against a principal getting specified combinations of permissions? You could have this or that, but not both. Matt Carle (36:39): That's a tricky one actually. This, not that. I mean, there's some lovely ones of these creative function and book function pass role. That's a lovely AWS confused deputy scenario. AWS is full of confused deputies actually, which involve numerous permissions. (36:52): The way we've tackled it here, especially when we look at sensitivity, we tend to focus on ensuring that we have all of the permissions that are offshoots, the most problematic ones are included in that list of the most sensitive permissions. It can be very, very difficult. That's a separation of duties problem, and it requires an evaluation of the workload, what it's doing. And whenever you find yourself in the scenario of this thing has invoke function, create function and pass role, you've got a problem not necessarily with the policy. You've got a problem with the definition of the workload's actual purpose. (37:29): Separating it down actually logically at the compute level, at the actual product level is actually probably a better choice there. It's very difficult to control that outside of it. It becomes a hammer and it's hard to tune once you do that, but this is exactly why these permissions are problematic, specifically when they get over-provided or over-assigned. Nigel, anything from you on that one? Nigel Sood (37:56): No, no, that seems about right. You would not believe the amount of toxic combinations that you've got where you combine pass role with another permission, and all of a sudden that becomes an assumed role because of what you can do with both of those together. Definitely removing access to at least one of them if you can. And if you can't, then maybe you need to take another look at exactly what that workload needs to do. Karen Levy (38:22): That makes sense. Lots of questions around star permissions and restricting sensitive permissions if someone has star. So specifically, there are other people I can't prevent from writing star permissions on certain things. How many times have we heard this when we're talking to customers and different organizations? So what's my best path on making sure, one, I restrict their sensitive permissions anyway, and two, they properly see errors when they try to use star. Nigel Sood (38:53): So I think talking about SCPs, that's a really great way, as Matt pointed out, talking about a sledgehammer from down on high. If you apply those at the org level, then somebody granting star permissions from within a given AWS account, even if they're trying to use those sensitive permissions, if those are denied at an org level in an SCP using star, you can at least block that permission specifically. And you can create exemptions for specific ARNs for specific role names, that sort of stuff within those SCPs for the identities that really need them. But for anybody who's not a part of that exemption, at that SCP, at that top level, you can prevent access to at least some specific permissions. What was the second part of that question? Sorry. Karen Levy (39:34): And then how do you make sure that the person... I think what they're saying is the person who wrote the star permissions, they see the errors. That is how they're being blocked. We hear a lot from customers that it's very hard to determine what policy is blocking a user and that creates a whole thing to unwind. Nigel Sood (39:54): And that really is one of the cruxes of solving this problem, is that you need a way to identify when somebody who's not allowed to use a given sensitive permission tries to do it. And you can set up some solutions around this with logging. You can set up CloudTrail to monitor for specific denying events. But you're going to have to build up a whole set of infrastructure around this. And we were talking about some other sessions earlier to keep an eye out. I definitely recommend checking out Jeff Mongreve's session later, where he is going to be talking about a solution. That might help a little bit with that problem specifically. Karen Levy (40:31): Matt, any advice from you that you've seen in customer environments? Matt Carle (40:37): I mean, we do see a lot of, I'll call it picking and choosing. We do see a lot of SCPs written where we all know collectively that network manipulation is something we do not want to happen anywhere, so we control those and then explicit exceptions are made at the very top level or at least at OU levels above these accounts. It's a fantastic approach. The issue that we've seen people struggle with is how do I scale that? (41:04): Creating networks is a wonderful thing to watch out for. It's a handful of the 950 things that you should watch out for in AWS. Needing tooling that can step in and actually track all the networking stuff as well as all of the other things that are sensitive and provide a scaling mechanism, we've talked about it way back. I would absolutely recommend anybody who has an interest in controlling permissions against workloads or against identities to look at ABAC as a mechanism to do that. We've got some customers we work with that do that fantastically. But yeah, it works well. Eventually scale becomes an issue and you need tooling that can step in and do this across the entirety of the cloud quickly, fast and dependably. Karen Levy (41:47): Okay. Let's see if there's one more question to wrap it up. All right. Nigel, do you have any other examples of the sneaky sensitive permissions? I think that got folks' attention. We talked about the one in the presentation you added in the one on AWS Connect. Are there any other in your knowledge base to share? [inaudible 00:42:06]. Nigel Sood (42:08): Yeah, a couple of quick ones. I think I mentioned SageMaker really briefly, but there's one there. createPresignedUrl, where you can use it and it creates a URL that know ostensibly can be used to grant somebody access to a given particular SageMaker notebook, and you can go and run code in it and you might think, okay, that's not too too bad. Then you realize, oh, once I open this URL, I can go and open up a terminal and go and extract some environment variables. (42:32): And in those environment variables are AWS credentials that you can go and paste into your local terminal and all of a sudden you've got access to a new AWS identity. That's a cool one. I think another really cool one is in AWS Bedrock. So dealing with large language models, there's this concept of guardrails, where you can restrict certain topics from being mentioned, and this is really useful for moderating content and making sure that no harmful stuff goes out. But the other really cool thing that you can do with it is to provide canned responses whenever certain topics are brought up. (43:06): So if I wanted to go and create something malicious about our company, I might create a guardrail that says, "Hey, whenever somebody brings up Sorai's new product, I want to respond with, "This URL that says go sign in here to get access to this documentation," and it redirects somebody to a phishing URL. Karen Levy (43:21): That's great. Nigel Sood (43:22): So taking advantage of large language stuff like that is some more sneaky things. Karen Levy (43:28): That's great. I think with that, we're going to wrap this session. Matt and Nigel are going to continue to answer questions in the chat, and we're going to roll into the next session. I think we have Chad Lawrence from AWS joining us. Thanks again to our presenters. Thanks, Matt. Thanks Nigel.