Exploring Practical Steps for Cleaning up Identity Sprawl
A lot of security tools act like finding the list of unused identities is the hard part. The reality, however, is that running a scan takes 30 seconds to uncover a year’s worth of work. Anyone who’s managed a cloud environment knows that identities sprawl before anything else. Even Latio’s tiny AWS account already has 7 users and 34 roles that I’ve used for different services and different testing at different times. In reality, only 2 of those users and 3 of the roles are regularly used. More to the point, none of those users should even exist – they should be roles for temporary access. But like most companies, that’s something for my future DevOps team to take care of later.
Security engineers like me tend to get really fixated on that idealized scenario – we know that everything should be ephemeral, least privileged roles that are all granting temporary access to resources only when needed. This was the original vision of CIEM – or cloud identity and entitlement management – to help teams identify these over-permissioned roles and update them.
Identifying Zombie Identities
Luckily, AWS has added a few features to make identifying zombie identities easier. In the old days, there was just the user’s page, sorted by console last sign-in.
Unfortunately, there are a lot of gotchas with just doing this:
- It’s not accounting for roles
- It’s not accounting for SSO users
- It’s not easy to automate
- It’s difficult to separate the multiple access keys from console use, and why an identity exists in the first place. Usually, names don’t reflect real use!
The next easiest thing to do is use the Credential Report:
This creates a CSV report of basically the same thing that was displayed on the user screen. This is handy for auditor hand offs, but typically not much better than where we started.
Back in 2019, AWS introduced the AWS Access Analyzer. This tool started very basic – looking at 90 days of CloudTrail logs to determine what permissions a user account was using, and you could use those results to tighten permissions for users.
Since then, the functionality has been greatly improved:
While no one has ever accused the AWS UI of being nice to look at, they have added a lot of more usable data here since the initial launch. Unfortunately, as I’ll argue later, it still doesn’t solve the core issue with zombie identities. Creating an unused access and external IAM Access Analyzer is simple from this page, but be careful about what you set the “unused days detector” to since you can’t change it without creating a new analyzer.
A quick note: at some point, AWS Access Analyzer got weirdly expensive. As a security engineer, I’ve been self trained to click the enable button before reading anything, this is why I’m just realizing this now. The pricing page shows that even a small account with 100 users in 5 accounts will run up about $3,000/year.
For our purposes, let’s look at using AWS Access Analyzer to find unused roles and credentials, is it worth the cost?
Figure Out What Permissions to Change
Going back to our users page, my AWS account has 7 users, of which 1 account is used via the CLI, 1 is just for console access, and 2 are vendor keys from testing (yes, they should be using roles, but we’ll give them a pass for now).
On roles, it’s important to note that this is probably the single most overlooked aspect of AWS security. I have 34 roles in my account, even though I personally use 0 of them directly, and only 3 of them were created by me to run https://list.latio.tech. Of these roles, 5 are related to vendors, 3 are related to my web app, and an unknown number of them are related to services.
Identifying these service roles is the hard part, take these eksctl roles for example:
I remember these roles getting created when I made my first cluster with eksctl in this account, but I don’t know the details of why they were created because it was done behind the scenes. Imagine I’m in an enterprise with thousands of similar roles; it’s impossible to tell who, what, where, how, and why these roles were created. More importantly, I won’t know if they’re needed again – maybe they’re essential for deploying the EKS cluster in an emergency? These kinds of questions are what create inaction for security teams on these findings.
In sum, I have 3 unused users and an unknown number of unused roles. AWS Access Analyzer helped in two main ways: first, it helpfully split out individual access keys in case they’re used for different things. Second, it helpfully showed unused permissions and gave a new policy suggestion to update those permissions.
At this point, AWS Access Analyzer would seem to suggest that I click-ops my way through every policy, creating custom per-user policies. In other words, beginning the time-consuming nightmare of trying to create per-user least privileged policies. This is what many companies have in mind, but it’s a project that just doesn’t warrant the time investment needed.
Doing the Needful
The first annoying thing about using the analyzer is that you can’t take any action directly from it. An “update policy” button would go a long way here. Instead, you have to click the finding, go to it, then go to the user, then delete the access keys.
Let’s see if the AWS Access Analyzer makes us feel comfortable deleting anything… long story short, it didn’t help as much as I wanted. Here were the problems:
- The analyzer doesn’t give broader context about the role or user, like what assets it’s deployed on or what IP addresses its last calls came from. In other words, you can’t figure out why the existing permissions exist.
- This is still scary to automate – it flagged identities I don’t want to delete, which means I can never “just do this” when it comes to deleting identities.
- The workflow is very time-consuming because most orgs will disable an identity, wait a week, and then delete it later. Multiply that across thousands of findings, and it’s a never-ending process.
- The unused permissions are still scary to change because you still need to know what a service is doing before you feel comfortable blocking actions it might need for a rare action.
Some additional issues with AWS Access Analyzer are:
- Understanding these results in an SSO context, where the disabling process can go beyond disabling the role on the AWS side or where multiple users likely share a single role. In this context, the AWS IAM Access Analyzer wouldn’t show you much.
- The analyzer suggests policies, assuming you’ll be okay creating what are effectively per role permissions. While this is good from an immediate security perspective, the downside is creating a ton of added complexity if you’re not already using IaC to define per-role permissions.
A brief note: the external access analyzer is a neat way to gather data on your third-party connections. However, it has the same problems with not providing context as the unused analyzer, so I view it more as beneficial for auditing.
Conclusion
In my experience in the past, and from this latest test drive, AWS IAM Access Analyzer is just a small step towards identifying and removing zombie identities. In my opinion, the core problem with zombie identities was never spotting them – sorting by last use time was always fine for that. Instead, the key problems are:
- Knowing why those identities were created
- Knowing why they’re no longer used
- Knowing if they will be needed in the future
- Being able to automate a process for removals. Usually, something involving a daily lambda function hitting cloud APIs, sending to SQS, and sending slacks with opt-in/opt-out
For most orgs, getting to number 4 is not something their in-house engineering teams can prioritize. Even if it was, it’s not the kind of work that tends to get prioritized at mid-market businesses!
The most useful part of AWS Access Analyzer is seeing what permissions are going unused by existing resources; however, I’ve rarely seen security teams in a place where they feel good leveraging these findings. These are always placed in the “deal with it later” bucket, where they’ll stay for quite some time.
Vendors like Sonrai help make this workflow a lot more actionable by maintaining the automation for you and quarantining zombie identities, which means all permissions are restricted until a workflow to grant them back is completed.
Their solution, the Cloud Permissions Firewall, does this for unused sensitive permissions, too, by setting custom block policies, creating approval workflows, and doing it across all of your accounts in a single place. This workflow is a lot more scalable than clicking between individual findings.
As a plus, they’re not paying me to say this, but the pricing even seems comparable to AWS Access Analyzer when you’re getting significantly more value out of Sonrai.
