JIT vs. Long-Lived Roles: Evaluating Cloud Security Strategies

James Berthoty (00:00): Thank you. Thanks for having me, everybody. I'm going to just share my screen here to get the slides going. I am super excited about this talk because I have wrestled with just-in-time access for quite a while now, and I think there is a huge difficulty in assessing the actual value of this because it seems like very quickly we went from just-in-time being this Slack workflow notification thing into being an entire approach to cloud and application security, and it has no end to the possibilities. (00:37): So really what I wanted to do was sort of assess through this time why we think this way about just-in-time access, and what some of the pros and cons really are, and then try to strip out how much of the value is a workflow thing versus a security thing. Because so often across security and IT and cloud security, I know we've all experienced this, where it is not at all easy to tell at what point you're doing real security work versus when you're just trying to make people's lives better and more usable. A good example of that is trying to go down the YubiKey rabbit hole of, man, those things can get real nerdy and secure, or they can be not that much, and it's really hard to balance the user satisfaction with the experience and what you're doing. So a simple question like just-in-time access can very quickly turn into a rather large spiral of, what are we doing? Why are we doing it? What's the security aspect? (01:34): So first just about me, I'm James. I started this Latio Tech thing about a year ago. I've been doing it full time for about two months. So I still consider myself a security engineer. It's still fresh enough that I'm not lying to say I'm a security engineer. And before this, I was at PagerDuty, which ironically for the purposes of this talk is actually super applicable experience for just-in-time access because a lot of what we were doing, a lot of early integrations in this category were hooking into PagerDuty to see when an alert triggered and then escalate permissions, assign an admin role or something to whoever was on call to take actions in an environment. Before PagerDuty, I was at ReliaQuest as well as at a startup called Very Good Security, working in different security roles, product roles, all across engineering really. (02:26): And my quote there is just one of my favorite quotes from when I just got started in cloud security. Someone from our security operations who was a poor kid straight out of school, and I'm saying poor, not because they were poor, but because I feel bad for them having to deal with a cloud security alert so early in their career, but their suggestion to me was if I had validated that the Amazon Web Service was running on the server. And it speaks to the broad sense of how much we ask of people, I think, when it comes to cloud and application security. A lot of what I want us to do as a community is to just know that we're not alone in the complexity of this, that it is super difficult, whether it's the identity stuff, the workload stuff, the application stuff. None of this is easy, and we're all sort of learning it together. (03:17): And so, as part of that learning it, I think this is a weird slide for me to make about the history of just-in-time access because honestly, it feels like just-in-time access just became a popular thing about six months ago. Maybe it's still on a hype cycle. Who knows, right? Only Gartner can reveal the truth to us of where just-in-time access is as far as hype goes. But within this brief history, we've seen the immediate need for a lot of people who are providing this capability to try to expand into bigger platforms. And I think that's what's led to a lot of the confusion that I've experienced, which is like, how valuable is this technology really? Is this a way to, as we see, the full PAM platform, which is privilege access management? Is this going to rip and replace my entire way that I think about accessing different things across my entire tech stack? Or the first point, is the value really where it started? (04:14): The start of this just-in-time idea was that most organizations, especially where I was at early in cloud security, it was very intuitive of us to create these break glass roles that could bypass MFA, could bypass Okta that were these pure admin roles that were meant to be used in a worst-case scenario. Imagine Okta's down, imagine all of our other services are down, Duo's down, whatever we're using for multifactor, we need a way that we can still admin our environment. And so, that was the purpose of these break glass roles. And over time, we started making little tweaks to those and having different teams have their break glass roles. Maybe someone who was on call would get an elevated permission set during that time. And so, that led immediately into integrating with tools like PagerDuty, with Slack, and creating these little workflows around, when something happens, escalate this user's privileges to more than they would really need. (05:09): From that, the concept of just-in-time started to get expanded into workloads, which then got rebranded as non-human identity more broadly. But just the idea of, what if we gave our workloads just-in-time identity, so that say you have a Terraform runner and you escalated the permissions on that runner only for when it was running a Terraform apply, and the rest of the time it didn't have any permissions as a simple example. But then, from there we've seen these combinations of features become sort of these full platforms, but at the heart of them is really just-in-time access. And so, that's what I want us to really dive into in full today is understanding what's the security benefit here, if there is any, and what's the workflow usability benefit and what's the trade-offs there and which one are we really going for when we do this? (05:59): First, just to talk about why I think just-in-time is appealing, and that face is the face I made hearing about just-in-time for the first time. I thought it was the answer to all of my problems in the entire world because it appeals to how security people think. And a lot of my reaction to just-in-time reminds me of the reaction that I had learning about least-privileged cloud permissions for the first time, where we get this starry-eyed vision of like, "You mean I can define every single action that every user in every role in every application can take in my cloud?" And you create this vision of 2,500 permission sets with perfectly-scoped privileges to only do what's happening, and they're only based on workflow requests, and developers have to ask for permission, and security is the authority to come in and approve or deny. (06:53): And it creates this vision that if you've done that and you've gone down that path, you know that it just doesn't scale. You end up with hundreds, thousands of permissions that mostly go unused, or you're just drowning in tickets of people trying to ask for additional permissions. Or they don't know what permissions they need. The developer's playing with RDS in their staging environment. They don't really know what they need yet. They're still messing with it. And so then, all right, you give them the full RDS star access, and then now they can go a little further. And then, you forget to take it away later, and now that permission's overprivileged, and that gets expanded over time to these 2000 permissions. (07:29): And I think just-in-time to me gave that same natural appeal where, "Oh, you mean I can only spin up those least privilege roles only when they're needed?" What if instead of that role sitting there permanently, it only existed for a short amount of time? And then, another benefit is the idea of consolidating all of your logs into a single place because if you have this access trail of who has requested what permissions when, and you expand it not just for these escalation use cases, but just broadly, it has this native appeal that you're going to get all of this in one place. And then, finally that least privilege on demand idea where once again we think we're going to get these standard set of policies people can request. Everyone's going to know all their policies and what they want. They can request, get it assigned. And so, this slide sounds like a sales pitch for just-in-time access because it's mostly highlighting why I got so excited about it, and a lot of the platforms have built around these ideas. (08:29): But let's really take a minute and just dig into what's actually happening here from an AWS specific, but this applies to other cloud roles, architecture. And the first reason that we have to split this apart is there's a difference between using just-in-time for machine-to-machine access and using it for human access to different machines and different roles. And so, I just want us to first level set here with how access within AWS works in general, which I know there's plenty of talks about this. And this isn't so much about just how the access works in general, but specifically about how long the credentials exist on a machine for. (09:10): So this idea from a security perspective, the idea of just-in-time is that there is not a permanent credential sitting on the machine. But what's interesting from picking this apart with this particular view in mind is that in general, that is already the case that credentials are not permanently sitting on users' machines. Now, obviously there is user access keys, which a lot of people still use. There's a lot of them out there, but we've already widely agreed that we should switch wherever possible to using roles, one benefit of which is the short-lived credentials. (09:43): And so, there already exists a cloud native solution for implementing roles. And really that's the thing of evaluating is just-in-time access might make the user access key scenario a little bit better because it's not like a permanent access credential sitting on my box. But if I'm using this flow, which most enterprises are using, where there's some off to Okta happening, and then I'm assuming role into my environment, and then I'm granted an STS token where the security team has already set within it. Within the STS token, I've already set a timeout period, whether that's an hour and I really want to frustrate my DevOps team all the time, or it's 12 hours because I've given up and allowed them to win the fight, or it's some compromise between those things, there's already a timeout on how long that token is going to work for either a user or an attacker. And that STS token is sitting on their box and only usable for a specific amount of time. (10:38): So in a way, what I want to emphasize with this is that with this normal off flow, it's already just-in-time in the sense of, there's not a permanent role that has a live credential always sitting on the user's machine. And AWS doesn't talk about these roles as being just-in-time because that's not how they talk about it, but just highlighting that the reality is that they are these short-lived credentials, and that's why we moved to roles in the first place. (11:06): And it's a very similar situation on machine-to-machine access, where there's some sort of script or application sitting on a box, it curls an endpoint, which is the IMDS stuff and the metadata endpoint for the instance roles, it gets back an STS token from it. And so, it's the same general sort of workflow that's happening where there's no long-lived credentials sitting on the server either (11:28): And so, if we think then from a just-in-time role in what it's doing, let's take that existing architecture between machine-to-machine and the user access patterns. And now let's just layer in a just-in-time provider and think about what we've done when we've done that. And so, again, the pitch of this makes it sort of unclear what's happening, but all we really did was add a couple of extra steps, and it's unclear what the benefit of doing that was. And again, this is talking about the idea of like, I'm going to implement a just-in-time everything approach to access management. (12:08): All we've done is made it so first from the human access pattern, the developer has to know what role they want, and then they request it from the just-in-time provider, which is sitting outside of this entire service with its own SaaS provider. And then, that access provider is creating a role in AWS temporarily, and then the same flow is happening where ultimately return to the developer is an STS token. And so, really when we've done that, we've increased complexity, and the benefit entirely relies on what was the workflow of requesting that role. And was it better than the alternative? And that really depends a lot on your existing architecture in terms of, if developers have a lot of roles that they're choosing between, there might be some benefit to having someone help select which role they want at which point in time. But if you already have some consolidation of roles, making it just-in-time as far as the role being there didn't really do that much. (13:06): And we'll talk about that a little bit more from an attacker perspective. In the next slide, we're going to layer in the attacker's perspective and think of it that way. But before we do that, let's apply this over to the machine-to-machine side, make sure we're not missing anything. So in this case, it's the same thing where the script or the application wants the credentials. It then talks to the third-party provider, and that third-party provider again generates the role. (13:29): Now, one potential benefit of this for machine-to-machine access is that it can spin up roles across different architectures. So here I'm focusing specifically on AWS, but there is a potential benefit of maybe the machine's going to create just-in-time API keys into GitHub or just-in-time essay accounts in Kubernetes or all sorts of really nerdy stuff. But all we really did then was again blow out the complexity in a different way because instead of the application interfacing with those tools directly, now it's just funneling through some other provider that sits outside of it. And so, really that's what I want to emphasize is that the credential on the box is still a short-lived just-in-time sort of access key. And all we did by implementing a just-in-time provider was move that access outside of the AWS boundary to another provider. (14:22): And so, if we think about this then from a security perspective of what we've done, the one thing I really want to highlight is that we've increased our attack surface by just adding this just-in-time provider that creates these roles for us. But aside from that attack surface change, the attacker's access to these credentials on the box remains pretty much the same. We've maybe made it so they have to wait longer to get the credential because they're waiting for the just-in-time provider to deliver it to them. Maybe there's different escalation points they can do as a result of that, but an attacker who's already on the box has the same access they had before. And so, from a pure security perspective, that's really my takeaway from just breaking down the architecture in this way is that there's not a lot of pure security benefit to switching to a just-in-time access provision because we've already been doing this with cloud access in general. (15:25): So even if we expand this outside of AWS specifically and think about the maturity of API security and how you should do credentialing the access APIs, we know that there's a lot of services out there that still use standard API keys that are just statically generated, never get rotated. You can set a timeout, but most people don't, and they just sort of live there forever. But we know that there's this OAuth solution we should use instead that's a lot more secure that's generating these short-lived credentials. And so, all I'm emphasizing is that those short-lived credential approaches already exist, and adding another provider just to focus on this really isn't doing much to increase that security perspective. (16:07): So if we go to the pros and cons of these different approaches, there are some real pros to this because I don't want the takeaway from this to be, okay, therefore just-in-time is useless, and there's no point to it. All I'm arguing is that just-in-time doesn't have a lot of additional pure security benefit. An attacker can still get on systems, still access short-lived credentials. (16:32): A better project for increasing security is probably either setting up boundaries, like the Sonrai approach, or going down the least privilege rabbit hole. Either of those approaches is doing more to increase your actual security than just this switch to a just-in-time access pattern that is essentially just moving your control plane for just-in-time access from AWS to now this third-party provider. If in AWS you had 150 roles that had bad permissions on them, and people were confused and there were too many, that same problem is going to happen in your just-in-time provider. If you had too many users that had too many different permissions, and no one knew how to access them, that same problem is going to exist, just now in your just-in-time identity provider instead of within AWS. So that's the argument is that you're really adding a lot of additional complexity, but the outcome from a security perspective is the same. (17:25): But if we then switch to the positives of it, so there are positives to this, but I would argue that they go back to that core idea of why did just-in-time start in the first place. And the reason it started in the first place was to solve that very real issue of doing privilege escalation in a way that was secure, fast, helpful, worked for people. It's a very real problem that, when you have a production outage, all of a sudden you don't really care that much if your DevOps team has the system master's permission in Kubernetes or not. That is not the concern. The concern is how long is it going to take to get the uptime back up, and you probably don't want security as a team having to get woken up to be a part of that process if they don't have to be. (18:10): Because you're essentially doing the same thing that we tried to get rid of with shifting left where developers could own their own services from top to bottom, and security wasn't this gateway blocking everything. We're trying to do the same thing for identity, where developers can contextually ask for additional permissions, and there's a workflow for that. And security doesn't need to be involved in every little privilege escalation that needs to happen across the environment. And so, that use case is still very real and still very helpful. And what makes a good just-in-time provider here, or if you're going to try to build it yourself, the way to think about it is to try to absorb the contextual data of the nature of the request. And that's why the most common integrations here are PagerDuty and Slack because they're looking at things like outages happening, they're looking at if there's an ongoing incident. That contextual data is key to knowing the risk of escalating developer permissions. (19:09): Beyond just that privilege escalation, the other couple of sub-benefits that exist that are a little bit less large in terms of their payoff, but are still there is a minor benefit in that, if you use a just-in-time provider and an attacker's already sitting on your container or in your environment in some way, they do have to wait for the just-in-time provider to give them the access before they can do anything with it. I think that's a pretty minor benefit, but I'm just trying to be thorough and giving the benefits where they may be. And then, the less cleanup, and this is where again, we're just shifting that out to the just-in-time provider. And so, there is potentially some benefit, but ultimately the root problem is going to be the same where you have these over-permission roles because of process issues and because of a lack of elegant simple architecture more than they exist because of a technical limitation within the cloud provider itself that's causing the sprawl to happen. (20:12): So to go back to what does this solve, just to again emphasize some of the positives here, is especially for accounts that are heavily user-based instead of role-based, that's where a just-in-time provider is essentially... That was really the commonality behind this entire argument is that role-based access is already sort of just-in-time access. But if you're not using role-based access, which you should be, and that should be probably your primary goal before looking at a just-in-time security provider is trying to move to role-based access, the shift to just-in-time will make user-based access safer because the credentials don't always exist on the box. (20:57): However, it doesn't fully solve that problem because there's no guarantee of how far you're going to go down this rabbit hole of like, "I've got the user access keys on my box. But how do I make sure developers didn't copy them into their code, or they didn't copy them and back them up into a different file after the provider got them?" There's a lot of ways that those access keys can persist, and so I would argue you didn't fully solve the problem, but they did made more secure at a very base level from doing it. (21:25): Second is the real solve, which is the privilege escalations for emergencies. That is a real, actually helpful use case that's happening. And then, finally that it makes things slightly more inconvenient for attackers in edge cases, and this could really go either way because you'll see that number two under doesn't solve is it's actually increasing your attack surface. So an attacker on the one hand might have to wait around slightly longer for a credential to end up on a workload, but you've also increased the attack surface of your application to now this third=party provider has all of the roles for your accounts if it's a full just-in-time PAM sort of solution. And that's now your highest attack vector, and basically you're just comparing if you trust them or AWS more from a security perspective. And that's up to you, I guess. (22:17): So other things it doesn't solve, the temporary credentials to resources, those are already roles. If you already have over-permissioned resources, those are going to continue. If you already have too many unused roles, it's the same thing. That's not going to fix it. And finally, it's that defense against the misuse of IAM credentials. And that's really where other approaches, I think, really help more to solve this at a bigger scale. (22:43): And obviously this is Sonrai's conference, but the permissions firewall is increasingly becoming an approach to solve this issue holistically in the sense of, you have this problem, and the core problem is too many identities with unclear purposes that have permissions that are dangerous. And they're either unused or they're overprivileged, and we've seen from the last five years in cloud security that we might get really hyped up that we're going to tackle this head on, we're going to finally solve this problem. But the problem continues to persist and persist, that more user accounts get created, more roles get created, the scope expands over time of the cloud. I've never seen a cloud account get smaller ever, and that's just the reality of the situation. And so, the boundary approach here, I think, does more for actual security than any sort of just-in-time, again, from a security perspective. (23:45): So if we go back to why we thought it was appealing, these are the sales pitches that resonated deeply with me when I was learning about just-in-time. And trying to think about, did we actually benefit ourselves by going like a full just-in-time approach? Well, the first is it spins up roles only when needed, but that's sort of what a role is in the first place. And so, there's not really a lot of security benefit there because it was already a role. The permissions were already granted only when they were needed. We haven't done much to change that. (24:17): The second was that it's easier to trace and audit what happened. And we didn't talk about this too much over the course of those diagrams, but really again, all we did was shift the logging architecture from CloudTrail where it was already consolidated and bucketed as well as from Okta, And we moved it into this just-in-time PAM provider's access logs instead. So by shifting the logs, we didn't really make it easier or harder to audit what happened. If anything, we just created another place to sync all of it. (24:49): And then, finally this least privilege on demand idea, we still have this over permission role problem. And so, unless you're going to go through your just-in-time provider and set least privilege roles across the board, that's still going to be the issue that happens within your just-in-time provider, that you're still going through and lowering all of these permissions. And you still have this cleanup problem, but again, you just shifted it outside of your cloud. (25:21): So finally, in summary, I think you should use just-in-time for privilege escalation during emergencies. And if there's absolutely no way that you can shift to roles, and you need access keys for some bizarre reason where roles just won't work, those are the actual use cases for just-in-time. But most importantly, it's the privilege escalation during emergencies. But I wouldn't look at just-in-time for increasing security, as a clean-up role or permission. And honestly, this is the thing I fall into all the time is I just love cool new tech stuff, and just-in-time access is a cool new tech thing, and it leans me to be biased towards it. But yeah, those are the high level summaries of what's going on here. Karen Levy (26:08): This is great. Thanks. See, it wasn't this easy answer of obviously JIT rules, roles are better. Couple questions that have come in, couple questions from the audience for you, James. We have a couple of minutes left. Can you talk a little bit maybe in your experience about the user side of this? So somebody who's being tasked or somebody who was given no standing privileges, and they are asked to use a just-in-time approval process. What have you experienced if you've tried to implement that in terms of user acceptance? James Berthoty (26:42): Yeah, I think it's one thing to conceptualize accessing an AWS environment, but it's a very different thing to get that working on the CLI. It's crazy to me that after all these years, Okta still does not have an AWS CLI native tooling, and you still have to build your own janky open-source versions of it. And I think that's sort of the critical inflection point from a workflow perspective is, if it can happen magically in the background, then it's a very nice workflow thing where it's like I'm requesting admin permissions. It's already tied to an alert that's come in of some kind. It's approved in some way quickly. And then, I can hit accept, and then it just sort of happens. (27:31): But a lot of times, the next thing you get after it gets accepted is like, "Okay, now copy this AWS credentials file and add it to your AWS credentials thing." And it's, "Oh, well, I don't use that, or that's actually managed by this open-source tool we use," and you've immediately blown up the complexity of the developer experience. And so, it's another benefit to the way I think that Sonrai is going about it is that granting it is sort of this magic experience, which is really the only thing that works in the long run. The second you give a credential file to copy paste, you've really broken the entire system. Karen Levy (28:10): The kind of seamless experience is certainly not there in that case. And then, once users don't want to use it, you have real problems when you're trying to implement a security solution. James Berthoty (28:21): And it all goes back to how quickly credential management from the CLI just blows up in complexity, and that's why most people stay away from it. And honestly, this has been a perpetual struggle at every organization I've been a part of is how do we manage developer access to AWS in general. And so, the idea that we're going to come in with a specialized JIT provider and now that's the way we do it is just not realistic for most organizations that already have created some sort of very complex brokering process that's very specialized to how they do it. Karen Levy (29:01): Yeah. Another question here from Tom, let me just read this to you. "Could you elaborate on comparison of JIT to zero trust sessions that are meant to be adopted in nature and dynamic and access levels based on real-time attribution from ABAC," this acronym keeps coming back, "from ABAC assessments, both privilege escalation and decrease or full revocation?" Zero trust term was not used today. Let's bring it into the conversation. James Berthoty (29:33): I love zero trust because I finally understand what it means after years of just being confused in the submission by every firewall provider saying there's zero trust all of a sudden. And so, I've done all this research just to be able to answer this question with some legitimacy. The idea of what's happening between zero trust and JIT as I understand, the reason you're tying those ideas together would be to have a access... The idea is you have to auth before you can do anything at any time, and you're always getting your auth checked basically. (30:13): And I just think, when you look at the real-time attribute-based stuff, there's some providers that are experimenting with this, but I think what happens is the complexity just balloons to insane proportions. I've talked with some of them, and a simple example is you can only access this resource if a Wiz scan has completed on it within the last 30 days or something. But that's a simple use case that makes some sense, even though it doesn't fully make sense. And I think the more down this attribute-based access rabbit trail you go... I've said rabbit hole like five times today, I don't know why. It's my thing for today. The further down this path we go, the more complexity that's going to be there. (30:57): And so, I think we can create this overly complex idea of what attributes, what dynamic access do we want to do. But to answer the question more directly, I think there are some simple things we can do with this like location-based stuff, IP-based stuff. But at that point, we're not doing full zero trust either. And so, that's a long way to say I think there's a little bit there, but I wouldn't go fully down the how contextual do we want to make our access grants. Karen Levy (31:31): That's interesting. You just said something interesting because I was thinking contextual. I've certainly heard that from a lot of the emerging vendors, and one of the things you just said is that's not really zero trust. So as you're thinking that if you're approving access based on certain factors in this contextual access, then that's not zero because there's no human approval. Is that your thinking? James Berthoty (31:56): It's not so much the human approval as much as just I am being asked to verify my identity to take any action, and the way that we typically verify that identity is via username, password. And I'm open to the idea of contextual verifications. We're seeing some experiments with it, whether it's like the magic link stuff. It tends to be stuff like, "We'll revoke your permissions if we see the IP address change," more than it's like, "Oh, this user has the..." You could do it certificate-based, but then it's the same thing. And so, all that to just say, when it comes to if you were to grant access without a human in the loop, is that zero trust? Probably not if the user is never having to authenticate in the first place. Karen Levy (32:51): Right. Riley had a comment, I'm not sure there's a specific question in here, but it's about JIT and Azure and ties in their PAM solution. Do you just have any specific thoughts about how Azure addresses the JIT use case and how it may be unique versus some of the other cloud situations? Do you have an opinion there? James Berthoty (33:15): I have done a great job running away from active directory over the course of my career, and so this would be way outside of what I... I am like three years removed from AD at this point, which is a great blessing to myself, but it makes it so- Karen Levy (33:31): I'm sure there are many people in our audience who understand and also respect that a thousand percent. James Berthoty (33:37): Yes, I don't doubt that it is more complicated, so yes. Karen Levy (33:45): Another question around compliance. "So everyone's trying to just meet these compliance check boxes, we call them, so they can't have any persistent credentials. Is there anything beside a JIT strategy to satisfy this situation for compliance?" James Berthoty (34:00): Yeah, that's actually super interesting because I've never interpreted it that way. I assume you're talking about, you're not supposed to have persistent developer access to lower environments. Typically, all that means, I think I'm very guilty of this, and so I imagine this was a security engineer asking this because we tend to get compliance requirements and look for super technical automated solutions to it. But really all the compliance wants is like, "Here's our PDF of how a developer has to access production," and it's a policy that says they phone call the CTO and they're granted an access role or whatever. (34:42): It's more about having some process by which the access is granted. And I've even seen where the persistent access stuff can get scoped to where you're saying like, "Oh, our developers don't have access to production, but our DevOps team has persistent access to production, or our product support team has access to production that's always on." And so, there's a lot of ways with the compliance requirement, but then to back it up to the just-in-time access, I think that's the privilege escalation idea. And so, it does apply in this case where typically a developer, this is not a role that they have. And so, not for security reasons, but for workflow reasons, you're wanting a way to grant permissions to developers, and this is a fine way to do that. And yeah, I think it's a legitimate use case for it. Karen Levy (35:36): Great. Another question, "I just want literally less junk in my cloud and fewer roles. Could we have something like automated JIT that only works for workload identities so we don't build up a bunch of garbage roles again?" James Berthoty (35:54): Yeah, I mean, shoot, this is probably the most complicated thing that you could try to do, if I'm interpreting the question correctly around creating application level identities. And this gets at the challenge of non-human identity as a category in the first place. I have a LinkedIn post that breaks down the categories of NHIs, and there is no provider that does every NHI that's out there because there's just too many of them. (36:25): And so, the real problem that I foresee with trying to do that is, again, it's a development workload that you have to prove to the business is worth the payoff. And so, something like querying for credentials and then deleting them after the fact, that's going to happen within your application code itself. It's not going to happen via AWS or whatever. Now, you can only move as fast as your specific application provider allows you to because there are some APIs provided by people that only support OAuth workflows. And in that case, absolutely you can enforce just-in-time, but we're back to the same idea of it's OAuth. Is OAuth just-in-time? I would argue, from a security perspective, it is because you're granted a short-lived access key, which is what you wanted in the first place. And so, we don't really need a just-in-time provider. We need more API providers to support OAuth workflows. (37:29): But outside of that, you're going to have to try to justify an application layer. We fetch our API key from GitHub, we clone the repos down, and then we delete the key from memory, and the value of doing that, the complexity of doing that is really debated. There's a great Reddit post about how to do Kubernetes permissioning. That really was interesting to me because it gets into, it's super unclear right now how you should even inject workload credentials in the first place, like whether mounting a drive and inserting credentials that way or fetching them from the Kube secret or fetching it from AWS Secrets Manager. Each of those ways of fetching credentials in the first place has security pros and cons to it, and it's not at all clear what you should or shouldn't be doing. And so, all of that is to say that just-in-time for application stuff, I think, is a lot harder than, "Oh, look, I use a provider and do this now." Karen Levy (38:32): Yeah, sure. Another thing that we haven't talked about is more back on the kind of people process side. Where should JIT sit? It's become, there are some solutions that are standalone. It's often thought of as part of a PAM solution. We had a question in a session earlier today about JIT and PAM and how it relates to firewall. Sonrai has introduced this new solution that typically is used by a cloud operations team. But does it belong in IAM? We just talked about JIT for applications in non-human, which is, where do you think it belongs or typically falls in an organization from an ownership perspective? James Berthoty (39:12): Yeah, I mean, I'm seeing more and more IT operations owning it, which from a security perspective, it's not ideal. And I post a lot about how crazy what our expectations of help desk has ballooned to in the last five years with cloud. Because I started my career in help desk. I was restarting laptops and switching out RAM cartridges in monitors and stuff, but now we expect these same people... And this is, I think, the core result behind the Okta breaches that have happened is these support people have to have very highly permissioned roles for super complex technologies. I don't blame help desk people for a second for not understanding the complexities of the AWS role that they're about to grant access to someone. But they already own the keys to granting all of that access in the first place. And so, this is sort of just the natural extension of where that ownership is going to go. (40:08): And the only alternatives I've seen from an operations perspective because there were hopes early on that maybe security and cloud security would get better funded and be able to fully staff and own this piece, but that's unrealistic for a lot of organizations. It's a very small team that's trying to cover everything from app scanning to cloud vulnerabilities to runtime alerts in the cloud to then they're also going to own cloud identity. It's too much. And if you don't have a dedicated IAM team, then the only option besides ITOps is cloud DevOps stuff. You don't get in the DevOps because you love permissions management. I'll just put it that way. It's hard to make it not an afterthought for people, and so that's why I don't have a lot of hope in this great cleanup mentality from an ownership perspective. It continues to spread out and get owned by more distributed teams. Karen Levy (41:12): I really like how you brought in the reality check of who has the money to basically support it as well. What's funded, who's got the budget, who's going to spend the money? James Berthoty (41:22): Even more than that, I mean, a big eye-opening moment for me that I talk about a lot was we finally got budget at one of my organizations to hire a second DevSecOps person. They would say another James was the idea behind it, and we put out the call to the development team and did a job posting, and I thought we would be turning people down. Who wants to join the security team? I figured it's going to pay more. It's a great career trajectory. Who's not going to want to do this? (41:47): And then, nobody applied. I was begging people to take a look at it. They all said no. And I had to find a guy who started in security, became a developer, and I talked him into coming back to security for it. And it just was a wake-up call to the reality of also it's hard to find people with the skillset who want to take on the career challenge itself. You're not going to find a lot of principal developers, like software architects who are like, "Hey, how would you like to be a security architect?" It's just not a common switch people make. Karen Levy (42:21): Yeah. I think when we're talking to organizations and we're trying to figure out what sits where, it often comes down to who has been hired, who has the interest, who has the skillset, who has the passion for it, and that's where they take responsibility. We have about one more minute. I could squeeze in this last one, but this question is around, "I know it's not a huge difference, but the roadblocks JIT puts up gives us a little bit of time, and that might matter to us to deter an unskilled attacker." I think this is in the presentation when you were talking about they have to wait for the credential. "Is there a way to get the same effect with a roles-based approach?" James Berthoty (43:00): It reminds me a little bit of security by obscurity in the sense of, if an attacker is already on your box, just-in-time is not the thing that's going to prevent the attack. I would focus more on, how do I detect that an attacker's on my infrastructure, more than, what permissioning strategy can I use to slow them down slightly? (43:25): And this is where I'm seeing some amazing innovation because I write all the time about CADRs, like a category I'm coming up with and stuff. I think other people are getting there too. So just to plug that, it's pulse.latio.tech. But there are some very cool runtime providers that are digging into this application, Kubernetes container, Lambda function context, and creating really end-to-end runtime detection response. And they're extending that to the identity side of that, which is they're saying, "If we're noticing that this identity has been compromised, it's more identity..." Why does IDR, it got threat thrown into it, so it's ITDR? I don't know, but those- Karen Levy (44:08): We have to wrap. James Berthoty (44:08): Sorry. Karen Levy (44:09): Sorry to interrupt here. We have to wrap and roll into our next session. Stay tuned for the next session. Thank you so much, James. Great discussion. James Berthoty (44:16): Yep.