People, Process, Tech with Eric Pittman, VP of Cybersecurity at Teradata

Eric Pittman [00:00:00]:

Have multiple plans, multiple playbooks for whether it's ransomware, a malware outbreak, any kind of zero day. What if your your primary communication method goes down? Whether that's zoom or teams or something else. Have all of these playbooks ready. Be ready for any scenario, not just ransomware.

 

Arek Dreyer [00:00:22]:

Today's guest is someone who spent his career on the front line of cybersecurity. Not just protecting systems, but rethinking how they're built in the first place. Eric Pittman is the Vice president of Cybersecurity at Teradata, where he's leading efforts to accelerate patching. In a world where the number of patches is ever increasing. From coordinating with the FBI during a major ransomware attack to navigating the expanding scope of cyber leadership, Eric brings a blend of deep technical expertise and big picture thinking. Eric, welcome to Patch Me if youf Can.

 

Eric Pittman [00:00:55]:

Thanks for having me.

 

Arek Dreyer [00:00:56]:

You've recently been streamlining vulnerability management. What made that a top priority for you?

 

Eric Pittman [00:01:03]:

Well, you already touched on it a little bit. We are in a world where just the number of patches are ever increasing and there's no signs of it slowing down. So that's definitely something that was top of mind for me. But more than just having more patches than ever, being more effective about how we're patching is really important. As the number of patches that need to go in increase, that ultimately generates a longer response time just with that increased volume. Unless you're staffing up and up and up and up, which really isn't scalable. And then probably another piece that plays into that is just disjointed tooling. As we have technology in more and more areas across multiple clouds on prem, you know, you name it, it's really easy to have tool sprawl where not everything's talking.

 

Eric Pittman [00:01:50]:

And so then you can create pockets where you're patching really well, but and then other pockets where you're not. And so kind of piecing all of that together, I realized that I needed to take a renewed focus on our vulnerability management program and kind of bring it into the next century and bring more efficiency in a cloud first world.

 

Arek Dreyer [00:02:12]:

Are there any other reasons why organizations are struggling with security patching?

 

Eric Pittman [00:02:18]:

That's probably a multi part answer as well. I'd say the big thing that I've seen, and this is just, this isn't current, this is across my career I've seen that there's been a lot of split responsibilities. Whether it's, you know, maybe it owns OS patching, whereas you have business owners that own patching. Of the applications that sit on top of those servers. Or maybe there's, you know, a cloud team that patches in the cloud while another one patches on prem, or even in some cases you have separate teams patching for individual cloud service providers, where you have the AWS team, you have the Azure team, you have the Google Cloud team. And so that kind of split responsibility model can introduce a lot of, I guess, issues when it comes to patching consistently. And so like I said, that's not anything new. That's something that I've seen for years.

 

Eric Pittman [00:03:11]:

And then probably the last piece that plays into it is just fear. We're all human. And so what happens if something breaks? What if we can't roll back and then what is that going to impact downstream or slow down? Whereas right now, hey, it's deployed, it's working, why do we need to patch it? So that's a hard mindset to overcome. And like I said, we're all human and so we don't want to touch something that's working. But ultimately us as security practitioners know that it's really important that we stay up on those patches.

 

Arek Dreyer [00:03:39]:

So yeah, you mentioned, like, what if we patch something and then that doesn't work? How do we roll back? What are some of the strategies that you've personally implemented to stay on top of the vulnerabilities?

 

Eric Pittman [00:03:52]:

Yeah, so for this one, well, actually for almost anything that I go to tackle, I look at it under a lens of people, process technology and business value. I always like to start with the people because without the people we wouldn't have businesses, we wouldn't be moving forward with innovation. And so it all starts with the people. And then there also are human firewalls. So awareness is key. So that's usually where I start, is I go on an awareness campaign, hey, this is important, this is why it's important. This is what could happen if you're not keeping systems secure. And so that's really where I start.

 

Eric Pittman [00:04:29]:

And then I look at the end users as well. And so one thing that we did was we wanted to kind of break it into individual user groups. You have your end users with their laptops. And so if there's something that needs to be updated, make it easy on them, have a little pop up with a grace period. They can defer if they need to. But ultimately that patch is going to go in or that set of patches because we have to make sure we're maintaining compliance across our entire end user base. And then similarly with server admins, they tend to work out of ticketing queues and so integrating into where they're working so that nothing's falling through the cracks. They're getting tickets, they're getting closed out as stuff's getting patched just through automation.

 

Eric Pittman [00:05:13]:

And then being in tech, we have tons of developers because we're constantly writing our product, our software, and so working where they're working. And so that's populating things in their backlogs, that's giving them tools directly in their IDs or their integrated development environments. So really making sure that we're focusing on the user, the Persona, instead of just trying to create a one size fits all so that everybody can work efficiently and effectively in the way that they're used to. And so that's just the people piece. And then going into process. I touched on that a little bit. Like things like having that grace period, making sure that things are going into those backlogs and all of that can't be accomplished without technology. And that's really where the rubber meets the road.

 

Eric Pittman [00:06:01]:

I'll keep things vendor agnostic for the sake of this conversation because I know everybody has their own little vendors that they like to work with. The only names I'll say is Windows and Mac, because we do have both in our environment. So making sure that we're updating processes consistently across those similarly with the server updates, making that as automated as possible and so bringing in automated patching tools that basically make patching just happen. Long gone are those days where you get a set of patches in, you're going to go through it, you're going to run a bunch of test suites, it's going to take weeks and then eventually you're going to do this manual push out to your server farms. It needs to happen unless it gets stopped. And so that's what we did where we basically put in automated server patching software that it's going to push things out. Now it's not going to push it out to our mission critical stuff day one. It's going to go to, you know, a QA system or a less critical system, but then that way we can see if things break.

 

Eric Pittman [00:06:59]:

And 98, 99% of the time nothing's going to break and we'll be fine and we'll just let the rollout continue as scheduled. And that way people don't need to touch it. The only time you have to touch it is, is to stop it. It's kind of a paradigm shift. Instead of you pushing, you're holding back. And so that's the approach that we took. And then last piece kind of related is with our software development lifecycle or sdlc integrating into multiple points across that. And if we have time, I'll touch on that, what we did there later.

 

Eric Pittman [00:07:32]:

But that's another key piece is it's not just patching, it's overall vulnerability management. Whether it has to do with our code, third party libraries, we bring in obviously the servers, newer technologies like containers and infrastructure as code. So all of those pieces play into a successful vuln management program. And then sorry for the long answer, but last piece is the business value. And so ultimately communicating what are we doing right, and how is this going to scale, how is this going to continue to be efficient in the future? And ultimately how is this going to save people's time so that they can tackle the challenging work instead of the redundant patching? Every patch Tuesday or every week or whatever the cadence is depending on the vendors that are releasing these updates.

 

Arek Dreyer [00:08:17]:

So to summarize, it's the people, process, technology and business value, is that right?

 

Eric Pittman [00:08:24]:

That's right.

 

Arek Dreyer [00:08:25]:

Love it. That it's not just here's a product I'm going to throw at it, it's all of that combined.

 

Eric Pittman [00:08:31]:

You got it. And actually so on that point I'll just expand a little bit. Again, the people are key. And so one of the things that we did at the very beginning was we brought our technology roadmap to our IT and product teams or organizations that we work with because we in information security can't be successful without all these partner teams that we work with. And so we wanted them to know exactly what was coming, why it was coming, what was going to be changing, what we needed from them. There's a little bit of negotiating on the timing, but giving them that foresight and making sure that there weren't surprises was I think, a key success factor in making sure that we were all working together cross functionally.

 

Arek Dreyer [00:09:10]:

Since one of the big themes is that the patch is just the start, what exactly were you able to unlock after solving the patch with more specific examples?

 

Eric Pittman [00:09:22]:

Definitely. Yeah. So a couple of things. So first with, you know, saving time for challenging work, that's definitely an efficiency gain. That's a business value right there. But it also allowed us to kind of shift and move forward the entire vault management program. So what started with a lot of like OS or OS and app based patching allowed us to move more into the tech side of our company, our product security. And so that's not to say that we, we haven't been doing, you know security around our products for years.

 

Eric Pittman [00:09:51]:

We definitely have, but it allowed us to have the time to take a renewed focus on that. And so a quote actually from a guy I used to work with years ago, I love, he said, building simple isn't simple. And that was totally true. I don't think we totally understood what we were taking on, but we decided to revamp the vuln management program as it pertained to application security. And so we did a lot and some of this we were already doing, but we brought in additional tooling. So we were covering our first party code, doing a lot of scanning there with static application security testing our third party codes. These are the libraries that we're bringing into our products with software composition analysis. We're doing like just the concept of golden images that you build.

 

Eric Pittman [00:10:41]:

So host image scanning, container image scanning. I mentioned infrastructure as code, even things like hard coded secrets. You constantly have new developers coming in just out of college, who knows how many secure coding courses they took sometimes to take the easy route and they want to just hard code credentials and that's definitely something that is not allowed. And so we want to make sure we're catching that. And then once our products are actually built in the binaries, we want to test them in a runtime environment. So things like dynamic application security testing, API scanning, I feel like I'm missing some stuff, but it's a huge list of capabilities. And as you're hopefully following, for those of you that know a traditional sdlc, we're working from left to right with those technologies and then even getting into manual reviews. So penetration testing, you know, we bring in independent third parties to do assessments and certify us.

 

Eric Pittman [00:11:35]:

And so all of that got wrapped together into a single program. Now it's not that we had point solutions for every single one of those capabilities. I just listed a lot of those were rolled into platforms. But again, I'll stay vendor agnostic so that we were operating efficiently. Things were working just out of the box, but we were touching on every piece that we could as early as we can in that development lifecycle. So that we're starting on the left, we're getting a lot of feedback to our developers right away around whether it's patches or just general code vulnerabilities. And then we're getting those fixed well before we're releasing. And the way that we did that was we actually went out and we got a purpose built security posture management or SPM platform that acted as our entire workflow engineering.

 

Eric Pittman [00:12:22]:

So you think of all those capabilities that I just listed. Now plug them all into one single pane of glass. That can not only give you dashboards and duplicate findings, but it's air traffic control. It's routing tickets into backlogs for component teams, sending these over to system owners through multiple different ticketing platforms even. And so that was really the key to our success, was centralizing just about everything that has to do with vulnerability management across the entire company, across the different teams, across the different environments, all in one place. Which is, you know, it's been a bit of a pipe dream for me. And so I feel like this is the first time in my career that I've actually been able to get to that point where it's, it's almost that holy grail of all management. Yeah, I'm not going to say it's perfect.

 

Eric Pittman [00:13:09]:

I don't think anything's ever perfect. But I've been really proud of what we've been able to build and centralized in that fashion. The goal is always perfection. And, you know, we'll, we'll get as much progress as we can take.

 

Arek Dreyer [00:13:20]:

Love it. I know you've also dealt with some high stakes incidents like ransomware affecting US cell phone infrastructure. Do you want to share the story behind that?

 

Eric Pittman [00:13:30]:

Yeah. So this was, gosh, it's probably 10 plus years ago back in my consulting days. So there weren't a lot of playbooks for ransomware. I mean, ransomware was a thing, but I don't think there was as much awareness or as much planning that companies are putting into today. And so that was actually one of my more fun stories in consulting. I wasn't even part of an IR or incident response team. It all started where I just got a call one morning saying, hey, we've got a ransomware attack on critical US cell phone infrastructure provider and we need you as boots on the ground right away. We're mobilizing a team, getting flights lined up.

 

Eric Pittman [00:14:11]:

We're going to have people arriving all morning and afternoon, but we need somebody there right now because time's critical. And so as luck would have it, the headquarters of the company was, I don't know, maybe eight or ten minutes from where I lived. So a bit of a scramble. Canceled my entire day and took off as quickly as I could. And when I arrived, it was a bit of chaos. We had all of IT leadership already on site for that company. They had the FBI lined up and on regular calls and they were already starting to basically rebuild their entire active directory forest because they didn't know how the attackers got in they didn't know what they used and didn't. And so they were literally building from scratch maybe an hour or two into this happening, which is not a great spot to be in.

 

Eric Pittman [00:15:02]:

Right. But ultimately it did work out. And so that was the route that they took took because they wanted to make sure everything was built right. And there was a lot of lessons learned out of that. But I think one of the most interesting things I learned at the time, and I didn't know until this happened, was that there's basically like crypto broker firms out there. And so they ended up getting put in touch with one of them where there are these companies that they'll come in and they'll work with the people that put the ransomware on your systems. They'll negotiate for some decryption keys up front just to prove that the data can be decrypted. And so typically they're fairly successful in getting a small subset of keys for free as proof before they even go down the path of, hey, are we going to be paying this thing or not? And then they also have pre set up crypto wallets with lots of different cryptocurrency, with Bitcoin being the most common one.

 

Eric Pittman [00:16:01]:

And they can facilitate payment because if you try to do that day one, you have to jump through a whole lot of hoops, load it in. You know, most of the exchanges want to make sure you're not facilitating, you know, terrorist activities and those types of things. And so the laws around this have continued to evolve. Thankfully, this was back when, you know, companies weren't sure if they would pay or not. And I won't say if they did or not, just in case you guys figure out who it is. But they worked with the FBI through the holding, they worked with this broker through the holding, and ultimately they were able to restore operations. It was definitely a dent to their bottom line for that year, but they were able to scramble teams back together and get things working. We supported anywhere and everywhere that we could.

 

Eric Pittman [00:16:48]:

There were some action items to put together, some ransomware plans or playbooks, and also talk about their technology stack and what they could do better. And so I'm guessing where you're going to go next is what were those lessons learned? So I'll cover those. So the first one definitely was have a plan. They did not have a plan. And so have multiple plans, multiple playbooks for whether it's ransomware, a malware outbreak, any kind of zero day, what if your primary communication method goes down, whether that's zoom or Teams or something else. Have all of these playbooks ready. Be ready for any scenario, not just ransomware. And then practice those plans at least once a year, ideally more often.

 

Eric Pittman [00:17:34]:

I'd say do quarterly, even if you're rotating the plans so that you're getting a lot of practice, not just in incident response, but with different scenarios. And then the biggest takeaway from this one specific scenario was immutable backups, basically backups that can't be changed because if they had those, it wouldn't matter if everything else got encrypted. You could just blow that storage away, restore the unencrypted backups and be on your way. Now, you still got to figure out how they got in, because if you restore from a backup, chances are you're restoring that configuration that got them in in the first place. But at least you're not starting from zero.

 

Arek Dreyer [00:18:10]:

I love the way you provided the lessons learned without even me having to ask. Did you specifically coordinate with the FBI during this experience? I'm just curious.

 

Eric Pittman [00:18:21]:

Tell me. So I was, Yes. I was one of the people in the room. I wasn't the primary coordinator. They were working with some of the executive leadership for the company and so they were the ones asking all the questions, well, what should we do now? What should we do now? Is there anything else that we can do? Should we pay? Should we not pay? So they had so many questions that, you know, I would give my input, typically after those discussions. But yeah, every time there was a call, you know, we'd. We'd all get in the war room, we'd have them on the speakerphone and there was, there's touch points. I think it was like every two hours, something like that.

 

Arek Dreyer [00:18:53]:

Super fascinating. Thanks so much for sharing that story. I want to shift gears a bit and ask what is one outdated mindset that you had to leave behind and why?

 

Eric Pittman [00:19:05]:

Well, I kind of mentioned this earlier. I'd say manually testing security patches. I wouldn't say that I necessarily had to leave this behind as much as I've had to encourage others to do so. And so I. With something like that, I'd say it's more of a three pronged approach. Again, I always start with the people. So awareness, making sure that teams understand. If you wait around until you manually test something, what are the chances it's not going to happen? What are the chances it's going to get deprioritized? The one person you have assigned to doing it is on vacation.

 

Eric Pittman [00:19:39]:

So making sure that there's as much automation in place as possible. Also speed automating that, that's. So that would be the second prong. Speed is critical. I mean, there's, there's some of these zero days that, yeah, the patch gets released, but sometimes it's been exploited for weeks, if not months, if not even longer, in the wild, and the vendor just didn't know it. And so by the time the patch is reaching us as practitioners, there's certain ones out there that they're already getting exploited. And so speed is really important. And you never know which ones are going to be important, which ones aren't.

 

Eric Pittman [00:20:15]:

So why not build that scalable program that doesn't rely on manual testing and can just get these patches in as quickly as possible. And then probably the last piece would be around almost the opposite, avoiding failures or at least reducing the risk of failures. And so that's the phase rollout that I kind of talked about earlier. You know, patching happens unless being stopped. And so you just, you have your set of servers, these ones get it first, and then these, and then these, and then these, and then you start all over again. And that way, you know, you're going to catch if something's wrong, but it's almost going to be the scream test where something broke. It's not. You know, some people may cringe when they hear that, but you look at companies like Netflix, they came out with again years ago, something called Chaos Monkey, where they, they challenged their developers to build systems that were so resilient that they wrote a program that literally just went around and it would crash servers, it would turn off VMs, it would, it literally was a Chaos Monkey.

 

Eric Pittman [00:21:15]:

And if your, if your system that you're responsible for went down and wasn't resilient enough to, to operate even with this in the environment, you weren't doing your job right. And the same thing should, I believe, should be extended across all of it. We should be building for resiliency constantly.

 

Arek Dreyer [00:21:32]:

And I come back to that phrase you just said, building simple isn't simple.

 

Eric Pittman [00:21:36]:

Yeah, for sure.

 

Arek Dreyer [00:21:38]:

From DJing to cybersecurity, what is one skill that carried over?

 

Eric Pittman [00:21:46]:

Oh, yeah, you didn't cover this in my intro, but yes. So back in a formal life, I was a DJ and mc, and so I actually had an entertainment company where we were out doing weddings, parties, corporate events, school functions, you name it. And I don't know that I could say there's much overlap between DJing and cybersecurity, but I could think of a couple things, and they're not even technical things. Even though there are technical aspects of DJing, I'd say first just reading a room. When you're out there DJing, you have to make real time decisions on what to play. You have to be able to read a crowd. That's important in leadership too, whether it's cybersecurity or any leadership role. You know, being able to have that emotional intelligence to understand if something's working, if something's not, and pivot on the spot.

 

Eric Pittman [00:22:39]:

Also, I guess public speaking, you know, when you're out on a microphone in front of hundreds of people, carries over really well when you're doing presentations to executive leadership, when you know you're, you're speaking on an all hands, whatever it may be, just having that comfort, knowing that, you know, you've got hundreds of eyes staring back at you.

 

Arek Dreyer [00:22:58]:

And how much higher stakes can you get than DJing a wedding?

 

Eric Pittman [00:23:01]:

Yeah, exactly.

 

Arek Dreyer [00:23:04]:

I mean, this is, you know, this is big deal and I, I, you know, I come back to your people, process technology and business value. And I think that framework, I think you could apply that to DJing a wedding as well.

 

Eric Pittman [00:23:16]:

Oh yeah, I apply it to a lot of things in my life for sure.

 

Arek Dreyer [00:23:19]:

All right, so one last question for you. If you could instantly patch something in your world, what would that be?

 

Eric Pittman [00:23:27]:

If I could patch people so that they recognize things like phishing attacks, I said there are human firewall. I wish that that was something that we could do. I'm not only a business leader, I'm a father. And so I get worried about these online scams and stuff too, affecting not only my company, but my family. But I'm guessing you're looking for a more technical answer, not people. So if I had to pick a technical answer or maybe I'm cheating by giving two answers, I'd say something that's, that's risky or painful, a platform like, like Java or something that's supporting a lot of other stuff. It's amazing the fear that I still see from, from teams that need to update Java and they're just worried it's going to break stuff left and right. And so if I could just instantly make that work, then I could remove that as a dependency and just focus on, on the actual business applications.

 

Eric Pittman [00:24:20]:

So that would be my more technical answer.

 

Arek Dreyer [00:24:22]:

Both answers are great. I love it. Thank you, Eric, for joining us on this episode of Patch Me If You Can. If you like this episode, hit follow and share it with someone who's ready to lead it and security from the front. We'll see you next time.