This is the first part in a series of posts where I plan to share my experience working in security and in particular as an embedded security engineer. With these posts I hope to document the lessons I’ve learned over the years and to provide guidance which I wish I had starting out on this journey.

Audience:

• Engineer thinking of taking up this role

• Engineering manager hiring for a similar role

• Anyone else that is curious

Background

For the past 3 and a half years, I have worked as an embedded security engineer within the Platform tribe at Wise (formerly Transferwise), firstly as an embedded engineer with 1 or 2 teams and then working wider across the tribe as a staff engineer. During this time, I’ve witnessed significant growth within Platform, including doubling in size and the company’s transition to a public entity. Wise is a fintech focusing on international money transfers which comes with the challenges of being heavily regulated and balancing large risks with development velocity of a scaleup company.

Prior to Wise, my experience was in a small Security Operations team performing a mix between analyst/ IR work and security engineering. When I joined Wise, I was the first embedded security engineer they hired in Platform and I was based in the team who built and maintained our estate of kubernetes clusters.

The combination of it being a new role in the company and being in a team full of experienced senior engineers, it took me a long time to feel like I found my place in the role. This series of posts aims to aid engineers embarking on similar journeys but should also be useful for managers planning to hire someone into this role.

It’s worth noting that most of my experience has been in companies where security is a high business priority, which obviously makes some of the job easier but certainly doesn’t mean security is the only priority. Being embedded in a team where security wasn’t a priority certainly would come with a lot more challenges.

Part 1 - Working with People

In this article I’ll cover some of the soft skills aspects of the role and what I’ve learnt. Wise likely has quite a different culture to other companies however the points below are generally how I work anyway. Prioritising pragmatism and showing empathy for other people has gotten me a long way over the years.

Imposter Syndrome

Being hired as a specialist outside a team’s core domain, you may feel intimidated by the depth of domain knowledge shown by your peers. It’s important to recognise that such comprehensive expertise isn’t expected of you (at least it shouldn’t be in my opinion). However, recognising this alone may not help if you aren’t aligned with your Team Lead and the other engineers on the team. Being transparent with them on how you see your role, should help to ensure you are aligned and identify any mismatched expectations. In my case the specialty is security but I feel like there would be a lot of overlap for any specialty.

What helped me and let me manage these feelings was realising I wasn’t hired to just be another engineer in the team and do the same work. As a specialist, you are there to bring an outside perspective and expertise. Lean in on this and identify where you can have a multiplying effect by focusing the expertise of the other engineers on the right problems.

Even as the “security” person in the team, you are not there to make all the security decisions, know how to secure every technology or do all the security related tasks. Your outside perspective when viewing a problem is valuable and how you can translate that to impact within the team is by driving the conversation on security and by asking the right questions to get the engineers that know the domain best to help you identify gaps. That isn’t to say you won’t do engineering work, you just need to be selective on what engineering work delivers most value - more on this later.

Communication

In my experience, effective communication is essential for delivering value in this role. The team you’re a part of likely juggles conflicting priorities beyond security, and Security teams might have bigger fires outside of your immediate focus. While communication is mentioned throughout this article, here are some sample strategies I’ve found helpful.

This section is focused on getting feedback from fellow engineers, with other audiences you should alway consider adapting your communication methods. For example, with senior leadership or audit functions, suggesting partially formed solutions may have negative consequences if they just accept the proposed solution.

Gathering Input

Something I struggled with at times, was getting ideas out of other engineers on how to solve a problem. There can be many reasons for this but I learned not to take it personally as when I’m in their shoes, someone asking for ideas on a solution can have quite a high cognitive load especially when you have other priorities.

To facilitate easier collaboration, a good starting point is to propose your own solution, acknowledging its potential flaws. I’ve found with engineers, if they don’t like some proposed change that will affect them, they are very quick to highlight concerns. Once you know where improvement is needed you can iterate and suggest improvements or ask specific questions on how they would improve the part they gave feedback on.

Sharing Opinions

Another area where I’ve seen people struggle is around sharing their opinion during group discussions in person or on video calls. Depending on your personality, you may struggle with sharing your opinion in these settings vs with async communication like a document. I’m not going to go into reasons for this but I will share some methods I find useful in group discussions.

Focusing on the problem:

A common issue with group discussions, especially with engineers, is a tendency to focus on solutionising during discussions before there is clear alignment on the problem.

Group discussions are expensive and when unstructured, can be really unproductive. In my experience when there isn’t a clear solution being discussed or the conversation is going in circles, bring the focus back to the problem. When aligned on the problem, suggest taking the solutionising offline to a document or any other format.

This is not only more productive but also gives you an opportunity to do more research and have time to think about solutions rather than doing it on the spot.

Fear of being wrong:

During group discussions the loudest voices who aren’t afraid to be wrong can dominate. This can be dangerous if decisions are based on that discussion. How I get comfortable with sharing opinions even when not confident in them, is by caveating what I say with context if it needs to be validated offline. This can make the difference between coming across arrogant vs just contributing to the discussion. There are many ways this could be phrased but ultimately you want to communicate that you are sharing the information you have that’s relevant to the discussion but if a decision is being made based on that some validation should be done offline.

Being Data Driven

I’m not going to go into all the details here but a key aspect to efficiently driving conversations is the use of data. Without data discussions can go in circles theorising about how big a problem is or what impact a proposed solution might have on a problem. However, getting data to facilitate a discussion may not always be easy. Whenever available it’s worth using as having data to backup your comments makes for a much more convincing argument.

When presenting data it can be easy to over use it and distract from the primary topic. The process I always follow is:

Firstly identify what outcomes you want to achieve Identify who your audience is and what conversations you need to drive to achieve those outcomes Plan reporting around those conversations and audience If any data or chart is distracting from the primary conversation, move it to a different view

Key consideration with using data is to not try to build the same view for all audiences.

For some topics like configuration issues and vulnerability management, data is much easier to collect while for others like network traffic metadata, public historical breach patterns, risk likelihood and impact it might not be worth the time investment during the planning phase of a project to collect it. For these harder data sources, prioritising based on theory may be more effective but consider investing in reporting to track progress over the longer term.

Related to data, a common question I get asked is about trying to put monetary loss value to a risk. With security risk, there are usually so many variants of how the risk could materialise that any value is a wild guess. For example, the same risk might materialise weekly with insignificant impact and once every 5 years with catastrophic impact. How you deal with this depends on your company’s risk process but I wanted to point it out as a common challenge.

Working with Security Conscious Engineers

Based on stereotypes and past experience working in more siloed Security teams, it was somewhat surprising at the start to find engineers pointing out and fixing security issues without anyone else telling them. Obviously this is how it should be, as security is everyone’s responsibility, but as you’ll find out being embedded within an Engineering team, it’s not always that simple.

I got lucky, in that some of the senior engineers I worked with were at times, more security conscious than I was. This is a great learning opportunity and can make your life easier but certainly doesn’t come without its own set of challenges.

Working with these engineers over time, I realised that for these engineers where I can bring most value is by helping them rationalise the risk and how to prioritise the issues. For the engineers, as they know the systems so well it can be easy to go down a rabbit hole and identify really niche hard to exploit weaknesses. Security isn’t binary where it is either secure or not. If you want to deliver business value outside fixing issues, you need to have a risk appetite and balance that with the need to invest in non-security projects.

If you’ve never had to help teams balance security with everything else they are responsible for, this could be a challenge for you and take some time to understand how the business prioritisation works. Security is just one prioritisation factor that a team needs to consider. You will also need to take into account service availability, cost, growth opportunities, etc..

This also works in reverse though. When you are suggesting a security improvement, whether it be a process or configuration change, generally all engineers will really question the value of the change in the context of the team’s priorities. The upside is, if you can convince them, they can be incredibly committed and supportive to solving that problem.

When working in standalone Security teams, it’s easier to follow best practice and not really question what is behind that guidance. Having to debate with engineers on the true value of a change will really test your knowledge and you have to be willing to admit some best practice/idea you believed in might not add much value. Wise has a core value of “customer > team > ego” and it’s in these situations where you really need to put ego aside.

Building Bridges

An important part of being an embedded security engineer is building relationships/ bridges between the Security teams and your team. This is valuable so that you and the teams you work with build up context on other parts of the business. Context is key to facilitating effective cross team collaboration as it allows you to view a problem from someone else’s perspective.

NB: When working across multiple teams, it’s critical to avoid becoming the messenger between those teams. You need to try to improve collaboration without them becoming dependent on you.

Building Context and Trust

Embedded roles may lack the support network of peers with similar priorities and backgrounds. Therefore, maintaining close ties with the relevant Security teams is crucial. Depending on the project, this could involve collaborating with Security Operations, Application Security, Governance, or Compliance teams. If your background aligns with one of these domains, reaching out may feel less daunting vs being in these teams and having to reach out to unfamiliar business functions. The benefit of being embedded, especially if you’re not the most outgoing person, is that building those relationships with the teams/engineers outside your normal area of expertise comes naturally as you are part of their team. As you spend time with these team’s, you’ll build up knowledge on their priorities and challenges. This contextual knowledge is key to driving better collaboration.

Building context on a team can be done relatively quickly, trust on the other hand takes time. On building trust and improving collaboration I’d recommend the book “The Software Engineer’s Guidebook” by Gergely Orosz. It has a section on collaboration for Staff+ engineers which I found strongly overlaps with the skill required as embedded security engineer and would be a great resource for anyone in that role.

Facilitating Collaboration

When you have a good understanding of multiple teams and have built trust with their engineers, you are in the best position to bring those teams closer. This will happen most naturally when one side needs something from the other.

Use these opportunities to allow the teams to collaborate, rather than throwing requirements over the wall hoping they get done. At times, you can feel like a translator but it’s worth it in the short term to improve collaboration. As you’ll have context on both teams, you can help by identifying any miscommunications or conflicting priorities. Helping the team past these should allow the teams to build trust and respect for each other’s challenges so that in future your “translation” isn’t required.

An example of this which I’ve seen is when a Security team will make a request for a solution without much context on the current engineering challenges/priorities. The engineering team doesn’t have context on why the request is important and think the suggested solution doesn’t make sense technically for their system.

This can result in the Engineering teams disengaging, saying the Security team doesn’t know what they are talking about and Security teams saying the engineering teams don’t care about security.

As the person with context on both sides, you will need to help both sides step back from the solution, understand the problem and then collaborate on a solution.

This breakdown in collaboration isn’t specific to security but can happen with a request from any team to another team where the request isn’t communicated with empathy for the other team.

Closing Comment

This post is based on my experience across my career so far, however it should be applicable regardless if you’re an embedded security engineer or a security engineer at all. Building strong cross team collaboration skills is key to delivering value outside your team scope and delivering projects that have cross team dependencies. Also as a security engineer in a Security team everything above should be applicable in reverse where you have to build relationships with the other Engineering teams, doing this is how I delivered the most value in my previous company and what helped get my current role.

The next post will focus on lessons I learned or wish I learned getting stuck into the technical work when I started out as an embedded security engineer.