In the first blog post of this series, I outlined some of my thoughts on collaboration and working with people. I chose that topic first as I believe it is one of the most fundamental skills to succeed in any role but especially as an Embedded Security Engineer. This blog post is going to focus on getting started with delivering work for your team and how to deal with pressures of a new role.

It is easy to get overwhelmed in security and feel like everything is a priority. This can lead to great engineers not delivering on their potential and getting lost juggling too many parallel streams of work. In this post I’m going to share some of my learnings on dealing with this, which I’m hoping can help someone make a start in an embedded role without becoming overwhelmed. I chose this topic second as getting a good start to a new role sets up the foundations which your future impact is built off.

Part 2 - Getting Started

1. Onboarding

As mentioned in the first blog post, gaining a deeper understanding of your team and their systems is a key benefit of this role. The best way to achieve this is by getting hands-on experience with their systems. While this approach may not be feasible for everyone or in every situation, in my opinion, it’s the most effective way to build the foundational technical knowledge necessary for future work.

Assuming you won’t be as experienced with the team’s primary technologies as the other engineers, a good starting point is to follow any onboarding exercises designed for more junior engineers. During this process, take notes on potential security improvements or gaps, but avoid making immediate fixes unless they are minor. I would recommend that you focus first on building your knowledge and observing how the systems function.

Personally I would prefer getting hands on, but if this isn’t an option, consider alternatives like training, shadowing an engineer, e.g. while they perform common tasks, or simply asking them to explain how they work day to day and how the team’s systems work. Early in my time in this role, I prioritised getting hands on (plus a mix of the other learning methods) but later I had to find a balance between delivering on security objectives and getting enough context through general engineering work.

2. Focusing on your Speciality

After some onboarding time, you will need to start focusing on delivering improvements in security. When it comes to knowing what to work on you generally have two options, take the initiative to proactively identify projects or wait for the direction to be outlined by your lead. When it comes to how I plan my work, I have a strong preference towards being proactive, however that doesn’t mean I work on whatever I want. When taking the initiative you need to know how to both communicate with your lead and team. The final piece of that puzzle is knowing how to effectively prioritise. Within the field of Security, I believe this is a common struggle for people as we are often choosing between equally important risks. Below I’ll share some guidance on prioritisation based on my experiences.

2.1 Prioritisation

There are generally two ways I view priorities. Firstly, you and your team’s initiatives relative to each other, secondly, those initiatives prioritised within the wider business context. The second point on the business context is more important as delivering business value is ultimately the goal but I mentioned it second as it takes time to build up the knowledge & support structure to do this. Assuming you have a list of initiatives to work on, the process of prioritising them against each other generally follows standard patterns in security and risk management, e.g. comparing likelihood and impact. As I mentioned in my first blog post, quantifying this can be extremely challenging but don’t be afraid to just go with your gut instinct. Not making a decision and trying to do both will likely do more harm than choosing the wrong priority. When following my instincts, at a minimum I try to describe my internal rationale for why I’m leaning one way or another. Even if this is vague it’s better than nothing, as you can challenge and iterate on it. When following instincts, you also need to be willing to admit when you are wrong and adapt.

Building up enough business context to know where your work fits in the wider business priorities takes time and isn’t always possible to do on your own. This is especially true when you are new in the role. Prioritising within the wider business context is where you should lean on your lead, team and product managers for input. They might know of some business context which will change up your priority order. Examples could be, cost saving directives, regulatory pressure, strategic initiatives which need support.

Even if your priorities are out of your hands, having an internal prioritisation model will help you understand the business context and refine your skills. Comparing your internal prioritisation vs final prioritisation will give you a feedback loop on your model and allow you to iteratively improve. Improving this over time will help you build trust and eventually gain more independence and provide value to your team’s planning.

Something not covered here is balancing long term strategic initiatives vs short term fixes which I’ll cover in a later post when talking about one-way door decisions. Tackling these strategic initiatives should be avoided when getting started which is why I’ve left it out.

2.2 Fire Fighting

I mentioned earlier how in security it can be easy to get overwhelmed by all the issues that need to be fixed. We are trained to see a threat in every misconfiguration and vulnerability but not all vulnerabilities are equal and not every threat can be fully mitigated. The following statement is important to remember when trying to prioritise what to fix:

You can’t solve every problem at once, and trying to do so will result in nothing getting fixed.

Just to be clear, if you find some obvious fires burning that will have a material impact on the company in the short term, you need to prioritise and escalate them. However, it’s important to remember you aren’t alone. There was a security team there before you (hopefully) and the company needs to deal with incidents when you are unavailable. Becoming a one-stop shop for all things security creates a single point of failure, which is in the business’s best interest to avoid.

As mentioned above, findings like an existing breach, publicly exposed data sets, exploitable public vulnerabilities, etc., are impossible to ignore and difficult to prioritise one vs the other. However, more systemic issues which can’t be directly exploited may need more pragmatic handling. Something I’ve always struggled to rationalise, is the tendency to treat a security issue as the highest priority when found even though it has been there for months or years. Sometimes a high priority is completely justified but other times creating a new high priority can distract from delivering existing impactful improvements. In this situation, there can be a fine balance between being pragmatic vs becoming jaded and not showing the appropriate level of urgency. When you are new to a company you are more likely to place excessive urgency on an issue, however this new urgency can be valuable. Bringing urgency to an existing issue adds a fresh perspective to people who may have been dealing with the same issue for too long. It’s important, though, when driving the level of urgency, to be open to hearing reasons why an issue can’t be fixed or why it’s a lower priority than you might think.

Every situation is different and needs to be evaluated on its own. However, when trying to figure out the urgency of some remediation action, try to think about the impact of a delay to the business:

What is the threat and worst case impact? What is the probability?

A common question you might get asked when dealing with urgent incidents is, can the fix wait until tomorrow or after the weekend? Being asked this question doesn’t mean you have to make the final call as you might not be comfortable/authorised to make it. What you can do is provide information and your perspective so that, if there are more senior people available, they can make an informed decision. When doing this, outline the facts that you know, what unknowns there are and what your informed choice would be, with a caveat of what risks that includes. This allows you to contribute and ensure an informed decision is made even if you don’t make it.

2.3 Using Past Experience

Unless this is your first role, everyone has worked in other companies and seen how problems can be solved in different ways. When you join a new company it’s easy just to compare your new and old company to see what gaps there are. However, the approach you take to using/sharing this information and experience is important to building trust. In my experience, someone saying we did it this way in my previous company may come off as someone who isn’t adaptable or didn’t fully understand the reasons behind choices. Every company has a different journey, working culture, risk appetite and depending on current business context varying priorities. It’s important to understand why historical decisions were made before challenging them.

Focus on the problems at hand and why they need to be solved in the context of your current company. When it comes to outlining potential solutions, this is where your experience becomes valuable. Based on the solutions you have seen elsewhere you can propose potential solutions. With these solutions, highlight any challenges you faced/observed with that choice that may not be obvious unless you’ve experience implementing it. The key point here is what you saw elsewhere should always only be a potential solution rather than the right solution by default.

3 Starting Small

Assuming there are no fires burning when you join, starting small is key to long-term success in this role. I mentioned in my first blog post the concept of building trust capital. To do this, you need to deliver impact and finish projects. In my experience, some of the most important and impactful work for improving a business’s security requires cultural or cross team process/technology changes. However, these large scale changes can take years to deliver and are the types of changes which require leveraging trust and goodwill which you need to build up.

Starting with small scoped changes will allow you to not only build trust but also get more of an understanding on how teams work and how to approach driving changes in the team and wider. After reflecting on my experience, below is some general advice which should help regardless of the starter projects you choose: (This advice is specific to when starting out in the role)

Ensure they can be delivered anywhere between a few weeks to max. 1 quarter. When just starting and if possible, avoid projects which need extensive cross-team collaboration. This can help you to deliver early on in your journey before you have extensive connections and business knowledge. Avoid sharing high level principles like “least privilege” as remediation advice. Any competent engineer could google security best practice and find general principles. To deliver value as an SME, you need to translate those principles to actionable insights specific to the system you are working with. To counterbalance the previous point on being too vague, try to avoid being overly prescriptive on the solution. Some solutions are obvious and don’t need to be over thought, e.g if it’s just changing a best practice setting in an existing system. However, when a problem could be solved in multiple ways, focus on making the problem well understood. Before you try to teach other engineers to do something on their own, e.g. system hardening, do it yourself first to experience any pain/ challenges they may face. This will allow you to mentor your peers more effectively as you can have empathy and they know you have shared experience of their challenges.

A final point on this topic is to focus on your team’s most critical system and work down the list. At some point either when you get to non-critical systems where the return on your time investment is diminishing or when you feel you’ve built up enough trust capital to take on bigger problems, you should start looking into any longer term strategic changes you could make.

Given this blog is specific to an Embedded Security Engineer, I want to give a few more specific suggestions of where to start. These suggestions are based on my experience embedded with Platform teams so treat them as a source of inspiration:

Review system configuration/hardening Perform access reviews and secret handling practices Improving existing security tooling/processes

The following three subsections are only relevant if you want more specific insights on my suggestions for these project topics, otherwise you can skip them.

3.1 Configuration Reviews

Whatever technology the team maintains, there will likely be some best practice guidance on how to harden it. This can be done through tooling, following an available guide or where there is nothing available by applying your experience and general best practice to the system.

Configuration reviews/system hardening is a good starting point as an embedded engineer as it has a finite scope and should be something you can either lead or contribute to remediation. This builds off what I mentioned in the onboarding section about being hands on but delivers even more value for your time investment as you are delivering improvements to security which is what you are there for.

For the scope of hardening work to be small, your team would need to primarily own the system you want to review. Doing any work on a system owned by multiple teams will increase the delivery time massively. I’m not going to go into detail on this but simply put, more people, especially cross-team, means more admin work and more people to convince before a change is allowed.

Something to remember with following standards or best practice guides: If you are asking other people to follow your recommendation, it will build much more trust if you understand the intent behind a requirement vs just using “the standard says” as the motivation for your request. This takes more time but it also means when you come across a system which doesn’t have a guide you’ll know what to look out for.

3.2 Access Reviews

A lot of the same advice for scope of configuration reviews would be applicable when performing access reviews, but remember to keep this somewhat simple and just try to reduce risk without trying to revolutionise the tooling or ways of working. The goal when starting out is to build experience while delivering impact.

When embedded or working with Platform/Infrastructure teams, the blast radius and impact from insider threats is massively increased. Therefore, for these teams with privileged access to critical systems, quick wins can be found by:

Reducing permissions that aren’t used. Reducing the attack surface, e.g., removing people with privileged access that don’t need it. Adding preventative/monitoring controls on the worst-case abuses of that privileged access.

Something that is easy to overlook when trying to reduce the risk from privileged access is the secrets management for the system you are reviewing. There is no value in reducing access to a system via one authentication path if the same engineers who lost some access can use break-glass methods or privileged service accounts to authenticate to the system with the same or more privileges as before.

The topic of privileged access management could be a whole blog post on its own but below are a few pointers to help when working on this:

You have to balance engineer productivity with security - unproductive engineers impose a high cost to the business too. Making your team’s day to day harder without their buy-in will make you lose trust and the goodwill capital you’ve built up. Before removing permissions, work with the team on what tooling/automation is needed to allow them to still do their job efficiently. If you don’t think about usability and productivity, engineers will find another way to do their job and bypass your controls (shadow IT).

3.3 Improving Tooling & Processes

Any Engineering team is likely to have existing security related processes and there will also likely be tools/processes owned by the Security team which they have to follow. This makes improving internal team tooling and processes a great starting project. The objective here is not to introduce something new but to identify frustrations with an existing tool/process and iterate on it to make them more effective and less toilsome.

Some more specific improvement examples:

Improving reporting Identifying unnecessary steps in a process and removing them. Building guardrails into CI/CD pipelines.

With processes managed by other teams, this is also a great opportunity to start building up your network and, in my experience, if you can make a team’s life easier by improving something that is outside their control it will build a lot of trust.

A common example of this is vulnerability management (tooling and processes). As you are embedded you should be able to view and experience the tooling from the perspective of the engineers vs just from the Security team. Using this and your security knowledge can allow you to suggest/make improvements to the process which would drive better engagement and enable teams to do the right thing vs frustrating them.

4 Definition of Done - Lessons Learned

In an embedded role, part of the difference from my perspective was the emphasis on delivering value for the team vs just delivering recommendations. The two aren’t mutually exclusive for example, delivering a threat model which shows where the system can be improved delivers value if the team on its own doesn’t have the time or expertise to deliver it. However, adding even more value would be delivering the threat model and then contributing on some of the remediation work.

When setting out on a project, two lessons I learnt is to one, be explicit in defining the scope and what done looks like, and two, define how you will measure the difference the project has made. That measurement doesn’t have to be a fancy dashboard that is populated automatically. It can be really basic like a manual measurement, for example a count of the number of people with admin rights. These two steps are critical to know when you have done enough in one specific project which allows you to move on without keeping too many parallel projects in progress.

With experience I learnt that there is another step after delivering remediation work to mitigate an issue. That step is being able to have confidence that your remediation work is still effective in 6 months, a year and that the same issue hasn’t popped up somewhere else in the company’s infrastructure. If you can deliver this on top of remediation work you will maximise your impact. Having reporting or another process to show the issue hasn’t come back adds value in a few ways:

Allows you to focus on new issues without having to fix the same issues twice or constantly check if what you did is still working. Allows you to demonstrate if audited that a control is in place and working. Allows you to quickly detect if a mitigation is reverted.

While this last step is important and should be the long term objective with any critical controls, it shouldn’t stop you from just delivering the remediation work. If you only have resources to deliver the remediation but nothing more, it’s still reducing the risk to the business.

5 Conclusion

This blog post is a reflection of my experience starting out in this role and in other roles. The most important points to remember are don’t try to fix everything at once, start small and focus on measurable impact and don’t jump into trying to change architecture or culture straight away, even if you know this is needed eventually.

My next blog post will be from a slightly different perspective and focus on looking at when you should consider hiring embedded engineers and what to consider when doing that.