This is post 8 of 9 in a multi-part series (hosted here) discussing the advantages, pitfalls, deployment methodology, and management of a multi-cloud account architecture. For this series, we are focusing strictly on using AWS as the Cloud Service Provider (CSP) but the concepts discussed port well to any provider or even to on-premise operations.

All aboard!

This journey cannot begin until you have everyone on the same page. By this point in the process, the gears should have really started to turn. Your organization can see how a multi-account architecture can really benefit everyone. We’re talking from customer to developer, operations and security, finance as well as stakeholders.

However, ideas can start to run rampant. Egos can go unchecked. Having a solid foundation with extreme flexibility can lead to a lot of carelessness. Now is more important than ever to make sure everyone is following procedure in order to properly support and maintain this new large-scale architecture.

From my experience, working with enterprise networks and leading teams that manage large-scale architecture, success can be distilled down to these 4 components:

1. Baseline
2. Documentation
3. Collaboration
4. Automation

Out of the box, this all just looks like work. A lot of work. Work that no one wants to take on. It will take time, but this is the under-the-hood work that needs to be done to make your architecture run well for the long haul.

Baseline

Having a defined starting point is critical. There could be many baselines within a fine-tuned architecture. A baseline in this context is really a defined set of instructions, values, conditions, and resources that will be used to deploy a new cloud account.

An organization could maintain a different baseline for the deployment of different types of accounts. For example, a different set of instructions for a department-specific account verses an application account. The baseline for what a development account looks like in comparison to a production account should definitely be different.

The key here with a baseline is they are fixed. Not that they cannot change and evolve over time, but once you’ve initiated the deployment of an account or process, the current baseline version will be followed through until completion. Organizations will evolve, grow, and prune their baselines over time. They are not set in stone.

Benefits of baselines

A well-defined baseline helps everyone within the team or organization know and understand the definite starting point for a project or resource. Baselines also allow developers and operators to set expectations on what is or should be present in every account allowing for automation and scalability. This four-letter acronym, RECD, helps describe the benefits.

Repeatability

When creating a baseline, deploying the exact same thing every time inherently becomes the norm. This allows solutions that are developed for a single account to scale to many. With the concept of repeatability, you have a cookie-cutter deployment model where operations know exactly how to deploy or maintain an account so developers can find or use whatever they need in the exact same place no matter what account they are in.

Evolution

Because architecture baselines should be deployed with coded templates, your organization will have a documented history of how the environment has evolved over time. This is important to help the organization understand where they may have gained or lost efficiencies and be able to adjust accordingly in the next iterations. If a key component to your baseline is removed because it may be viewed as no longer necessary, you will have a historical record to reference if that component ever needs to be reintroduced.

Compliance

It’s hard to imagine a space any more where there isn’t some level of regulation dictating what or how you should be doing something. Even if it is only from your internal security team, baselines allow operations to better provide compliance information as well as maintain a more secure architecture. Critical security configurations should be near-identical across all accounts as a result of being deployed with a baseline. As stated earlier, having confidence that something is configured in a specific way in a specific location is paramount to being able to gather compliance information as well as make bulk modifications.

Destruction

Knowing that the steps to recreate something identically if need significantly reduces re-work. Having a baseline means you can (in most cases) completely tear-down something that seems to be causing issues. Maybe someone made a configuration change out-of-band. Being able to recreate and environment the exact way it’s supposed to be in short order increases the ability to simply cut-bait when something isn’t working as it should.

It’s cyclical

There are many benefits to having baselines with regards to your environment. We really can’t come up with a scenario where not having a baseline would be a net positive. Even baselining a development sandbox enables cost reduction and more secure testing practices.

Creating baselines, however, does come at a cost. Documentation is critical to maintaining robust baselines.

Documentation

It should come as no surprise that in order to support and maintain a large-scale multi-account cloud architecture requires documentation. But it might not be the type of documentation you are thinking. Granted the skill level, experience, and tenure of your operators and developers will at times determine how detailed your documentation needs to be.

Because of the vast majority of your architecture and deployment of that architecture should be in coded templates, there is no need to create seemingly endless numbers of detailed SOPs containing hundreds of steps and dozens of screenshots. Your documentation for this type of architecture should be very basic and enable autonomy. The templates and scripts should be well commented on. The logical and agreed-upon naming convention goes a long way. Lastly, less documentation of higher caliber enables more reuse. If a significant amount of context is required to understand a process, your repeatability is reduced greatly.

Naming convention & standards

Determining upfront what the labels, names, and variables for resources and processes within your environment should look like is critical to maintenance success. Having a system for naming ‘things’ is not only important for information gathering but it is a necessity in order to make mass modifications at scale.

Have documentation that lays out what names of resources should look like. Agree on what Snake case, Camel case, Kebab, and many others should look like for each category. Use different standards for different resource types.

Avoid redundancy. There are some (very few) cases where (especially in AWS) you need to include the type of the resource in the name of the resource. This is especially true when the people caring the most about the resource works mainly in the console. The is no need to name your Development VPC – DevVPC.

Layout the key labels required. What are the minimum pieces of information a resource should have to tell an admin, developer, or operator what the resource if for, what level of protection it needs, and how long it should be allowed to live?

Procedural

Mapping out the steps to deploy a solution template does not need to include every input for every field. In fact, it shouldn’t include any. Your steps should be type-o proof. We’ll discuss in a different series about automation. Anyone present in your environment after 6 months of spinup time should be able to glean the information required to conduct any process with the vaguest of steps.

Any process should have the minimum amount of detail required to successfully conduct. Only add clarification and additional details where required AFTER a step in the process has failed or been misinterpreted by more than one individual. This is an indication that more clarity may be required. However, do not add this information to the master procedure. When possible, abstract additional detailed steps to a separate document and reference that document from the primary instructions.

The key to solid documentation for maintaining a robust baseline and deploying secure and manageable architecture is simplicity. Maintaining up-to-date documentation can be very difficult. The information may be continually changing. Get as close to the process without introducing confusion and misinformation. No information is much better than incorrect information. Especially with regards to deploying critical infrastructure.

Location of information

Documentation that describes mindset, mentality, and the logic behind an organization’s process is more important in most cases than the information itself. Create resources that enable autonomy and decision making at the lowest level. When there are already agreed-upon naming conventions and procedural understandings within the organization, support and maintenance move faster.

Share instructions on how to use the organizational standards to get the information that is required. Documentation for the sake of having a process documented is worthless if no one is going to follow the steps. Document only the situations where manual intervention is required and could most likely cause a discrepancy. This would be ‘custom settings’, one-off configurations, or dependencies. The bulk of your documentation should represent where and how to find the required information when it is needed – not demonstrate the precise steps required. Parameterize and use variables in your documentation as much as possible to scale with your architecture.

Collaboration

Communicate between teams the standards, conventions, and procedures frequently. At a reasonable frequency for your organization, get together and discuss what is/not working for all stakeholders.

Any opportunity to reduce the back and forth between developers, administrations and operators will significantly impact the supportability and maintainability for the positive. If teams become annoyed or upset about something, it is important to understand why and agree on corrective action before groups begin drifting from the standard or coming up with their own processes.

Collaboration enables team cohesion, builds comradery, and further reduces the requirement for over detailed documentation. The idea is to build a culture that supports large-scale architecture. Not one that sabotages it.

Automation

About as obvious as documentation is automation. If manual intervention is a requirement for many steps in an architecture deployment, one could imagine at some point in the scaling process, systems begin to break down.

Automate anything that you can that happens more than once. Automation is a form of documentation. The appropriate process for automating something should include commented code and the ability to logically follow a process in case that process were to ever break down.

Automation does so much more than shave seconds or minutes off the front end of a process. As we stated earlier, it is important to enable autonomy. Automation does this. The more cogs you have in a process, the more potential for things to go wrong or steps to get missed.

Even a simple process, that only takes a minute or two, conducted weekly could have a profound impact on the amount of rework it could cause if the process is completed incorrectly and deploys or configures a resource incorrectly. Remember, the naming convention is critical for the ability to make a mass modification at scale. If the individual responsible for the process does so incorrectly or misses the process entirely due to a busy schedule or absence, considerable harm to your architecture could result.

Automation doesn’t completely eliminate human error; however, it can significantly reduce it.

Tools, 3rd-party solutions, & systems

As a bonus or honorable mention, it is important to state that these four components for successfully supporting and maintaining MMA have some sort of tool, service, or system behind them.

How you conduct your automation and where you store your documentation is considerably less important than actually having these components as a solid part of your foundation. However, it would be impossible to support and manage this type of architecture without an array of different tools. This series and this specific post is not the place to share the many options available.

Each organization will also be very different than the next with regards to their preferred tools and systems for deployment. Budget, technical competency, time resources, etc. all factor into what should be considered a viable solution for each architecture.

Check back to Tactful Cloud frequently for updates and information on the tools and solutions that are available to meet your support and maintenance requirements.

Series parts are as follows:

1. 1. Series Introduction
   2. My experience with Multi-Account Architecture
   3. What does the end-state look like
   4. Reasons you might consider multi-account architecture
   5. When it is right to implement
   6. What the potential architecture might look like
   7. How do you get there
   8. How do you support and maintain the architecture
   9. Lessons learned

This is post 6 of 9 in a multi-part series (hosted here) discussing the advantages, pitfalls, deployment methodology, and management of a multi-cloud account architecture. For this series, we are focusing strictly on using AWS as the Cloud Service Provider (CSP) but the concepts discussed port well to any provider or even to on-premise operations.

Choose your own adventure

There’s really no ‘one-size-fits-all’ solution when it comes to deploying accounts within your organizational structure. In our previous post about ‘end-state’, we discuss very briefly a few different types of accounts you’d want to include in your architecture and why they might be beneficial. We also shared this diagram:

In this diagram, we show at the top a pretty standard foundation for managing a cloud architecture of any size that should support nearly any scale.

The AWS Organizations structure works by implementing inheritance through the use of containers called Organizational Units. These containers can be nested and used to organize different accounts based on an infinite configuration scheme that best suits your needs. Once the management structure is determined, the organization of sub-accounts can take place.

At the top

In our ‘End-State’ post, we already discussed the key foundational accounts recommended for creating a robust MAA. The number of those types of accounts and for what service they provide should be determined by organizational needs and available resources. There can be a fine line between logical separation of services and total chaos.

At the very least you have your AWS Organizations account that is used to spawn subsequent accounts. In a later series or individual post, we will share how this process takes place. It is in this account you enable the sharing of other AWS Services across all accounts linked in the Organization. This is an important component.

Alongside your Organization account, an architecture should at the very least have a single account for security and enterprise services as well as an account for shared resources.

However ya wanna slice it

Once you’ve established the Root Organizations account and any other management accounts, you can begin placing the accounts into Organizational Units (OU) or ‘functionality buckets.’

These buckets allow you to apply the maximum level of permissions as well as available AWS Services allowed to be used in each account. Controlling the permissions allowed in an account is done through the use of Service Control Policies (SCPs). Too much to cover here; however, these policies allow you to completely disable services in all accounts within the OU they are placed.

One especially beneficial use-case for SCPs is implementing available services within AWS GovCloud Regions. For example, when working with the DOD and their Cloud Computing Security Requirements, you will learn that different AWS services are allowed for different information types or ‘Impact Levels’. However, there a not different types of AWS GovCloud account to accommodate the different impact levels. All services regardless of their accreditation for use are enabled in the GovCloud Regions. You can use OUs and SCP to disable the use of services that have not been vetted for use within a specific impact level. This will prevent even administrators from inadvertently leveraging a service within an environment that it has not been accredited for.

Below is an example of an Organizational Hierarchy for such a situation. Within our testing organization, we have 6 AWS accounts: Orgs/Master, Dev, Prod, Security, Management, and Release.

Prod and Release are in a higher impact level OU, while Dev, is in the lower to enable R&D. The Orchestration Accounts can be excluded from either to enable the use of shared services.

Be logical

The following images are taken directly from a presentation we’ve given on creating and managing via code a multi-account architecture. You can use them to spark ideas on the best way to structure your account hierarchy.

It is important to note that the shaded and unshaded boxes do not need to denote an actual Account or an OU. Depending on the situation, they could both be OUs where the lowest level OU contents all the accounts for that category or service.

Project

A sub OU structure can be created to organize all projects into their own Organizational Units where the first OU is the for the whole project, and all accounts can fall under that single OU, or additional sub-OUs can be containers for like account types.

Billing

Each initial level OU in your structure could be for all accounts that are related to the same cost-center or source of funding. Many organizations have different ‘colors of money’ where lumping like accounts into the same OU structure could enable the disabling of services. For now, you must still use tags to allocate the cost of all services across all accounts in a structure like this, but expect AWS to enhance AWS Organizations to accommodate more granular billing views in the future.

Isolation

If you’ve properly configured your networking and other application-wide services within shared services accounts, it might be logical to completely break out your OU structure to tier specific roles. Not only can you segregate database admins from the developers and their services, but you can also completely disable services that have nothing to do with hosting data from consuming cost in the respective accounts.

Failover & High-availability

This is a concept that we have not personally tested but it seems like, in the right circumstance, it could prove to be very beneficial. There are many different ways to implement blue/green or virtually zero down-time deployments in the cloud. One solution could be to dedicate full accounts to releases of your application or service.

This solution provides many benefits. Depending on your release cycle and how long a release is live, you could have unique billing information for every release. This would give you the ability to more easily see if new changes to your infrastructure, scaling, and/or code reduced the cost of resources or increased.

This solution also more easily gives you the ability to give weighted access to the new release for a small percentage of customers and increase over time. Since nothing at all should have changed in a completely separate account with regards to the development and deployment of new code, you can very easily revert back to the old release if something starts going wrong.

You get to decide

As you can see, there is virtually an infinite number of accounts and OU configurations with regard to the hierarchy of your multi-account architecture. Each business case could serve different needs.

The benefit of this flexibility increases the manageability and scalability of whatever solution you are working to deploy. However, you logically decide to layout your accounts doesn’t have to be concrete either. This can evolve over time. With the proper configuration of shared services, it may even be possible to introduce accounts or services from entirely different organizational structures. But that’s enough to cover for one post.

Series parts are as follows:

1. 1. Series Introduction
   2. My experience with Multi-Account Architecture
   3. What does the end-state look like
   4. Reasons you might consider multi-account architecture
   5. When it is right to implement
   6. What the potential architecture might look like
   7. How do you get there
   8. How do you support and maintain the architecture
   9. Lessons learned

This is post 4 of 9 in a multi-part series (hosted here) discussing the advantages, pitfalls, deployment methodology, and management of a multi-cloud account architecture. For this series, we are focusing strictly on using AWS as the Cloud Service Provider (CSP) but the concepts discussed port well to any provider or even to on-premise operations.

This all seems crazy

So why even consider all the effort to implement a multi-account architecture in the first place?

In short… Because AWS said so.

Initially, when I heard this and learned about the release of AWS Organizations in 2018 and the subsequent release of AWS Control Tower in 2019, I just kept thinking that this is all just a way for AWS to make more money. Waaayy more money.

While it may be true that AWS does benefit financially from selling this mindset, in just the relatively short amount of time I’ve been doing enterprise cloud architecture I’ve come to learn that all of these new services are released for the benefit of the customer. What isn’t seen on the backend is all the planning and road-mapping for the ability to use some of the amazing features of the cloud, that in order to do so requires multiple accounts.

Leveraging a multi-account configuration has many benefits. Some of which we don’t even realize currently. As customers request the ability to do something else within their environments, new features will be released that lean heavily into a well-configured multi-account architecture.

What’s the strategy

There are many different reasons why a small business or large organization would want to and should implement a multi-account architecture similar to what I shared on the previous post. Here are just a few reasons but there could be many more based on your organization. One business case I won’t discuss in detail is acquisition potential or sell-off.

Landing Zones / Control Tower

As I stated right away, AWS says to do it. It’s really that simple. If you want to be able to be nimble and leverage all the great new features that AWS releases weekly, you really need to stay up on the best practices they put out. Those are put out for a reason and that reason is they know the direction they are taking all of their new services. It’s best to keep up.

Not only does segmenting your resources into different accounts provide the benefits of the points I will speak of next, but it also makes your life much easier to adapt to the rapid rate of change that being in the cloud exposes you to. Implementing new and beneficial features doesn’t have to always include a massive overhaul. You can use a phased approach.

Shared Services

The ability to more easily (and securely) share AWS services requires the AWS Organizational structure. When you start to leverage shared services, you have the ability to reuse work already done. You can even broker services to other departments within your organization without having to reinvent the process every time.

Shared services allow for better management by the responsible parties by centralizing access to a single account. This prevents the ability for someone to inadvertently cause much more harm than intended while bouncing around in the Console or using the Command-Line.

Security

It should be obvious by now that using multiple accounts significantly increases your security posture. As stated in the previous post, creating a security account allows for auditing best practices and preserving log integrity.

Having multiple accounts enables granular access restrictions. It helps prevent accidental access to protected resources. Developers should not be able to make changes or see instances that control enterprise services. Sure, IAM permissions can be configured to prevent accidental termination for instance, but why even bother having Domain Controllers intermingled with development instances if you don’t have to.

Billing

Early on in the cloud, tagging was really important if you wanted to separate out the cost for different functions or services. Tagging itself has evolved greatly over time but was never the most reliable way of breaking out the cost for reimbursement. If you wanted to know your total storage cost for your production application, you still had to somehow tag buckets used for the application to distinguish between storage consumed as related to an application vs. storage utilized operationally.

Utilizing different accounts within an organizational structure not only gives you the opportunity to have a better understanding of what your production application storage costs are, but you can further implement tagging within the account specific to the production release of your application to provide more context.

If you have multiple projects or divisions within your organization sharing compute or other resources within a single account, it becomes a mess very quickly if from a budgeting standpoint your IT department needs to recoupe operating costs from the HR and Accounting departments. If each department had its own account, this happens inherently further allowing each department to more easily tag their own resources if they wanted to provide more granular detail on their cloud expenses. Again, sharing resources and VPCs makes configuring department-specific accounts a breeze.

Isolation

Separating resources based off of organizational responsibility for billing purposes is one thing, but how about making sure the Sales team resources aren’t bumping into or gaining access to the Marketing team resources. One of the biggest caveats to accessing cloud resources is it is very easy to elevate your own access or find ways to access resources you should never have had access to. Understanding and managing IAM permissions (Policies, Roles, and Profiles) at a large scale is difficult. Everywhere you look in a policy you could find methods that provide some way privilege escalation.

Breaking resources out into their own accounts not only created added safeguards from and access standpoint, but it also creates a barrier to the creation of resources that should never have been launched and more importantly creates a barrier for accidentally running destructive operations across all account resources of a specific type. You’ve significantly reduced your blast radius for destructive operations. You may lose an entire departments resources at once but should never take down the whole organization.

Recovery

Last but not least in this our list of reasons why multi-account architecture is a good idea is recovery. Related to isolation, the recovery component means that when something goes awry in your organization, you should very easily be able to diagnose what went wrong. May the tech gods have mercy on the soul of someone who negligently terminated every EC2 instance in an account but at least it was only the one account. With other protections and proper configuration in place, that should never be able to happen. But what could happen is the bulk misconfiguration of some Security Groups, or the modification of an Instance Profile, that instead of affecting instances across all departments sharing the same account, only affected a single department and the rest of the teams may not even notice an issue.

If there is something askew with network access for all departments regarding your cloud architecture, there are only really two possible causes:

1. Your on-prem configuration to the cloud has been misconfigured and completely unrelated to your configuration and virtually unpreventable.
2. Someone that had access to your shared services account for Networking made a misconfiguration. Human error can happen.

If you notice that all resources within the Sale and Marketing teams seem to be working just fine and the only team having a problem is Accounting, then you have a good basis to believe you need to start investigating changes to the environment made in their account or to their specific systems. You should be able to immediately narrow down who had the access to make a particular service disruption change, in a specific segment of your architecture, and most importantly when.

If all your resources are shared in a single or only a few shared accounts, anyone at any time in the recent history could have made a change with no expectation it would have had the fallout that it did.

Not so crazy after all

You can see here with just these six reasons that the effort to get a well-architected multi-account configuration could save you boatloads of time in the future — in troubleshooting and outages alone.

All of these reasons relate primarily back to one key theme – security. AWS has many resources that can help you in the implementation, the structure, and the strategy for multiple account configuration. I encourage you to review the AWS Multi-Account Security Strategy to learn more and of course, continue reading on in this series.

Series parts are as follows:

1. 1. Series Introduction
   2. My experience with Multi-Account Architecture
   3. What does the end-state look like
   4. Reasons you might consider multi-account architecture
   5. When it is right to implement
   6. What the potential architecture might look like
   7. How do you get there
   8. How do you support and maintain the architecture
   9. Lessons learned

This is post 2 of 9 in a multi-part series (hosted here) discussing the advantages, pitfalls, deployment methodology, and management of a multi-cloud account architecture. For this series, we are focusing strictly on using AWS as the Cloud Service Provider (CSP) but the concepts discussed port well to any provider or even to on-premise operations.

Background

The context provided in the previous post is not hypothetical. It is the reality I faced within about three months of starting enterprise cloud work for a new organization. Prior to this transition in mid-2016, I had some experience with cloud. Since late 2011 I had been exploring Microsoft Azure App Services, Databases, Storage Accounts, and virtual machines. But nothing at enterprise scale. From 2012 to 2014 I had deployed some web and mobile applications using the Azure App Services Stack, built up a very small user base, and began exploring Amazon’s suite of cloud services to do the same thing. This was more for curiosity sake than it was for job requirements. At the time I was being paid to be a Windows Systems Administrator. Anything I did in the cloud was for my own career advancement and self-interest.

In early 2016 I was ‘presented’ an opportunity to explore using AWS S3, Storage Gateway, and Virtual Tape Library for enterprise cloud backup. I say ‘presented’ because I really sought the opportunity myself regardless of who wants to take credit. Granted, had I not been afforded the opportunity, my rapid transition into cloud architecture would not have happened at the pace it did. I would have still gotten here, just maybe not as quickly and with the knowledge I plan to share throughout the rest of this series.

After exploring and deploying enterprise cloud backup for a bit, I was pulled completely into the cloud program. That is where shortly after the team was broken into two. An operations team and a design team. I was asked to lead the operations team. Not because of my experience in the cloud yet but more so because I was the most ‘senior level’ as well as most outspoken or opinionated.

At this time the discussion was to create as few AWS accounts as possible. These accounts were to provide managed services to allow the enterprise to leverage the cloud in a very scalable, yet manageable fashion. This was due to manpower experience as well as the ability to pay for more people. How could we manage even a half-dozen accounts with so few support personnel?

The current configuration

Before I go any further with the transition to as few accounts as possible to literally hundreds, I need to briefly explain the current architecture configuration.

The goal was to give individual development teams the freedom to create cool and innovative things in the cloud. All the while segmenting cost and not bumping into other development teams working completely different and unrelated projects. There was a primary account for managing Cloud Active Directory, Web Application Firewall, Ingress/Egress, and a few other enterprise security tools. The rest of the architecture was dedicated to letting anyone with a new application or idea to leverage the cloud. The team that I was a part of was charged with keeping strict baselines, securing all the accounts, and supporting the developers by enabling ‘approved’ AWS services. The organization’s initiative as a whole became ‘cloud-first.’ Anything new, any request for new hardware first had to answer the question – Why not go cloud?

‘Cloud’ and “Cloud-First’ meant something different to nearly everyone but when I arrived on the program, there was about a dozen AWS accounts that were all pretty much managed to the best of anyone’s ability at this stage in the game. Lots of drift and nuance from account to account. Nothing was standardized even though team members were under the impression it was. Every ‘production’ application account at the time needed to first have an existing ‘development’ account already deployed and an application accredited to be ‘production’ prior to getting an account to make the application live. In a very short time, any consideration of deploying another AWS account was of high concern. Keeping all the accounts in line and near baseline was virtually impossible.

Furthermore, as we explored how to better manage and scale through the use of shared or managed services, we realized there was no great way to provide the level of service required while being able to break out usage and charges in a productive manner. The ideal that was first 80% managed service accounts and 20% custom developer accounts soon shifted to the inverse. When the team laid out to management the facts and caveats to managing fewer accounts versus more, the reality became clear. The paradigm shifted. We needed to support as many AWS accounts as possible. This is when I was asked, “what do we need to do to support hundreds of AWS accounts while maintaining minimal support staff (a.k.a. my team)?”

Supporting the masses

At this time we still had just over a dozen AWS accounts and beginning to understand the transition to managing many more. Deploying a new AWS account, ‘baselining it’, and providing access to developers took a minimum of two weeks. Since development teams had access limited only to the services we allowed and we prevented any changes to networking configuration for security reasons, changes to any account at the time took anywhere from 1 to 3 weeks. This was due to the fact that we first had to call a CAB, discuss the change, and the responsible parties had to review and implications of the change before manually making that change in the respective accounts.

It’s early 2017 and because of the environment we were in, some of the great services AWS released were not available to us quite yet. Take AWS Organizations for example. A vast majority of the automation we had in place was a hodgepodge of scripts and Jenkins jobs that in some cases cause more issues than they solved. It was a mess. But again, everything was done in good faith and using the best tools and know-how we had at our disposal at the time.

A new mindset

After nearly a year of limping along, supporting a slow growth of AWS accounts, discussing our issues, prioritizing (re)work of processes, and (re)factoring of automation, we decided to take a 3-month break from deploying new accounts to build out our new systems and plan for the future. Throughout the year I researched other organizations, AWS Talks, and Webinars, spoke with team members about what was working and what wasn’t, and I re-evaluated our change management process. I ultimately prioritized with the team what we believed needed to be done in order to support a seemingly infinite amount of AWS accounts with basically a team of no more than 8 support personnel.

During our break from deployment, we built new systems and automation as well as leveraged some AWS services that had recently become available to us. As we reopened the doors and began to deploy new accounts, we quickly realized our new processes had introduced a new problem. We were now able to scale very quickly. Support was still going to be an issue at some point but now we were having Private IP Space issues and running out quickly. Around the same time, we hit a wall with IP addressing, we also hit an undocumented limitation of AWS Transit VPC. Any newly advertised routes into our cloud environment would bring down the network. All the work we had done to deploy cloud accounts quickly and stick them to a baseline was now stuck in its tracks. We solved ourselves into a new problem, even if it was a good problem to have.

AWS to the rescue

It’s now the end of 2018 and time for re:Invent again. Regardless of all the team was able to accomplish as a result of limping along in 2017, the release of AWS Organizations, and the implementation of our new processes and automation resulting from our strategic pause at the beginning of 2018, we were stuck. We manufactured a new problem. Until AWS released Shared VPCs and RAM (or Resource Access Manager).

This opened up many opportunities for the program to meet the needs of any developers looking to work on new projects in the cloud. This new feature released by AWS enabled the team that I was a part of to begin scaling to the number of AWS accounts we were asked to support just 18 months prior.

Sooo… Mission accomplished?

Not quite. Shortly after our implementation of AWS Shared Resources and another refactor of our account deployment process, we found ourselves needing to address many other growing pains. Things like the retirement or upgrade of existing services to cloud-native services as well as the implementation of newly approved services, after evaluating if the meant many different use-case requirements.

Up to this point, the growth trajectory of the program has been insane. So much so that I personally believe organizations of any size would find it impressive. Here are some quick numbers to demonstrate the history of the program.

• Beginning of 2017
  • Active Accounts: Approximately 12
  • Deployment Time: Greater than 2 weeks
  • Permission Roles/Policies: 1
  • Change Review Time: 2-3 weeks
• Beginning of 2018
  • Active Accounts: Approximately 30
  • Deployment Time: Less than 1 week
  • Permission Roles/Policies: About 20
  • Change Review Time: 1-2 weeks
• Beginning of 2019
  • Active Accounts: Greater than 65
  • Deployment Time: 30 minutes
  • Permission Roles/Policies: 3
  • Change Review Time: Virtually in real-time

While I was ‘involved’ with the amazing success of this growth and progress, the team was instrumental in the implementation. I did very little from a technical implementation standpoint. I merely did what I could as a lead to keep things on track, focused, and prevent perceived ‘fires’ from derailing our progress.

In August of 2019, I left this program to pursue other opportunities but I didn’t take leaving lightly. I learned a ton from being part of this team and attribute the following posts in this series to the experience I gained working with the team and that organization.

The following posts in this series do not share any ‘secret sauce’. There isn’t any. The concepts used are well documented in the AWS Multi-Account Security Strategy and the best practices are part of the AWS Well-Architected Framework. Presented here is a unique twist on the lessons learned and the hurdles jumped to be successful in implementing a secure, baselined, and well-documented multi-account cloud architecture.

Series parts are as follows:

1. 1. Series Introduction
   2. My experience with Multi-Account Architecture
   3. What does the end-state look like
   4. Reasons you might consider multi-account architecture
   5. When it is right to implement
   6. What the potential architecture might look like
   7. How do you get there
   8. How do you support and maintain the architecture
   9. Lessons learned