AWS Windows password rotation with Custom Secret Manager

The Issue

When you create a Windows machine in AWS you can get the password some minutes after the machine creation using the pem key from the web console to retrieve it.

The problem is that this password will remain the same for the whole life of the machine. So also if a person leaves the company and loose AWS credentials if he has the password saved he will retain this kind of access. Of course, there are others way to prevent it like VPN firewall, nominal accounts etc. but it is not the purpose of this article.

The purpose of this article is to enable an automatic password rotation for the Administrator account in an AWS Windows EC2 Machine.

The AWS Proposed Solution

If you search in the AWS Blog for this topic you can find two articles:

1 — Password Rotation

The first one is Password Rotation for Windows on Amazon EC2 Made Easy with EC2Rescue and it uses the System Manager action called EC2Rescue that as it is possible to read from the official documentation is a very deep impact action, only to change a password here a summary of the actions:

• The system creates a temporary VPC
• The system launches a new EC2 temporary helper instance.
• The system stops your original instance and creates a backup. It then attaches the original root volume to the helper instance.
• The system uses Run Command to run EC2Rescue on the helper instance. EC2Rescue identifies and attempts to fix issues on the attached, original root volume. When finished, EC2Rescue reattaches the root volume back to the original instance.
• The system restarts your original instance and terminates the temporary instance. The system also terminates the temporary VPC and the Lambda functions created at the start of the automation.

All these actions clearly generate a downtime and are done only to change a password that at the end it is a command in PowerShell. In my opinion, this can be acceptable if you lose the password but not for a recurrent job of password rotation.

2 — Eliminate keypairs using Secret Manager

Another one with the title How to eliminate EC2 keypairs from password retrieval of provisioned Windows instances using Secrets Manager and CloudFormation , at the end the article is a little bit misleading, it says <<which will also rotate the password for you>> but this is not true in this contest. Potentially secret manager can rotate the password for you but it is not done with the automation proposed in the article. I have deployed the cloud formation template and the rotation is disabled in the secret manager record generated.

To have this rotation feature you need to write the password rotation function and it is not an easy one. So the only purpose of this article is to store the password in a secret manager record at instance creation time.

For these reasons, I decided to write my own solution.

My Solution with Custom Secret Manager

Prerequisites

• The ec2 machine is already created and needs to have a role with a policy ssm arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
• The SSM needs to work on the machines if it is not the password is not replaced and the old one with pem key remains, nothing brokes.
• To change a password it is not necessary to know the previous password because SSM runs as a daemon.

Infrastructure and steps

Using the terraform modules it will be created

• One lambda function for account and region
• For each machine a secret manager record, all the records call the same functions.

Creations steps:

• Create one lambda function for each region you are working on. Using this code

module "windows-password-lambda-rotation" {
  source  = "giuseppeborgese/windows-password-lambda-rotation/aws"
  prefix  = "pep"
}

• Before applying the rotation try to run a simple command like this to the machine, to see if ssm commands can run, check the output in the run command history

aws ssm send-command --instance-ids i-xxxxxxxxx --document-name AWS-RunPowerShellScript --parameters commands="dir c:"

• For each EC2 Windows machine create a new secret manager record and connected to the function using this code

module "windows-password-rotation-secret-manager2019" {
  source  = "giuseppeborgese/windows-password-rotation-secret-manager/aws"
  secret_name_prefix = "vault_"
  instanceid = "i-xxxxxx"
  rotation_lambda_arn = "${module.windows-password-lambda-rotation.lambda_arn}"
}

• You can rotate the password manually using the rotation button or wait the numbers of days defined

• You can still recover the old password from the web console but it will NOT work

Youtube video

I did a full creation and configuration in this video

AWS Windows password rotation with Custom Secret Manager

so you can see all the details and don’t miss anything. There is also a troubleshooting phase.

Rotation steps what happens behind the scene:

This image describes all the steps every time there is a rotation manual or automatic

Let's read in details:

1. The secret manager record triggers a lambda function passing its parameters.
2. The Lambda extract the instance id from the called record and generates a new password with the predefined criteria
3. The lambda stores the new password in the original Secret Manager record using the default key aws/secretsmanager
4. The Lambda stores encrypted the password in the system manager parameter store using the default key aws/ssm
5. The lambda runs a call to SSM run command this runs a PowerShell command in the EC2 machine.
6. The EC2 machine recovers the password from parameter store by Powershell
7. The lambda deletes the password from the parameter store.

You cannot pass the password as a parameter in the shell script because it will be shown in cleartext the System Manager log, it has to “travel” between services always encrypted.

Further investigation

Most of the work was to write a custom Lambda function if you are interested to know more about this topic you can read:

• Rotating AWS Secrets Manager Secrets for Other Databases or Services
• Understanding and Customizing Your Lambda Rotation Function

Feedback

If you like this article and the module leave a comment or a thumbs up/clap in the youtube video or here in the article.