Using AWS VPC Flow Logs to Diagnose Intermittent RDS Connections

In practice, staying efficient as a company means that as your product evolves, sometimes parts are no longer needed and can be shut down. Doing this safely can be challenging.

One common problem area is databases, because they often serve as the hub for many use cases. Even after you’ve taken the main workload off a database, there may still be intermittent users that aren’t obvious. This is especially true in fast growing startup environments, where constant change can make it challenging to track down jobs which run infrequently.

In this post, I wanted to share a technique for determining if an AWS RDS instance is truly unused, and if not, tell you what’s connecting to it.

Introducing AWS VPC Flow Logs

In Mid 2015, Amazon released VPC Flow Logs, which are a way for you to monitor network traffic, even between instances such as load balancers and RDS managed databases that you can’t log into directly. In this post, we’ll use a Flow Log to determine what’s connecting to a database.

Enabling VPC Flow Logs

If your organization hasn’t enabled VPC Flow Logs, you’ll need to enable them and give them some time to collect data. Setup is straight forward, you can use these instructions.

How long should you collect data for? It takes at least 15 minutes to get data at all, but since timed jobs might be using your database, you’ll probably want to collect multiple days (or even weeks) of data before you can authoritatively say that no one is using an instance.

Determining your RDS Instance’s Elastic Network ID (ENI)

To even find the appropriate Flow Log, you’ll need to start with an ENI. For RDS instances, AWS does not provide this association explicitly, so you’ll have to dig for it using these steps:

1. Use the AWS RDS Console to obtain the instance’s endpoint (this will be in the form [xxx].[region].rds.amazonaws.com).
2. Use dig or something similar to lookup the IP address of the endpoint. A CNAME is not sufficient, you’ll need the actual IP.
3. Use the AWS Network Interface Console to search for the IP Address you got, and obtain the Network Interface ID (this will be eni-XXXXXXXX) from the resulting record.

Querying Your Flow Logs

1. Use the VPC Console to locate your VPC. Go to the Flow Log tab, and click through to the CloudWatch Logs Group.
2. In the CloudWatch Logs Group, use the search box to find your Network Interface ID.
3. Click into the record you’ve found, and you’ll see all the traffic for your instance!