Using AWS Boto3 to Paginate EC2

What is ec2 pagination? 

AWS boto3 clients will only return 1000 entries max.   That means if you have 1002 ec2 instances and use boto3 client to describe those instances what you get back is a list of 1000 EC2s, not the full 1002.   This is the same for other clients as well, like the s3 client.

So how can you get the full list of resources from AWS?

The answer is to use pagination when describing the resources.    This article shows you 2 methods to get the full list.

Method 1 – Client with paginator.

1st method uses the boto3 client paginator.   This method relies on the client pulling 1000 entries at a time while your code must iterate over the returned items.   Here’s an example:

import boto3
# Setup the client
mysession = boto3.Session()
ec2client = mysession.client('ec2')

# Setup the paginator
paginator = ec2client.get_paginator('describe_instances')
page_iterator = paginator.paginate()

# Start iterating over the paginator
for page in page_iterator:
    for eachres in page['Reservations']:
        instance = eachres['Instances'][0]
        print(instance['InstanceId']

What you see happening is the paginator handles the requesting of the instances from AWS. Paginator will continue to pull/describe instances beyond the 1000 per request limit. Your code just iterates over the returned ‘page’, paginator handles the rest.

Now this method works, I’ve used it often. But here’s a method I just discovered that I like better.

Method 2 – EC2 Resouces

Yes, the EC2 resource can pull the full list. The ec2 resource’s describe instances method automatically handles pagination for us.

import boto3
# Setup the resource
mysession = boto3.Session()
ec2resource = mysession.resource('ec2')

# Get full list
instances = ec2resource.instances.all()

# Iterate over the list.Note the use of instance as a resource with attribute here.   
for eachinst in instances:
    print(instance.instance_id)

That’s it. I like Method 2 much better. For one, its much more readable. More importantly, I like the fact that you can use methods when getting the full list. The above example uses instances.all(). You could also use instances.filter() to limit the return. For example:

...
# Get partial list
instances = ec2resource.instances.filter(
    Filters=[
        {
            'Name': 'tag:role',
            'Values: [
                 'Production'
             ]
         }
    )
)
...

This example limits the returned list to EC2 instances with a certain tag and value, in this case, a tag called ‘role’ with value of ‘Production’. Very useful and something the the paginator in Method 1 can not offer.

That’s it for today. Happy coding.