Recently, we have all seen many reports of AWS buckets that contained private data that were misconfigured and fully open to the public. Whether the result is slight embarrassment or a serious exfiltration of data, the risk of a misconfigured bucket can be mitigated fairly easily. Amazon S3 buckets can… Continue reading
What is ec2 pagination? AWS boto3 clients will only return 1000 entries max. That means if you have 1002 ec2 instances and use boto3 client to describe those instances what you get back is a list of 1000 EC2s, not the full 1002. This is the same for other clients… Continue reading
The Amazon Command Line (AWSCLI) is the tool from AWS that lets you interact with the AWS API. When using the AWSCLI you must pass it credentials like key/secret key or you can setup awscli config so that AWS can cache a single key/secret key to use with the commands. … Continue reading
So it happened. One of our techs ‘accidentally’ deleted all the directories and files in one of our S3 buckets. I enabled S3 Bucket Versioning on all our important buckets. So the deleted files are still there with the ‘latest version’ of the file being a Delete Marker. If you… Continue reading
In this example I want to open a file directly from an S3 bucket without having to download the file from S3 to the local file system. This is a way to stream the body of a file into a python variable, also known as a ‘Lazy Read’. import boto3… Continue reading
Recently I was asked to scour multiple AWS accounts to find any users or host role that had the S3FullAccess policy applied. So I came up with the following that will go through all users and roles to identify the ones with the S3FullAccess policy assigned. You can use this… Continue reading
The South Florida Amateur Astronomers Association is currently working on a project to coordinate multiple Radio Telescope collection points. This would require very precise time stamps on recorded data, much more accurate than what public NTP server pools could deliver. So we successfully built as close to… Continue reading
I was recently asked to create a report showing the total files within the top level folders and all the subdirs under the folder in our S3 Buckets. S3 bucket ‘files’ are objects that will return a key that contains the path where the object is stored within the bucket…. Continue reading
Here are 2 sample functions to illustrate how you can get information about Tags on instances using Boto3 in AWS. import boto3 def get_instance_name(fid): # When given an instance ID as str e.g. ‘i-1234567’, return the instance ‘Name’ from the name tag. ec2 = boto3.resource(‘ec2’) ec2instance = ec2.Instance(fid) instancename =… Continue reading
I’m writing this on 9/14/2016. I make note of the date because the request to get the size of an S3 Bucket may seem a very important bit of information but AWS does not have an easy method with which to collect that info. I fully expect them to add… Continue reading
Mike's Guides to Learning Boto3 Volume 1: Amazon AWS Connectivity and Basic VPC Networking
Mike's Guides to Learning Boto3 Volume 2: AWS S3 Storage: Buckets, Files, Management, and Security
Or Feel free to donate some beer money through paypal.
You have my Thanks.