[AWS](EN) Export DynamoDB table to S3 in Java
Export DynamoDB table to S3 in Java
Environment and Prerequisite
- Java
- Gradle
- AWS(DynamoDB, S3, IAM)
- AWS Java SDK(1.x or 2.x)
- IntelliJ
Before Start
- This post assumes reader knows about user creation, assigning role and policy to user, making user’s key(aws_access_key_id, aws_secret_access_key) in AWS IAM.
- This post assumes reader knows about S3 bucket concept.
- This post assumes that DynamoDB table already exists and PITR is enabled in that table. See also DynamoDB data export to Amazon S3: how it works.
Preparation
AWS Setting
User and Credential Setting
- Let’s consider a user is already created in AWS IAM.
- Consider credential of that user is already set in running environment. Related content is on Step 1: Set up for this tutorial.
(Option) In case of using AWS IAM role not AWS IAM user
- When using an AWS IAM role, just like granting an IAM role to an EC2 instance and using it, you only need to use the ARN of the role and the method of applying the policy is the same as below
- Attach user’s IAM policy to role and apply that role’s ARN in Target S3 Bucket Policy Setting.
User Permission Setting
- AWS User which is created in AWS IAM needs below permissions.
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataExport.Requesting.html#DataExport.Requesting.Permissions
- Of course it cannot be used its own. Below things like ARN, Region, AWS Account ID should be modified depends on running environment.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowDynamoDBExportAction",
"Effect": "Allow",
"Action": "dynamodb:ExportTableToPointInTime",
"Resource": "arn:aws:dynamodb:us-east-1:111122223333:table/my-table"
},
{
"Sid": "AllowWriteToDestinationBucket",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::your-bucket/*"
}
]
}
Target S3 Bucket Policy Setting
- Target S3 bucket needs below bucket policy setting.
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataExport.Requesting.html#DataExport.Requesting.Permissions
- Of course it cannot be used its own. Below things like user, ARN, Region, AWS Account ID should be modified depends on running environment.
- If it is accessed between cross AWS account, then ACL setting is needed. Check Resource-based Access Control List (ACL) and IAM policies.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ExampleStatement",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/Dave"
},
"Action": [
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::awsexamplebucket1/*"
}
]
}
IntelliJ Setting
- SDK is imported using build.gradle. Check each version in below codes.
Code
Java SDK 1.x
build.gradle
plugins {
id 'java'
}
group 'org.example'
version '1.0-SNAPSHOT'
repositories {
mavenCentral()
}
dependencies {
implementation group: 'com.amazonaws', name: 'aws-java-sdk-dynamodb', version: '1.12.42'
testCompile group: 'junit', name: 'junit', version: '4.12'
}
ExportDynamoDbTableToS3.java
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.amazonaws.services.dynamodbv2.model.ExportTableToPointInTimeRequest;
public class ExportDynamoDbTableToS3 {
public static void main(String[] args) {
// Please fill out below values
String tableArn = "";
String s3Bucket = "";
String s3Prefix = "";
String exportFormat = "";
String s3BucketOwner = "";
AmazonDynamoDB amazonDynamoDB = AmazonDynamoDBClientBuilder.defaultClient();
ExportTableToPointInTimeRequest exportTableToPointInTimeRequest = new ExportTableToPointInTimeRequest();
exportTableToPointInTimeRequest.setTableArn(tableArn);
exportTableToPointInTimeRequest.setS3Bucket(s3Bucket);
exportTableToPointInTimeRequest.setS3Prefix(s3Prefix);
exportTableToPointInTimeRequest.setExportFormat(exportFormat);
exportTableToPointInTimeRequest.setS3BucketOwner(s3BucketOwner);
amazonDynamoDB.exportTableToPointInTime(exportTableToPointInTimeRequest);
}
}
Java SDK 2.x
build.gradle
plugins {
id 'java'
}
group 'org.example'
version '1.0-SNAPSHOT'
repositories {
mavenCentral()
}
dependencies {
implementation platform('software.amazon.awssdk:bom:2.17.14')
implementation 'software.amazon.awssdk:dynamodb'
testCompile group: 'junit', name: 'junit', version: '4.12'
}
ExportDynamoDbTableToS3.java
import software.amazon.awssdk.services.dynamodb.model.ExportTableToPointInTimeRequest;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
public class ExportDynamoDbTableToS3 {
public static void main(String[] args) {
// Please fill out below values
String tableArn = "";
String s3Bucket = "";
String s3Prefix = "";
String exportFormat = "";
String s3BucketOwner = "";
DynamoDbClient dynamoDbClient = DynamoDbClient.builder().build();
ExportTableToPointInTimeRequest exportTableToPointInTimeRequest = ExportTableToPointInTimeRequest
.builder()
.tableArn(tableArn)
.s3Bucket(s3Bucket)
.s3Prefix(s3Prefix)
.exportFormat(exportFormat)
.s3BucketOwner(s3BucketOwner)
.build();
dynamoDbClient.exportTableToPointInTime(exportTableToPointInTimeRequest);
}
}
Result
DynamoDB
- I concealed AWS ID and bucket name.
S3
GitHub
Reference
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataExport.html
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataExport.HowItWorks.html
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataExport.Requesting.html
- https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html#get-started-setup
- https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/
- Java 1.x Interface AmazonDynamoDB
- Java 1.x Class ExportTableToPointInTimeRequest
- Java 1.x AmazonDynamoDB.exportTableToPointInTime
- Java 2.x Interface DynamoDbClient
- Java 2.x Class ExportTableToPointInTimeRequest
- Java 2.x DynamoDbClient.exportTableToPointInTime