[AWS](EN) Export DynamoDB table to S3 in Java

Export DynamoDB table to S3 in Java


Environment and Prerequisite

  • Java
  • Gradle
  • AWS(DynamoDB, S3, IAM)
  • AWS Java SDK(1.x or 2.x)
  • IntelliJ


Before Start

  • This post assumes reader knows about user creation, assigning role and policy to user, making user’s key(aws_access_key_id, aws_secret_access_key) in AWS IAM.
  • This post assumes reader knows about S3 bucket concept.
  • This post assumes that DynamoDB table already exists and PITR is enabled in that table. See also DynamoDB data export to Amazon S3: how it works.


Preparation

AWS Setting

User and Credential Setting

  • Let’s consider a user is already created in AWS IAM.
  • Consider credential of that user is already set in running environment. Related content is on Step 1: Set up for this tutorial.

(Option) In case of using AWS IAM role not AWS IAM user

  • When using an AWS IAM role, just like granting an IAM role to an EC2 instance and using it, you only need to use the ARN of the role and the method of applying the policy is the same as below
  • Attach user’s IAM policy to role and apply that role’s ARN in Target S3 Bucket Policy Setting.

User Permission Setting

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowDynamoDBExportAction",
            "Effect": "Allow",
            "Action": "dynamodb:ExportTableToPointInTime",
            "Resource": "arn:aws:dynamodb:us-east-1:111122223333:table/my-table"
        },
        {
            "Sid": "AllowWriteToDestinationBucket",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::your-bucket/*"
        }
    ]
}

Target S3 Bucket Policy Setting

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ExampleStatement",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/Dave"
            },
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::awsexamplebucket1/*"
        }
    ]
}

IntelliJ Setting

  • SDK is imported using build.gradle. Check each version in below codes.


Code

Java SDK 1.x

build.gradle

plugins {
    id 'java'
}

group 'org.example'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    implementation group: 'com.amazonaws', name: 'aws-java-sdk-dynamodb', version: '1.12.42'
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

ExportDynamoDbTableToS3.java

import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.amazonaws.services.dynamodbv2.model.ExportTableToPointInTimeRequest;

public class ExportDynamoDbTableToS3 {

    public static void main(String[] args) {

        // Please fill out below values
        String tableArn = "";
        String s3Bucket = "";
        String s3Prefix = "";
        String exportFormat = "";
        String s3BucketOwner = "";

        AmazonDynamoDB amazonDynamoDB = AmazonDynamoDBClientBuilder.defaultClient();
        ExportTableToPointInTimeRequest exportTableToPointInTimeRequest = new ExportTableToPointInTimeRequest();

        exportTableToPointInTimeRequest.setTableArn(tableArn);
        exportTableToPointInTimeRequest.setS3Bucket(s3Bucket);
        exportTableToPointInTimeRequest.setS3Prefix(s3Prefix);
        exportTableToPointInTimeRequest.setExportFormat(exportFormat);
        exportTableToPointInTimeRequest.setS3BucketOwner(s3BucketOwner);

        amazonDynamoDB.exportTableToPointInTime(exportTableToPointInTimeRequest);

    }
}

Java SDK 2.x

build.gradle

plugins {
    id 'java'
}

group 'org.example'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    implementation platform('software.amazon.awssdk:bom:2.17.14')
    implementation 'software.amazon.awssdk:dynamodb'
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

ExportDynamoDbTableToS3.java

import software.amazon.awssdk.services.dynamodb.model.ExportTableToPointInTimeRequest;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;

public class ExportDynamoDbTableToS3 {

    public static void main(String[] args) {

        // Please fill out below values
        String tableArn = "";
        String s3Bucket = "";
        String s3Prefix = "";
        String exportFormat = "";
        String s3BucketOwner = "";

        DynamoDbClient dynamoDbClient = DynamoDbClient.builder().build();
        ExportTableToPointInTimeRequest exportTableToPointInTimeRequest = ExportTableToPointInTimeRequest
                .builder()
                .tableArn(tableArn)
                .s3Bucket(s3Bucket)
                .s3Prefix(s3Prefix)
                .exportFormat(exportFormat)
                .s3BucketOwner(s3BucketOwner)
                .build();

        dynamoDbClient.exportTableToPointInTime(exportTableToPointInTimeRequest);
    }
}


Result

DynamoDB

  • I concealed AWS ID and bucket name.

S3


GitHub


Reference

Read More

[CI/CD] workflow_dispatch와 workflow_call을 사용하지 않을 때 기본값 설정하기

Published on September 16, 2024

[CI/CD](EN) Setting default value when not using workflow_dispatch and workflow_call

Published on September 16, 2024

[CI/CD] Github Actions Matrix 사용법과 예제

Published on September 01, 2024