Post

AWS S3서버 연동하기

Posted
By So Hee Park 12 min read

✅ AWS S3

Simple Storeage Servce, 주로 파일 서버로 사용
⭐️ Scalability: S3는 트래픽이 증가함에 따라 서버 인프라, 용량 변경을 대신 처리해 줌
⭐️ Durability: 여러 영역에 데이터 복사본을 저장해 한 영역이 다운되어도 데이터 복구 가능

  • Bucket: 다수의 객체를 관리하는 컨테이너, 파일시스템
    • entities stored in Amazon S3
    • data + metadata
    • identified by unique key
  • Object: 파일과 파일 정보로 구성된 저장단위
    • Containers for storing objects

⭐️ S3 권한 설정 IAM

IAM
Identity and Access Management

  • IAM policies can control access to S3 resources

  • IAM:
    • 사용자를 생성하고 사용자의 버킷 권한 액세스 관리
    • 사용자 단위로 접근 제어할 때
  • ACL: 액세스 제어 목록
    • 개별 오브젝트(객체)를 액세스 가능하게 만듬
    • 객체 단위로 접근 제어할 때
    • 버킷 정책만큼 세분화된 액세스 모드를 제공하지는 않음
  • Bucket Policy:
    • 단일 버킷 내 모든 객체에 대한 권한 세부적으로 구성
    • 버킷 단위로 접근 제어 확정할 때
    • JSON을 통해 세부 권한 설정가능
    • 버킷에 대해서만 권한 설정 가능하나, ACL은 버킷 뿐만 아니라 개별 객체에도 가능

✅ S3 버킷 생성

AWS console ▶️ S3 ▶️ 버킷 ▶️ 버킷 만들기 ⭐️ EC2랑 리전 동일하게 설정할 것!

✅ IAM 사용자 생성

AWS console ▶️ IAM ▶️ 액세스 관리 ▶️ 사용자 ▶️ 사용자 추가

✅ 액세스 키, 시크릿 키 생성

AWS Console ▶️ IAM ▶️ 엑세스 관리자 ▶️ 사용자 ▶️ 생성한 사용자 이름 클릭 ▶️ 보안 자격 증명 ▶️ 엑세스 키 만들기
이 때 .csv파일로 저장해 둘 것

💡참고

AWS 설정 https://gaeggu.tistory.com/33

✅ build.gradle

1
2

    // amazon s3
    implementation 'org.springframework.cloud:spring-cloud-starter-aws:2.2.6.RELEASE'

✅ application.yaml

cloud.aws.stack.auto=false
EC2에서 Spring Cloud 프로젝트를 실행시키면 기본적으로 CloudFormation 구성을 시작
그러나 우리는 CloudFormation이 없으므로 해당 기능을 사용하지 않도록 false

region.static
우리는 한국에 있으므로 ap-northeast-2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

server: port:8080

spring:
  mvc:
    pathmatch:
      matching-strategy: ant_path_matcher

  datasource:
    username: ${DATABASE_USERNAME}
    password: ${DATABASE_PASSWORD}
    driver-class-name: org.mariadb.jdbc.Driver
    url:

  jpa:
    show-sql: true

cloud:
  aws:
    s3:
      credentials:
        access-key: ${ACCESS_KEY}
        secret-key: ${SECRET_KEY}
      bucket-name: ${BUCKET_NAME}
      region.static: ap-northeast-2
      stack.auto: false

jwtpassword:
  source: ${JWT_SECRET_KEY}

  logging:
    level: debug

✅ S3Config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

package com.github.drug_store_be.config.security;

import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class S3Config {

    @Value("${cloud.aws.s3.region.static}")
    private String region;

    @Value("${cloud.aws.s3.credentials.access-key}")
    private String accessKey;

    @Value("${cloud.aws.s3.credentials.secret-key}")
    private String secretKey;

    @Bean
    public AmazonS3Client amazonS3Client(){
        return (AmazonS3Client) AmazonS3ClientBuilder.standard()
                .withCredentials(
                        new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey))
                )
                .withRegion(region)
                .build();
    }
}

✅ 사진 업로드만 간단ver.

☑️ FileUploadController

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

package com.github.drug_store_be.web.controller;

import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.github.drug_store_be.web.DTO.ResponseDto;
import lombok.RequiredArgsConstructor;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import java.io.IOException;

@RestController
@RequestMapping("/upload")
@RequiredArgsConstructor
public class FileUploadController {
    private final AmazonS3Client amazonS3Client;

    @Value("${cloud.aws.s3.bucket-name}")
    private String bucket;

    @PostMapping("/pics")
    public ResponseDto uploadFile(@RequestParam("file") MultipartFile file){
        try{
            String fileName= file.getOriginalFilename();
            String fileUrl= "https://"+ bucket+ "/test"+ fileName;
            ObjectMetadata metadata= new ObjectMetadata();
            metadata.setContentType(file.getContentType());
            metadata.setContentLength(file.getSize());

            amazonS3Client.putObject(bucket, fileName, file.getInputStream(), metadata); //s3에 저장
            return new ResponseDto(HttpStatus.OK.value(), "file upload success", fileUrl);
        } catch (IOException e) {
            e.printStackTrace();
            return new ResponseDto(HttpStatus.INTERNAL_SERVER_ERROR.value(), "upload fail");
        }
    }
}

☑️ 포스트맨 성공

multipart 데이터를 전송해야 하므로 Body 유형을 form-data으로 선택하고, KEY의 속성을 File을 선택 정상적으로 보내지면 200응답이 올 것이고, AWS S3 콘솔에서 파일 들어온 것 확인 가능

image

Screenshot 2024-05-30 at 00 21 14

✅ 여러개 사진 업로드, 삭제, 수정

1️⃣ 여러개 사진 업로드

☑️ StorageController

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

package com.github.drug_store_be.web.controller;

import com.github.drug_store_be.service.service.StorageService;
import com.github.drug_store_be.web.DTO.ResponseDto;
import com.github.drug_store_be.web.DTO.awsS3.FileDto;
import com.github.drug_store_be.web.DTO.awsS3.SaveFileType;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;

import java.util.List;
import java.util.Optional;

@RestController
@RequestMapping("/storage")
@RequiredArgsConstructor
public class StorageController {
    private final StorageService storageService;

    //여러개 업로드
    @PostMapping("/multipart-files")
    public ResponseDto uploadMultipleFiles(@RequestPart("uploadFiles")List<MultipartFile> multipartFiles,
                                           @RequestParam(required= false) Optional<SaveFileType> type
    ){
        List<FileDto> response= storageService.fileUploadAndGetUrl(multipartFiles, type.orElseGet(()-> SaveFileType.small));
        return new ResponseDto(response);
    }
}

☑️ SaveFileType Enum

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

package com.github.drug_store_be.web.DTO.awsS3;

public enum SaveFileType {
    small("일반파일"),
    large("대용량파일");
    private final String kor;

    SaveFileType(String kor) {
        this.kor = kor;
    }
    public String getTypeKor(){
        return this.kor;
    }
}

☑️ FileDto

1
2
3
4
5
6
7
8
9
10
11
12

package com.github.drug_store_be.web.DTO.awsS3;

import lombok.AllArgsConstructor;
import lombok.Getter;

@Getter
@AllArgsConstructor
public class FileDto {
    private String fileName;
    private String fileUrl;
}

☑️ ResponseDto

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

package com.github.drug_store_be.web.DTO;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Getter;
import lombok.Setter;
import org.springframework.http.HttpStatus;

@Setter
@Getter
public class ResponseDto {
    private int code;
    private String message;
    @JsonInclude(JsonInclude.Include.NON_NULL)
    private Object data;

    public ResponseDto() {
        this.code = HttpStatus.OK.value();
        this.message = HttpStatus.OK.name();
    }

    public ResponseDto(int code, String message) {
        this.code =code;
        this.message = message;
    }

    public ResponseDto(Object data) {
        this.code = HttpStatus.OK.value();
        this.message = HttpStatus.OK.name();
        this.data = data;
    }

    public ResponseDto(int code, String message, Object data) {
        this.code =code;
        this.message = message;
        this.data= data;
    }

}

☑️ StorageService

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

package com.github.drug_store_be.service.service;

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.AmazonS3Exception;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.github.drug_store_be.service.exceptions.StorageUpdateFailedException;
import com.github.drug_store_be.web.DTO.awsS3.FileDto;
import com.github.drug_store_be.web.DTO.awsS3.SaveFileType;
import lombok.RequiredArgsConstructor;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.UUID;

@Service
@RequiredArgsConstructor
public class StorageService {

    @Value("${cloud.aws.s3.bucket-name}")
    private String bucketName;

    private final AmazonS3 amazonS3Client;

    public List<FileDto> fileUploadAndGetUrl(List<MultipartFile> multipartFiles, SaveFileType type) {
        List<FileDto> response= new ArrayList<>();

        switch(type){
            case small:
                for(MultipartFile file: multipartFiles){
                    PutObjectRequest putObjectRequest= makePutObjectRequest(file);
                    amazonS3Client.putObject(putObjectRequest);
                    String url= amazonS3Client.getUrl(bucketName, putObjectRequest.getKey()).toString();
                    response.add(new FileDto(file.getOriginalFilename(), url));
                }
                break;
            case large:
                break;
        }
        return response;
    }

    private PutObjectRequest makePutObjectRequest(MultipartFile file) {
        String storageFileName= makeStorageFileName(Objects.requireNonNull(file.getOriginalFilename()));
        ObjectMetadata objectMetadata= new ObjectMetadata();
        objectMetadata.setContentType(file.getContentType());
        objectMetadata.setContentLength(file.getSize());
        try{
            return new PutObjectRequest(bucketName, storageFileName, file.getInputStream(), objectMetadata);
        } catch (IOException e) {
            throw new StorageUpdateFailedException("File Upload Failed", file.getOriginalFilename());
        }
    }

    private String makeStorageFileName(String orignialFileName) {
        String extension= orignialFileName.substring(orignialFileName.lastIndexOf(".")+1);
        return UUID.randomUUID() + "." + extension; //이미지 저장할 때 고유번호
    }
}

☑️ StorageUpdateFailedException

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

package com.github.drug_store_be.service.exceptions;

import lombok.Getter;

@Getter
public class StorageUpdateFailedException extends RuntimeException{
    private final String detailMessage;
    private final String request;

    public StorageUpdateFailedException(String detailMessage, String request) {
        this.detailMessage = detailMessage;
        this.request = request;
    }
}

☑️ ExceptionControllerAdvice

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

@RestControllerAdvice
@Slf4j
public class ExceptionControllerAdvice {
    @ResponseStatus(HttpStatus.NOT_FOUND)
    @ExceptionHandler(NotFoundException.class)
    public ResponseEntity<ResponseDto> handleNotFoundException(NotFoundException nfe){
        log.error("Client 요청이후 DB 검색 중 에러로 다음처럼 출력합니다. " + nfe.getMessage());
        ResponseDto responseDto = new ResponseDto(HttpStatus.NOT_FOUND.value(), nfe.getMessage());
        return new ResponseEntity<>(responseDto, HttpStatus.NOT_FOUND);
    }

    @ResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR)
    @ExceptionHandler(StorageUpdateFailedException.class)
    public ResponseEntity<ResponseDto> handleFileUploadFailedException(StorageUpdateFailedException sufe){
        ResponseDto responseDto = new ResponseDto(HttpStatus.INTERNAL_SERVER_ERROR.value(), sufe.getMessage());
        return new ResponseEntity<>(responseDto, HttpStatus.INTERNAL_SERVER_ERROR);
    }

}

image

image

image

image

2️⃣ 사진 삭제

☑️ StorageController

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

@RestController
@RequestMapping("/storage")
@RequiredArgsConstructor
public class StorageController {
    private final StorageService storageService;

    //여러개 업로드
    @PostMapping("/multipart-files")
    public ResponseDto uploadMultipleFiles(@RequestPart("uploadFiles")List<MultipartFile> multipartFiles,
                                           @RequestParam(required= false) Optional<SaveFileType> type
    ){
        List<FileDto> response= storageService.fileUploadAndGetUrl(multipartFiles, type.orElseGet(()-> SaveFileType.small));
        return new ResponseDto(response);
    }

    //업로드 취소(삭제)
    @DeleteMapping("/multipart-files")
    public ResponseDto deleteMultipleFiles(@RequestParam(value= "file-url") List<String> fileUrls){
        storageService.uploadCancel(fileUrls);
        return new ResponseDto();
    }
}

☑️ StorageService

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

@Service
@RequiredArgsConstructor
public class StorageService {
    @Value("${cloud.aws.s3.bucket-name}")
    private String bucketName;
    private final AmazonS3 amazonS3Client;
    public List<FileDto> fileUploadAndGetUrl(List<MultipartFile> multipartFiles, SaveFileType type) {
        List<FileDto> response= new ArrayList<>();

        switch(type){
            case small:
                for(MultipartFile file: multipartFiles){
                    PutObjectRequest putObjectRequest= makePutObjectRequest(file);
                    amazonS3Client.putObject(putObjectRequest);
                    String url= amazonS3Client.getUrl(bucketName, putObjectRequest.getKey()).toString();
                    response.add(new FileDto(file.getOriginalFilename(), url));
                }
                break;
            case large:
                break;
        }
        return response;
    }

    private PutObjectRequest makePutObjectRequest(MultipartFile file) {
        String storageFileName= makeStorageFileName(Objects.requireNonNull(file.getOriginalFilename()));
        ObjectMetadata objectMetadata= new ObjectMetadata();
        objectMetadata.setContentType(file.getContentType());
        objectMetadata.setContentLength(file.getSize());
        try{
            return new PutObjectRequest(bucketName, storageFileName, file.getInputStream(), objectMetadata);
        } catch (IOException e) {
            throw new StorageUpdateFailedException("File Upload Failed", file.getOriginalFilename());
        }
    }

    private String makeStorageFileName(String orignialFileName) {
        String extension= orignialFileName.substring(orignialFileName.lastIndexOf(".")+1);
        return UUID.randomUUID() + "." + extension;
    }


    public void uploadCancel(List<String> fileUrls) {
        try{
            for(String url: fileUrls){
                String[] parts= url.split("/");
                String key= parts[parts.length-1];
                amazonS3Client.deleteObject(bucketName, key);
            }
        }catch(AmazonS3Exception e){
            e.printStackTrace();
            throw new StorageUpdateFailedException("File Delete Failed "+ e.getMessage(), fileUrls.toString());
        }
    }
}

image

3️⃣ 사진 수정

☑️ StorageController

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

@RestController
@RequestMapping("/storage")
@RequiredArgsConstructor
public class StorageController {
    private final StorageService storageService;

    //여러개 업로드
    @PostMapping("/multipart-files")
    public ResponseDto uploadMultipleFiles(@RequestPart("uploadFiles")List<MultipartFile> multipartFiles,
                                           @RequestParam(required= false) Optional<SaveFileType> type
    ){
        List<FileDto> response= storageService.fileUploadAndGetUrl(multipartFiles, type.orElseGet(()-> SaveFileType.small));
        return new ResponseDto(response);
    }

    //업로드 취소(삭제)
    @DeleteMapping("/multipart-files")
    public ResponseDto deleteMultipleFiles(@RequestParam(value= "file-url") List<String> fileUrls){
        storageService.uploadCancel(fileUrls);
        return new ResponseDto();
    }

    @PutMapping("/multipart-files")
    public ResponseDto modifyMultipleFiles(@RequestParam(value="file-url") List<String> deleteFileUrls,
                                           @RequestPart("uploadFiles") List<MultipartFile> multipartFiles,
                                           @RequestParam(required = false) Optional<SaveFileType> type){
        List<FileDto> response= storageService.fileUploadAndGetUrl(multipartFiles, type.orElseGet(()-> SaveFileType.small));
        storageService.uploadCancel(deleteFileUrls);
        return new ResponseDto(response);
    }
}

✅ AWS S3 CORS에러 해결

AWS S3 bucket에 CORS(Cross-origin 리소스 공유) 규칙을 추가한다.
물론 자신의 SecurityConfig 파일에 적합한 코드를 넣어야 할 것이다. 

1
2
3
4
5
6
7
8
9

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST", "DELETE", "HEAD"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": ["Authorization", "Authorization-refresh", "Token"],
    "MaxAgeSeconds": 3600
  }
]

Screenshot 2024-05-31 at 00 35 40

🔴 Trouble Shooting

🔴 URL을 붙여넣었을 때 사진이 뜨지 않고 이런 에러가 뜨는 문제

Screenshot 2024-05-29 at 18 53 39

🔵 문제 원인

객체가 퍼블릭으로 설정되지 않아서 생기는 문제이다. 객체가 퍼블릭이 아니기 때문에 외부에서 접근할 수가 없다.

🟢 Solution

외부에서 해당 버킷에 접근 가능하도록 버킷 정책을 수정한다. https://inpa.tistory.com/entry/AWS-%F0%9F%93%9A-S3-%EB%B2%84%ED%82%B7-%EC%83%9D%EC%84%B1-%EC%82%AC%EC%9A%A9%EB%B2%95-%EC%8B%A4%EC%A0%84-%EA%B5%AC%EC%B6%95

https://awspolicygen.s3.amazonaws.com/policygen.html

Screenshot 2024-05-31 at 00 25 33

💡 참고

https://gaeggu.tistory.com/33

https://github.com/JBA-jeju-basketball-association/JBA-BE/blob/feat-S3bucket/src/main/java/github/com/jbabe/web/controller/StorageController.java

https://inpa.tistory.com/entry/AWS-%F0%9F%93%9A-S3-%EB%B2%84%ED%82%B7-%EC%83%9D%EC%84%B1-%EC%82%AC%EC%9A%A9%EB%B2%95-%EC%8B%A4%EC%A0%84-%EA%B5%AC%EC%B6%95

Deploy, AWS
deploy
This post is licensed under CC BY 4.0 by the author.