TWpower Reading time ~1 minute

Case study of incident response


Environment and Prerequisite

  • Cloud Service Provider(AWS, GCP, Azure)
  • API


Background

  • I wrote this post because there was an issue that API of Cloud Service Provider used in running service did not function properly.
  • I used word ‘CSP’ to refer to the Cloud Service Provider in one of AWS, GCP, Azure to avoid pick one specific provider.


Timeline

  • 5/20 08:30: Receive Slack Issue
  • 5/20: Investigate reason of issue
  • 5/21: Open support case after find cause
  • 5/22: Issue resolved in CSP
  • 5/23: Issue resolved


Issue and Cause

  • Some of exported history were not returned when using CSP’s API


Impact

  • An incident occurred in daily data collection which leads to subsequent failures in the connected pipeline and services.


Act History

  • Manually create fin file which represents raw data collection is done after issue occured
  • Open a support case due to a CSP issue.


Resolve

  • Issue resolved after CSP’s API issue is resolved


Retrospective

  • I thought there will be no issue in CSP or other third party software. However I changed my mind that CSP also can have issues.
  • I studied incident response of Kakao issue which Kakao’s services were down due to a data center fire. The relevant YouTube video link is referenced in below.


Reference

TWpower Reading time ~1 minute

GKE Ingress에서 504 상태 코드에 대한 Timeout 수정 사례


환경

  • GKE


배경

  • E2E 테스트를 실행하던 중에 API 요청에 대한 코드는 실행되었지만 상태 코드가 504로 반환된 상황


해결 방법 및 과정

504 상태 코드의 경우 “Gateway Timeout”으로 서버가 게이트웨이 혹은 프록시 역할을 하는 동안 업스트림 서버로부터 정해진 시간안에 응답을 받지 못한 경우를 의미한다.

조금 더 쉽게 정리하면 요청을 받고 뒤에 있는 서버에 넘겼는데 설정된 Timeout 시간이 넘어가서 504 상태 코드를 반환한걸 의미한다.

아래는 MDN Web Docs에 있는 설명이다.

The HyperText Transfer Protocol (HTTP) 504 Gateway Timeout server error response code indicates that the server, while acting as a gateway or proxy, did not get a response in time from the upstream server that it needed in order to complete the request.

사용했던 API의 Endpoint는 GKE Ingress라서 Timeout이 걸려있는 부분이 있나 살펴봤다.

GCP 콘솔의 Ingress에서 아래와 같은 방식으로 타고 갈 수 있었다.

ingress => load balancer => backend services

“Backend services”에서 우리가 요청했던 API를 받는 곳이 있었고 해당 부분에 Timeout이 설정되어 있었다.

해당 Timeout을 변경해서 적용하니 Timeout이 길어졌으며 504 상태 코드가 아닌 202 상태 코드를 받을 수 있었다.


참고자료

TWpower Reading time ~1 minute

Case of modifying timeout for 504 status code in GKE Ingress


Environment and Prerequisite

  • GKE


Background

  • During E2E test, the code for API requests was executed but the status code returned 504


Solution and Steps

504 status code means that server like gateway or proxy did not get a response in time from upstream server. Normally called with “Gateway Timeout”.

Simply say, requests were forwarded to a server behind but returns 504 status code because of exceeding timeout period.

Below is explanation from MDN Web Docs.

The HyperText Transfer Protocol (HTTP) 504 Gateway Timeout server error response code indicates that the server, while acting as a gateway or proxy, did not get a response in time from the upstream server that it needed in order to complete the request.

I examined the endpoint of the API which is GKE Ingress that I used, to see if there are any points where timeouts are configured.

I could trace the path through the GCP console’s Ingress as follows:

ingress => load balancer => backend services

There was a timeout setting in one of “Backend services” which I call API.

After modifying that timeout, api call timeout changed to longer. Also receive 202 status code instead of 504 status code.


Reference

TWpower Reading time ~1 minute


환경

  • Java
  • IntelliJ


환경 설정

Spring Test 준비하기

Rest Assured Gradle 설정

plugins {
	id 'java'
	id 'org.springframework.boot' version '3.2.2'
	id 'io.spring.dependency-management' version '1.1.4'
}

group = 'me.twpower'
version = '0.0.1-SNAPSHOT'

java {
	sourceCompatibility = '21'
}

configurations {
	compileOnly {
		extendsFrom annotationProcessor
	}
}

repositories {
	mavenCentral()
}

dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-data-jdbc'
	implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
	implementation 'org.springframework.boot:spring-boot-starter-jdbc'
	implementation 'org.springframework.boot:spring-boot-starter-security'
	implementation 'org.springframework.boot:spring-boot-starter-web'
	compileOnly 'org.projectlombok:lombok'
	runtimeOnly 'com.h2database:h2'
	annotationProcessor 'org.projectlombok:lombok'
	testImplementation 'org.springframework.boot:spring-boot-starter-test'
	testImplementation 'org.springframework.security:spring-security-test'
	testImplementation 'io.rest-assured:rest-assured:5.4.0' // 추가됨
}

tasks.named('test') {
	useJUnitPlatform()
}


코드

  • given(): 요청하기 전에 필요한 헤더나 파라미터와 같은 부분을 세팅
  • when(): 실제로 요청하기위해 URI나 Method를 입력
  • then(): 검증하기 위한 부분
  • header, pathParam, queryParam, body, log 그리고 extract 메소드 사용 방법도 아래에 추가
package me.twpower.restassuredpractice;

import io.restassured.RestAssured;
import io.restassured.response.ExtractableResponse;
import io.restassured.response.Response;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;

import static io.restassured.RestAssured.*;
import static org.hamcrest.Matchers.equalTo;

@SpringBootTest
public class RestAssuredPracticeTest {

    @BeforeAll
    static void beforeAll(){
        // Setting BaseURI
        RestAssured.baseURI = "http://echo.jsontest.com/";
    }

    @Test
    void restAssuredPracticeTest() {
        // JSON Example
        // Request Method: GET
        // Call http://echo.jsontest.com/key1/value1/key2/value2/key3/3?queryParameterKey=queryParameterValue
        /*
        {
            "key1": "value1",
            "key2": "value2",
            "key3": "3"
        }
        */

        // given(): Start building the request part of the test io.restassured.specification.
        // when(): Start building the DSL expression by sending a request without any parameters or headers etc.
        // then(): Returns a validatable response that's lets you validate the response.

        // given() and when() returns RequestSpecification object
        ExtractableResponse<Response> extractableResponse = given().log().all().
            header("Content-Type", "application/json"). // Specify the headers that'll be sent with the request.
            pathParam("pathParameter", 3). // Specify a path parameter.
            queryParam("queryParameterKey", "queryParameterValue"). // Specify a query parameter that'll be sent with the request.
            //body(). // Specify request body.
        when().
            get("/key1/value1/key2/value2/key3/{pathParameter}").
        then().
            body("key1", equalTo("value1")).
            extract();

        Assertions.assertEquals(200, extractableResponse.statusCode());
        Assertions.assertEquals("value1", extractableResponse.jsonPath().getString("key1"));
    }
}


결과

  • 성공
  • 실패
java.lang.AssertionError: 1 expectation failed.
JSON path key1 doesn't match.
Expected: value2
  Actual: value1


참고자료

TWpower Reading time ~1 minute


Environment and Prerequisite

  • Java
  • IntelliJ


Setting

Prepare Spring Test

Setting Rest Assured in Gradle

plugins {
	id 'java'
	id 'org.springframework.boot' version '3.2.2'
	id 'io.spring.dependency-management' version '1.1.4'
}

group = 'me.twpower'
version = '0.0.1-SNAPSHOT'

java {
	sourceCompatibility = '21'
}

configurations {
	compileOnly {
		extendsFrom annotationProcessor
	}
}

repositories {
	mavenCentral()
}

dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-data-jdbc'
	implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
	implementation 'org.springframework.boot:spring-boot-starter-jdbc'
	implementation 'org.springframework.boot:spring-boot-starter-security'
	implementation 'org.springframework.boot:spring-boot-starter-web'
	compileOnly 'org.projectlombok:lombok'
	runtimeOnly 'com.h2database:h2'
	annotationProcessor 'org.projectlombok:lombok'
	testImplementation 'org.springframework.boot:spring-boot-starter-test'
	testImplementation 'org.springframework.security:spring-security-test'
	testImplementation 'io.rest-assured:rest-assured:5.4.0' // Added
}

tasks.named('test') {
	useJUnitPlatform()
}


Code

  • given(): Set up necessary components such as headers or parameters before making a request
  • when(): Input URI or method for the actual request
  • then(): Define verification steps
  • Add the usage methods for header, pathParam, queryParam, body, log and extract below.
package me.twpower.restassuredpractice;

import io.restassured.RestAssured;
import io.restassured.response.ExtractableResponse;
import io.restassured.response.Response;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;

import static io.restassured.RestAssured.*;
import static org.hamcrest.Matchers.equalTo;

@SpringBootTest
public class RestAssuredPracticeTest {

    @BeforeAll
    static void beforeAll(){
        // Setting BaseURI
        RestAssured.baseURI = "http://echo.jsontest.com/";
    }

    @Test
    void restAssuredPracticeTest() {
        // JSON Example
        // Request Method: GET
        // Call http://echo.jsontest.com/key1/value1/key2/value2/key3/3?queryParameterKey=queryParameterValue
        /*
        {
            "key1": "value1",
            "key2": "value2",
            "key3": "3"
        }
        */

        // given(): Start building the request part of the test io.restassured.specification.
        // when(): Start building the DSL expression by sending a request without any parameters or headers etc.
        // then(): Returns a validatable response that's lets you validate the response.

        // given() and when() returns RequestSpecification object
        ExtractableResponse<Response> extractableResponse = given().log().all().
            header("Content-Type", "application/json"). // Specify the headers that'll be sent with the request.
            pathParam("pathParameter", 3). // Specify a path parameter.
            queryParam("queryParameterKey", "queryParameterValue"). // Specify a query parameter that'll be sent with the request.
            //body(). // Specify request body.
        when().
            get("/key1/value1/key2/value2/key3/{pathParameter}").
        then().
            body("key1", equalTo("value1")).
            extract();

        Assertions.assertEquals(200, extractableResponse.statusCode());
        Assertions.assertEquals("value1", extractableResponse.jsonPath().getString("key1"));
    }
}


Result

  • Success
  • Fail
java.lang.AssertionError: 1 expectation failed.
JSON path key1 doesn't match.
Expected: value2
  Actual: value1


Reference