kubectl port-forward usage and example


Environment and Prerequisite

  • Kubernetes
  • kubectl


Background

  • While using Kubernetes, I frequently use port forwarding but I often forget how to use it so I post it remember it.


kubectl port-forward

Official Document

  • There are good examples on official document.
  • Not only Pod but also Deployment and Service can be used.
  • You can use --address option to specify the IP address to listen on. It is separated by , and default values are localhost and 127.0.0.1.

Usage

kubectl port-forward TYPE/NAME [options] [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N]

Example

 # Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in the pod
 kubectl port-forward pod/mypod 5000 6000
  
 # Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in a pod selected by the deployment
 kubectl port-forward deployment/mydeployment 5000 6000
 
 # Listen on port 8443 locally, forwarding to the targetPort of the service's port named "https" in a pod selected by the service
 kubectl port-forward service/myservice 8443:https
 
 # Listen on port 8888 locally, forwarding to 5000 in the pod
 kubectl port-forward pod/mypod 8888:5000
 
 # Listen on port 8888 on all addresses, forwarding to 5000 in the pod
 kubectl port-forward --address 0.0.0.0 pod/mypod 8888:5000
 
 # Listen on port 8888 on localhost and selected IP, forwarding to 5000 in the pod
 kubectl port-forward --address localhost,10.19.21.23 pod/mypod 8888:5000
 
 # Listen on a random port locally, forwarding to 5000 in the pod
 kubectl port-forward pod/mypod :5000


Usage Examples

Port forward to Pod

kubectl port-forward pod/[pod name] [local port]:[pod port]

Port forward to Deployment

kubectl port-forward deployment/[deployment name] [local port]:[pod port]

Port forward to Service

kubectl port-forward service/[service name] [local port]:[service port]

Set listen IP address

  • Listen on all IP addresses
kubectl port-forward --address 0.0.0.0 pod/[pod name] [local port]:[port]
  • Listen on local and specific IP address
  • In the example below, it is assumed that the OS instance has the address 10.19.21.23. This is like specifying the IP address for port connection rather than a concept like a whitelist that controls external access.
kubectl port-forward --address localhost,10.19.21.23 pod/[pod name] [local port]:[port]


Reference

네트워크 문제 발생 시 CloudTrail 기록 여부에 대한 정리


환경

  • AWS
  • CloudTrail


배경

  • 서비스 장애를 디버깅 중에 CloudTrail에 기록이 없는 부분을 발견하여 그 이유를 조사함.


과정

  • VPC에 서비스를 elasticloadbalancing.ap-northeast-2.amazonaws.com으로 갖는 VPC Endpoint를 생성하였으며 private_dns_enabled 옵션을 true로 설정.
  • VPC 내부에 있는 프라이빗 서브넷의 EC2 인스턴스에서 ELB 관련한 API(Endpoint의 경우 elasticloadbalancing.ap-northeast-2.amazonaws.com)를 호출.
  • private_dns_enabledtrue로 되어있기 때문에 프라이빗 서브넷에서의 API(Endpoint의 경우 elasticloadbalancing.ap-northeast-2.amazonaws.com)요청의 경우 위에서 만든 VPC Endpoint로 가도록 되어있음.
  • 위에서 만든 VPC Endpoint에 Security Group에서 막혀 Timeout이 발생했으며 CloudTrail에 로그가 남지 않음을 발견.
  • AWS Support Case에 문의.


결론

  • AWS API 사용 시 네트워크 문제로 Timeout이 발생해 API 요청이 이루어지지 않으면 CloudTrail에 로그가 남지 않는다.
  • 관련하여 AWS에 Support Case를 오픈하여 위 부분에 대한 확인 완료.


참고자료

Post about CloudTrail log result in network issue


Environment and Prerequisite

  • AWS
  • CloudTrail


Background

  • While debugging a service issue, I discovered there is no log in CloudTrail. So I investigated the reason.


Process

  • Create VPC Endpoint which service is elasticloadbalancing.ap-northeast-2.amazonaws.com and set private_dns_enabled to true.
  • Call ELB related API(in this case the endpoint is elasticloadbalancing.ap-northeast-2.amazonaws.com) from an EC2 instance in a private subnet within a VPC.
  • Because private_dns_enabled is set to true, API(in this case the endpoint is elasticloadbalancing.ap-northeast-2.amazonaws.com) call in private subnet would be forwarded to VPC Endpoint.
  • Due to above created VPC Endpoint Security Group, timeout occurs and found there was no log on CloudTrail.
  • Ask to AWS Support Case.


Conclusion

  • When there is a network issue, such as a timeout while using the AWS API, there will be no logs in CloudTrail.
  • I opened support case on AWS and checked that above fact is right.


Reference

장애 대응 사례 정리


환경

  • Cloud Service Provider(AWS, GCP, Azure)
  • API


배경

  • 운영중인 서비스에서 사용하는 Cloud Service Provider의 API가 제대로 작동하지 않아 이슈가 발생한적이 있어서 정리해보면 좋을거 같아 정리한다.
  • 사용했던 CSP의 경우 AWS, GCP, Azure 중에 하나지만 특정해서 적으면 약간 Blame하는 느낌이 들수 있어서 CSP로 표현


타임라인

  • 5/20 08:30: Slack에 운영중인 실데이터 수집 이슈 알림
  • 5/20: 이슈 파악 작업 진행
  • 5/21: 이슈 파악 후 CSP에 Support Case 작성
  • 5/22: CSP에서 이슈 해결되어 정상 동작 확인
  • 5/23: 이슈 해결


이슈 및 원인

  • 사용중인 CSP의 API 호출 결과에 일부 내역이 누락되는 현상 발생


영향도

  • 운영중인 Daily 실데이터 수집 장애로 이후 연결된 파이프라인 및 서비스에서 장애 발생


조치 이력

  • 이슈 발생 후 수동으로 작업해서 실데이터 추출을 내보냈다고 표시를 할 수 있는 fin파일을 수동으로 생성
  • CSP 이슈로 판단하여 Support Case 작성


해결

  • CSP에서 해당 API 이슈를 해결함을 통해 이슈는 해결됨


회고

  • CSP 같은 곳은 서비스나 어떤 소프트웨어를 만들 때 이슈가 없을거라고 당연히 가정하고 개발을 했었는데 이러한 생각을 바꿀 필요도 있을거 같다.
  • 이전에 2022년 10월 15일에 데이터 센터 화재로 카카오 서비스가 장애가 발생해 대응한 사례가 있었는데 이를 다시 공부해봤다. 아래 참고자료에 해당 유튜브 링크를 올린다.


참고자료

Case study of incident response


Environment and Prerequisite

  • Cloud Service Provider(AWS, GCP, Azure)
  • API


Background

  • I wrote this post because there was an issue that API of Cloud Service Provider used in running service did not function properly.
  • I used word ‘CSP’ to refer to the Cloud Service Provider in one of AWS, GCP, Azure to avoid pick one specific provider.


Timeline

  • 5/20 08:30: Receive Slack Issue
  • 5/20: Investigate reason of issue
  • 5/21: Open support case after find cause
  • 5/22: Issue resolved in CSP
  • 5/23: Issue resolved


Issue and Cause

  • Some of exported history were not returned when using CSP’s API


Impact

  • An incident occurred in daily data collection which leads to subsequent failures in the connected pipeline and services.


Act History

  • Manually create fin file which represents raw data collection is done after issue occured
  • Open a support case due to a CSP issue.


Resolve

  • Issue resolved after CSP’s API issue is resolved


Retrospective

  • I thought there will be no issue in CSP or other third party software. However I changed my mind that CSP also can have issues.
  • I studied incident response of Kakao issue which Kakao’s services were down due to a data center fire. The relevant YouTube video link is referenced in below.


Reference