Check command substitution’s result exists or not in conditional statement

Environment and Prerequisite

• Linux
• Bash shell(/bin/bash)

Situatiaton and Method

Situatiaton

• Below is situatiaton that conditional statement is needed to check command substitution’s result exists or not
• command substitution is allowing the output of a command to replace the command itself by using subshell.

#!/bin/sh

result=$(ls -al | grep sample)
...

Method

• Use below conditional statements.
• Handle different way by using -z option.
• (Method 1) You can check it by only adding variable in conditional statement. If its length is 0 or it is not set, then it is considered as false.
• (Method 2) Use -z option. If variable next to -z is not set or length is 0, then it is considered as true.

#!/bin/sh

result=$(ls -al | grep sample)

# Return true if its length is not zero.
if [ "${result}" ]
then
    echo "result is not empty"
else
    echo "result is empty"
fi

# Checks if the given string operand size is zero; if it is zero length, then it returns true.
if [ -z "${result}" ]
then
    echo "result is empty"
else
    echo "result is not empty"
fi

Example

• Add specific string if it not exists in file
• Consider file exists

$ cat test.txt
String

$ vi test.sh

#!/bin/sh

set -e

set +e
result=$(cat test.txt | grep "test string")
set -e

if [ -z "${result}" ]
then
    echo "test string" >> test.txt
fi

$ chmod +x test.sh
$ ./test.sh

$ cat test.txt
String
test string

etc

• In grep command, normally the exit status is 0 if a line is selected, 1 if no lines were selected, and 2 if an error occurred. So if you use set -e option, then you need to unset that option before using it.
• Reference: https://www.gnu.org/software/grep/manual/grep.html#Exit-Status

...
set +e
result=$(ls -al | grep sample)
set -e
...

Reference

• https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html
• https://en.wikipedia.org/wiki/Command_substitution
• https://www.tutorialspoint.com/unix/unix-string-operators.htm
• https://stackoverflow.com/questions/12137431/test-if-a-command-outputs-an-empty-string
• https://www.cyberciti.biz/faq/unix-linux-bash-script-check-if-variable-is-empty/
• https://www.gnu.org/software/grep/manual/grep.html#Exit-Status

forEach에서 async await를 사용할 때 결과값이 순차적으로 오지 않는 상황이 발생했는데 이에 대한 해결법과 원인에 대해서 알아보자

환경

• Javascript
• async and await
• Python
• Flask

들어가기에 앞서

• 간단하게 async await에 대해 정리하고 테스트할 환경을 정의한다.

async await

async

• 비동기 함수를 정의하는 키워드
• 하단의 await를 사용하기 위해 붙여줘야합니다.

await

• Promise 객체를 기다리며 async 함수안에서 사용 가능합니다.
• Promise가 resolve되거나 reject될 때까지 기다립니다.
• 쉽게 설명해 비동기 요청에 대한 응답이 올때까지 기다리는 키워드라고 보시면 됩니다.

예제

function timer(x) {
  return new Promise(resolve => {
    setTimeout(() => {
      resolve(x);
    }, 2000);
  });
}

async function test() {
  var x = await timer(10); // wait 2000 milliseconds
  console.log(x); // 10
}

test();

테스트 환경

• 클라이언트에서 본인의 ID를 보내서 그 ID값을 그대로 반환하게 환경을 구축
• 클라이언트는 자바스크립트로 서버는 파이썬으로 했으나 테스트 용도이기에 서버측은 몰라도 이해에 문제는 없습니다.

테스트 구조

서버 코드

from flask import Flask, request
import json
import time

app = Flask(__name__)

host_addr = "0.0.0.0"
port_num = "8080"

@app.route("/test", methods = ['POST'])
def test():
    print("Time:" + str(int(round(time.time() * 1000))) + ", reqeust from client : " + str(request.json['data']))
    return str(request.json['data'])

if __name__ == "__main__":
    app.run(host = host_addr, port = port_num)

문제

• forEach를 사용 시 순차적으로 요청 응답이 오지 않는다.

요청 코드

arr = [1, 2, 3, 4];

arr.forEach(async (i) => {
  const response = await axios.post("http://0.0.0.0:8080/test", {
      data: i,
  });
  console.log("forEachRequestUsingAsyncAwait server response: " + response.data);
});

응답

• 응답이 순서대로 오지 않았다.

$ node test.js
forEachRequestUsingAsyncAwait server response: 1
forEachRequestUsingAsyncAwait server response: 4
forEachRequestUsingAsyncAwait server response: 3
forEachRequestUsingAsyncAwait server response: 2

서버측 로그

• 순서대로 요청이 오지 않았다.

Time:1602409922301, reqeust from client : 1
127.0.0.1 - - [11/Oct/2020 18:52:02] "POST /test HTTP/1.1" 200 -
Time:1602409922302, reqeust from client : 4
127.0.0.1 - - [11/Oct/2020 18:52:02] "POST /test HTTP/1.1" 200 -
Time:1602409922303, reqeust from client : 3
127.0.0.1 - - [11/Oct/2020 18:52:02] "POST /test HTTP/1.1" 200 -
Time:1602409922303, reqeust from client : 2
127.0.0.1 - - [11/Oct/2020 18:52:02] "POST /test HTTP/1.1" 200 -

이유

이유

• forEach는 배열에 대해 Callback을 순차적으로 실행하기 때문이다. Callback을 순차적으로 실행하지만 Callback이 하나가 끝나기까지 기다렸다가 다음이 실행되는게 아니기 때문이다.
• 반복문처럼 하나가 완료되면 다음 순서에 따라 진행되는게 아니라 각각을 콜백으로 실행하기 때문이다.

공식문서 자료

Array.prototype.forEach ( callbackfn [ , thisArg ] )

• ECMAScript Language Specification에 보면 forEach는 “callbackfn should be a function that accepts three arguments. forEach calls callbackfn once for each element present in the array, in ascending order. callbackfn is called only for elements of the array which actually exist; it is not called for missing elements of the array.”라고 나와있다.

arr.forEach(callback(currentValue[, index[, array]]) {
  // execute something
}[, thisArg]);

• MDN Web Docs에 보면 forEach는 “forEach() calls a provided callback function once for each element in an array in ascending order.”라고 나와있다.

해결방법

for-in 혹은 for-of 사용

for-in

• 요청 코드

async function forInRequest () {
  for (const i in arr){
    const response = await axios.post("http://0.0.0.0:8080/test", {
        data: arr[i],
    });
    console.log("forInRequest server: " + response.data);
  }
}

forInRequest();

• 응답 결과

node test.js
forInRequest server: 1
forInRequest server: 2
forInRequest server: 3
forInRequest server: 4

• 서버 응답 결과

Time:1603631271374, reqeust from client : 1
127.0.0.1 - - [25/Oct/2020 22:07:51] "POST /test HTTP/1.1" 200 -
Time:1603631271383, reqeust from client : 2
127.0.0.1 - - [25/Oct/2020 22:07:51] "POST /test HTTP/1.1" 200 -
Time:1603631271386, reqeust from client : 3
127.0.0.1 - - [25/Oct/2020 22:07:51] "POST /test HTTP/1.1" 200 -
Time:1603631271388, reqeust from client : 4
127.0.0.1 - - [25/Oct/2020 22:07:51] "POST /test HTTP/1.1" 200 -

for-of

• 요청 코드

async function forOfRequest () {
  for (const i of arr){
    const response = await axios.post("http://0.0.0.0:8080/test", {
        data: i,
    });
    console.log("forOfRequest server: " + response.data);
  }
}

forOfRequest();

• 응답 결과

node test.js
forOfRequest server: 1
forOfRequest server: 2
forOfRequest server: 3
forOfRequest server: 4

• 서버 응답 결과

Time:1603631202328, reqeust from client : 1
127.0.0.1 - - [25/Oct/2020 22:06:42] "POST /test HTTP/1.1" 200 -
Time:1603631202335, reqeust from client : 2
127.0.0.1 - - [25/Oct/2020 22:06:42] "POST /test HTTP/1.1" 200 -
Time:1603631202338, reqeust from client : 3
127.0.0.1 - - [25/Oct/2020 22:06:42] "POST /test HTTP/1.1" 200 -
Time:1603631202342, reqeust from client : 4
127.0.0.1 - - [25/Oct/2020 22:06:42] "POST /test HTTP/1.1" 200 -

참고자료

• https://developer.mozilla.org/ko/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach
• https://tc39.es/ecma262/#sec-array.prototype.foreach
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await
• https://tc39.es/ecma262/#sec-for-in-and-for-of-statements

When I use async await in javascript, there was problem that results were not in sequential order. Find out reason and solution for it.

Environment and Prerequisite

• Javascript
• async and await
• Python
• Flask

Before In

• Briefly explain about async await and test environment.

async await

async

• Asynchronous function define keyword
• Add it to use below await.

await

• It waits Promise object and can be used in async function.
• Wait until Promise becomes resolve or reject.
• In simple terms, it is a keyword which waits until response of asynchronous request comes.

Examples

function timer(x) {
  return new Promise(resolve => {
    setTimeout(() => {
      resolve(x);
    }, 2000);
  });
}

async function test() {
  var x = await timer(10); // wait 2000 milliseconds
  console.log(x); // 10
}

test();

Test Environment

• Send client’s id and server returns it
• Server is written in python but there is no problem with understanding because we only focus on client code.

Test Structure

Server Code

from flask import Flask, request
import json
import time

app = Flask(__name__)

host_addr = "0.0.0.0"
port_num = "8080"

@app.route("/test", methods = ['POST'])
def test():
    print("Time:" + str(int(round(time.time() * 1000))) + ", reqeust from client : " + str(request.json['data']))
    return str(request.json['data'])

if __name__ == "__main__":
    app.run(host = host_addr, port = port_num)

Issue

• Response does not come sequentially when we use forEach..

Request Code

arr = [1, 2, 3, 4];

arr.forEach(async (i) => {
  const response = await axios.post("http://0.0.0.0:8080/test", {
      data: i,
  });
  console.log("forEachRequestUsingAsyncAwait server response: " + response.data);
});

Response

• Responses are not sequential.

$ node test.js
forEachRequestUsingAsyncAwait server response: 1
forEachRequestUsingAsyncAwait server response: 4
forEachRequestUsingAsyncAwait server response: 3
forEachRequestUsingAsyncAwait server response: 2

Server Log

• Requests are not sequential.

Time:1602409922301, reqeust from client : 1
127.0.0.1 - - [11/Oct/2020 18:52:02] "POST /test HTTP/1.1" 200 -
Time:1602409922302, reqeust from client : 4
127.0.0.1 - - [11/Oct/2020 18:52:02] "POST /test HTTP/1.1" 200 -
Time:1602409922303, reqeust from client : 3
127.0.0.1 - - [11/Oct/2020 18:52:02] "POST /test HTTP/1.1" 200 -
Time:1602409922303, reqeust from client : 2
127.0.0.1 - - [11/Oct/2020 18:52:02] "POST /test HTTP/1.1" 200 -

Reason

Reason

• forEach calls a provided callback function once for each element in an array in ascending order. Callback runs in sequential but it does not wait until one is finished.
• That is because it runs not like iteration which runs next step when one step is finished. It runs each callback.

Documents

Array.prototype.forEach ( callbackfn [ , thisArg ] )

• According to ECMAScript Language Specification, it says “callbackfn should be a function that accepts three arguments. forEach calls callbackfn once for each element present in the array, in ascending order. callbackfn is called only for elements of the array which actually exist; it is not called for missing elements of the array.”

arr.forEach(callback(currentValue[, index[, array]]) {
  // execute something
}[, thisArg]);

• According to MDN Web Docs, it says “forEach() calls a provided callback function once for each element in an array in ascending order.”

Solutions

Use for-in or for-of

for-in

• Request code

async function forInRequest () {
  for (const i in arr){
    const response = await axios.post("http://0.0.0.0:8080/test", {
        data: arr[i],
    });
    console.log("forInRequest server: " + response.data);
  }
}

forInRequest();

• Response result

node test.js
forInRequest server: 1
forInRequest server: 2
forInRequest server: 3
forInRequest server: 4

• Server response result

Time:1603631271374, reqeust from client : 1
127.0.0.1 - - [25/Oct/2020 22:07:51] "POST /test HTTP/1.1" 200 -
Time:1603631271383, reqeust from client : 2
127.0.0.1 - - [25/Oct/2020 22:07:51] "POST /test HTTP/1.1" 200 -
Time:1603631271386, reqeust from client : 3
127.0.0.1 - - [25/Oct/2020 22:07:51] "POST /test HTTP/1.1" 200 -
Time:1603631271388, reqeust from client : 4
127.0.0.1 - - [25/Oct/2020 22:07:51] "POST /test HTTP/1.1" 200 -

for-of

• Request code

async function forOfRequest () {
  for (const i of arr){
    const response = await axios.post("http://0.0.0.0:8080/test", {
        data: i,
    });
    console.log("forOfRequest server: " + response.data);
  }
}

forOfRequest();

• Response result

node test.js
forOfRequest server: 1
forOfRequest server: 2
forOfRequest server: 3
forOfRequest server: 4

• Server response result

Time:1603631202328, reqeust from client : 1
127.0.0.1 - - [25/Oct/2020 22:06:42] "POST /test HTTP/1.1" 200 -
Time:1603631202335, reqeust from client : 2
127.0.0.1 - - [25/Oct/2020 22:06:42] "POST /test HTTP/1.1" 200 -
Time:1603631202338, reqeust from client : 3
127.0.0.1 - - [25/Oct/2020 22:06:42] "POST /test HTTP/1.1" 200 -
Time:1603631202342, reqeust from client : 4
127.0.0.1 - - [25/Oct/2020 22:06:42] "POST /test HTTP/1.1" 200 -

Reference

• https://developer.mozilla.org/ko/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach
• https://tc39.es/ecma262/#sec-array.prototype.foreach
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await
• https://tc39.es/ecma262/#sec-for-in-and-for-of-statements

“SyntaxError: Non-ASCII character …, but no encoding declared”와 같은 이슈가 생겼을 때 해결 방법을 알아보자

환경

• Python

문제와 해결 방법

문제 상황

• 파이썬 코드를 실행했는데 아래와 같은 오류를 보여준다.

python /home/twpower/MailReminder/reminder.py
...
SyntaxError: Non-ASCII character '\xeb' in file /home/twpower/MailReminder/reminder.py on line 12, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

원인

• 실행중인 코드에 ASCII가 아닌 문자가 들어있는데 인코딩을 명시해주지 않아서 생기는 오류

해결 방법

• 파이썬 파일 제일 상단에 # -*- coding: utf-8 -*-를 추가한다.

# -*- coding: utf-8 -*-
...

참고자료

• https://stackoverflow.com/questions/18078851/syntaxerror-of-non-ascii-character

Solution for “SyntaxError: Non-ASCII character …, but no encoding declared”

Environment and Prerequisite

• Python

Problem and Solution

Issue

• Error comes out like below when running python code.

python /home/twpower/MailReminder/reminder.py
...
SyntaxError: Non-ASCII character '\xeb' in file /home/twpower/MailReminder/reminder.py on line 12, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

Cause

• Issue which there was non ascii character in running code but encoding was not declared

Solution

• Add # -*- coding: utf-8 -*- to top of python code file.

# -*- coding: utf-8 -*-
...

Reference

• https://stackoverflow.com/questions/18078851/syntaxerror-of-non-ascii-character