[Dev](EN) Timing issue in Pub/Sub and database write operation


Environment and Prerequisite

  • Pub/Sub
  • RDB
  • Java


Background

  • While processing list in parallel to publish Pub/Sub events, an issue occurred where the final state in the database was saved differently than intended.
  • Data inconsistency is occurring even though all database write logs appear normal.


Issue

  • Issues occurred in the logic that uses parallelStream() to send bulk messages and batch update their statuses.
  • Although the system received a “Processing Succeeded (SUCCESS)” response from GCP Pub/Sub and updated the DB, the final result remained in the “Published (PUBLISHED)” state.
  • The following pseudo-code illustrates the logic where the issue occurred.
// 1. Parallel event publishing (Main Thread)
jobList.parallelStream().forEach(job -> {
    // Publishing Pub/Sub messages
    publishEvent(job); 
});

// ---------------------------------------------------------
// [Key Point] The SUCCESS response from the Pub/Sub subscrition arrives first!
// A separate subscription thread (Sub Thread) has already updated the DB to SUCCESS.
// ---------------------------------------------------------

// 2. Status update after publishing (Main Thread)
// This runs late and overwrites the DB with "PUBLISHED"
jobRepository.findAllById(ids).forEach(job -> {
    // AS-IS: Unconditional overwrite without checking the current status
    job.setStatus(PUBLISHED); 
    jobRepository.save(job); // Integrity error: SUCCESS reverts to PUBLISHED
});


Reason

Pub/Sub subscription response arriving faster than the DB write

  • Expected Flow: Publish to Pub/Sub -> Change to “Published” (DB: PUBLISHED) -> Receive Subscription Message (DB: SUCCESS)
  • Abnormal Flow: Publish to Pub/Sub -> Receive Subscription Message (DB: SUCCESS) -> Change to “Published” (DB: PUBLISHED)


Thoughts

I have encountered similar issues before regarding parallel or asynchronous processing and state updates ([Javascript](EN) async await is not work sequentially in forEach). It seems this is a recurring pattern that requires constant care during development.