[C++] Sporadic test errors in parallel mode

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

I run C++ tests in parallel (26 threads) and I have started to get random sporadic errors like this:

76/97 Test #13: arrow-gtest-util-test ........................***Failed    0.15 sec
Running arrow-gtest-util-test, redirecting output into /build/build-test/build/test-logs/arrow-gtest-util-test.txt (attempt 1/1)
Running main() from gmock_main.cc
[==========] Running 8 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from TestAssertContainsNaN
[ RUN      ] TestAssertContainsNaN.BatchesEqual
[       OK ] TestAssertContainsNaN.BatchesEqual (1 ms)
[ RUN      ] TestAssertContainsNaN.TableEqual
[       OK ] TestAssertContainsNaN.TableEqual (0 ms)
[ RUN      ] TestAssertContainsNaN.ArrayEqual
[       OK ] TestAssertContainsNaN.ArrayEqual (0 ms)
[ RUN      ] TestAssertContainsNaN.ChunkedEqual
[       OK ] TestAssertContainsNaN.ChunkedEqual (0 ms)
[ RUN      ] TestAssertContainsNaN.DatumEqual
[       OK ] TestAssertContainsNaN.DatumEqual (0 ms)
[----------] 5 tests from TestAssertContainsNaN (2 ms total)

[----------] 2 tests from TestTensorFromJSON
[ RUN      ] TestTensorFromJSON.FromJSONAndArray
[       OK ] TestTensorFromJSON.FromJSONAndArray (0 ms)
[ RUN      ] TestTensorFromJSON.FromJSON
[       OK ] TestTensorFromJSON.FromJSON (0 ms)
[----------] 2 tests from TestTensorFromJSON (0 ms total)

[----------] 1 test from AssertTestWithinUlp
[ RUN      ] AssertTestWithinUlp.Basics
[       OK ] AssertTestWithinUlp.Basics (0 ms)
[----------] 1 test from AssertTestWithinUlp (0 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 3 test suites ran. (3 ms total)
[  PASSED  ] 8 tests.
find: ‘/tmp/s3fs-test-7cm44otu’: Aucun fichier ou dossier de ce nom

Component(s)

C++

[pitrou]

I think this is due to the use of GNU find in #50935 . @kou @hiroyuki-sato

[pitrou]

It seems to happen in print_coredumps. I've tried passing -ignore_readdir_race but it doesn't seem to change everything.

Edit: the patch I've tried:

diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index 2dc18860a1..9f2828a9d9 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -141,7 +141,7 @@ function print_coredumps() {
   FILENAME=$(basename "${TEST_EXECUTABLE}")
   FILENAME=$(echo "${FILENAME}" | cut -c-15)
 
-  COREFILES=$(find /tmp -maxdepth 1 -type f -name "core.${FILENAME}*" -exec basename {} \;)
+  COREFILES=$(find /tmp -ignore_readdir_race -maxdepth 1 -type f -name "core.${FILENAME}*" -exec basename {} \;)
   if [ -n "$COREFILES" ]; then
     for COREFILE in $COREFILES; do
       COREPATH="/tmp/${COREFILE}"
@@ -199,12 +199,12 @@ for ATTEMPT_NUMBER in $(seq 1 "$TEST_EXECUTION_ATTEMPTS") ; do
     # contents before and after the test runs, and deleting anything new.
     #
     # The comm program requires that its two inputs be sorted.
-    TEST_TMPDIR_BEFORE=$(find "$TEST_TMPDIR" -maxdepth 1 -type d | sort)
+    TEST_TMPDIR_BEFORE=$(find "$TEST_TMPDIR" -ignore_readdir_race -maxdepth 1 -type d | sort)
   fi
 
   if [ "$ATTEMPT_NUMBER" -lt "$TEST_EXECUTION_ATTEMPTS" ]; then
     # Now delete any new test output.
-    TEST_TMPDIR_AFTER=$(find "$TEST_TMPDIR" -maxdepth 1 -type d | sort)
+    TEST_TMPDIR_AFTER=$(find "$TEST_TMPDIR" -ignore_readdir_race -maxdepth 1 -type d | sort)
     DIFF=$(comm -13 <(echo "$TEST_TMPDIR_BEFORE") \
                     <(echo "$TEST_TMPDIR_AFTER"))
     for DIR in $DIFF; do

[pitrou]

What we could do perhaps is only print coredumps on error. This would not eliminate a potential race condition in find, but would make it much less likely:

diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index 2dc18860a1..c618e90c98 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -229,18 +229,16 @@ for ATTEMPT_NUMBER in $(seq 1 "$TEST_EXECUTION_ATTEMPTS") ; do
   fi
   if [ "$STATUS" -eq "0" ]; then
     break
-  elif [ "$ATTEMPT_NUMBER" -lt "$TEST_EXECUTION_ATTEMPTS" ]; then
-    echo Test failed attempt number "$ATTEMPT_NUMBER"
-    echo Will retry...
   fi
+  print_coredumps
+  echo Test failed attempt number "$ATTEMPT_NUMBER"
+  echo Will retry...
 done
 
 if [ "$RUN_TYPE" = "test" ]; then
   post_process_tests
 fi
 
-print_coredumps
-
 popd
 rm -Rf "$TEST_WORKDIR"