sync_runtime() regenerates etc/sphinx_train.resolved.json from every script
that loads SphinxTrain::Config, so the parallel comp_feat shards can enter
write_file() concurrently. It opened the target with ">", truncating it in
place and bumping its mtime; a sibling shard then judged the file fresh, read
it while still empty, and died "malformed JSON string ... (end of string)" at
BEGIN -- surfacing as "Parallel job failed with exit code 255" with no shard
log. This is the actual cause of the flaky train-g2p-lda-vtln / train-parallel
failures. Write a per-process temp file and rename() it over the target
(atomic on POSIX) so a concurrent reader always sees a complete file.
The fail-fast added for parallel comp_feat read $? unconditionally after
waitfor_job(). For Queue::POSIX, waitfor_job() is waitpid(): if the child was
already reaped (or is not ours) waitpid() returns -1 and $? is stale, so
$? >> 8 became -1, i.e. a spurious "exit code 255" that aborted the whole
stage even though no job actually failed. Only derive an exit status when
waitpid() truly reaped the child, and report a signal death as 128+signo
instead of masking it to a clean zero exit.
WaitForScript exits on nonzero child status; POSIX comp_feat jobs get unique
qmanager names per part/warp. Add diagnose_comp_feat.sh and run it on CI
failure for train-parallel and train-g2p-lda-vtln.
Insert training stage 21.multipron_align after CI to run multipron_align.pl
when CFG_MULTIPRON is set and not no. Use multipron transcript for CD and
later only when the file exists (Util::ShouldUseMultipronTranscript). Default
CFG_MULTIPRON to yes in sphinx_train.cfg; omit or unset variable keeps legacy
behavior without stage 21.
Stage 21 driver: import dirname for lib path; do not pass an extra etc path
to multipron_align.pl. sphinxtrain decodes POSIX wait status from os.system()
so failed Perl stages fail the shell on Unix.
Add CFG_MULTIPRON to sphinx_train.cfg: when yes, Baum-Welch and related steps use BASE_DIR/multipron_align/EXPT.multipron.transcription. Extend GetLists and the untied-HMM, lattice, and BW slaves accordingly. Add multipron_align.pl and multipron_align.py to merge dicts, run sphinx3_align, and write that transcript (beam from CFG_FORCE_ALIGN_BEAM or default 1e-308).
Thanks to Paulo E. Castro
See for details https://sourceforge.net/p/cmusphinx/patches/101/
git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/SphinxTrain@11706 94700074-3cef-4d97-a70e-9c8c206c02f5
consistency with modern trends and other ASR frameworks.
git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/SphinxTrain@11327 94700074-3cef-4d97-a70e-9c8c206c02f5