20 Commits
Author SHA1 Message Date
Kevin Lenzo f13a2d4a16 Resolved: write the resolved config atomically to fix a parallel-shard crash
sync_runtime() regenerates etc/sphinx_train.resolved.json from every script
that loads SphinxTrain::Config, so the parallel comp_feat shards can enter
write_file() concurrently. It opened the target with ">", truncating it in
place and bumping its mtime; a sibling shard then judged the file fresh, read
it while still empty, and died "malformed JSON string ... (end of string)" at
BEGIN -- surfacing as "Parallel job failed with exit code 255" with no shard
log. This is the actual cause of the flaky train-g2p-lda-vtln / train-parallel
failures. Write a per-process temp file and rename() it over the target
(atomic on POSIX) so a concurrent reader always sees a complete file.
2026-08-16 20:09:12 -04:00
Kevin Lenzo 2a2280c425 WaitForScript: don't turn an already-reaped shard into a spurious failure
The fail-fast added for parallel comp_feat read $? unconditionally after
waitfor_job(). For Queue::POSIX, waitfor_job() is waitpid(): if the child was
already reaped (or is not ours) waitpid() returns -1 and $? is stale, so
$? >> 8 became -1, i.e. a spurious "exit code 255" that aborted the whole
stage even though no job actually failed. Only derive an exit status when
waitpid() truly reaped the child, and report a signal death as 128+signo
instead of masking it to a clean zero exit.
2026-08-16 20:02:23 -04:00
Kevin Lenzo 8a421961fb Fail fast on parallel comp_feat shards and diagnose CI gaps.
WaitForScript exits on nonzero child status; POSIX comp_feat jobs get unique
qmanager names per part/warp. Add diagnose_comp_feat.sh and run it on CI
failure for train-parallel and train-g2p-lda-vtln.
2026-06-04 14:29:29 -04:00
Kevin Lenzo f0c09cbf9e Integrate PocketSphinx decode in CI and resolved config; document PS #468 for align 2026-06-04 11:39:52 -04:00
Kevin Lenzo 64784a3108 Additional resolve files 2026-06-04 11:13:41 -04:00
Kevin Lenzo b6b8e9695b Resolve procedural config to static json and use that for params 2026-06-04 11:06:38 -04:00
Kevin Lenzo 321fd173fe Update Util 2026-04-08 15:22:24 -04:00
Kevin Lenzo dbaaa4446e Multipron on by default: stage 21 after CI, CFG_MULTIPRON=no to disable
Insert training stage 21.multipron_align after CI to run multipron_align.pl
when CFG_MULTIPRON is set and not no. Use multipron transcript for CD and
later only when the file exists (Util::ShouldUseMultipronTranscript). Default
CFG_MULTIPRON to yes in sphinx_train.cfg; omit or unset variable keeps legacy
behavior without stage 21.

Stage 21 driver: import dirname for lib path; do not pass an extra etc path
to multipron_align.pl. sphinxtrain decodes POSIX wait status from os.system()
so failed Perl stages fail the shell on Unix.
2026-04-08 14:42:39 -04:00
Kevin Lenzo 2272750f4f Add multipron force alignment and CFG_MULTIPRON training hook
Add CFG_MULTIPRON to sphinx_train.cfg: when yes, Baum-Welch and related steps use BASE_DIR/multipron_align/EXPT.multipron.transcription. Extend GetLists and the untied-HMM, lattice, and BW slaves accordingly. Add multipron_align.pl and multipron_align.py to merge dicts, run sphinx3_align, and write that transcript (beam from CFG_FORCE_ALIGN_BEAM or default 1e-308).
2026-04-08 13:52:32 -04:00
David Huggins-Daines 487c3dba1b fix: VTLN takes precedence over (any) force-alignment 2022-10-13 09:41:50 -04:00
David Huggins-Daines f3bd9763cb fix: do not try to run "python" directory 2022-10-13 09:17:21 -04:00
David Huggins-Daines 42780488d1 fix: try harder to detect failure (still broken) 2022-10-12 20:46:18 -04:00
David Huggins-Daines 8a1d5f1a22 fix: remove final vestiges of cargo cult and fix MLLT 2022-08-02 17:58:20 -04:00
David Huggins-Daines 886fc44279 fix: don't wait for norm, it doesn't exit, fixes TiedWait 2022-08-02 17:41:01 -04:00
Andrew Au ecd4797ce8 Merge remote-tracking branch 'upstream/master' 2018-01-04 20:05:14 -08:00
nshmyrev eb8bfba148 Fix problem with new perl
See for details https://sourceforge.net/p/cmusphinx/discussion/help/thread/2b11d6b0



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinxtrain@13271 94700074-3cef-4d97-a70e-9c8c206c02f5
2017-06-07 16:53:29 +00:00
Andrew Au a456cfc6f9 Output an error message when we cannot execute a tool 2016-10-23 11:55:46 -07:00
nshmyrev 670ff52e89 Allow CI training with small dataset
git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinxtrain@13031 94700074-3cef-4d97-a70e-9c8c206c02f5
2015-06-18 07:36:47 +00:00
nshmyrev 5a89376856 Make sure training scripts run with the correct dictionary
Thanks to Paulo E. Castro

See for details https://sourceforge.net/p/cmusphinx/patches/101/



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/SphinxTrain@11706 94700074-3cef-4d97-a70e-9c8c206c02f5
2013-01-28 22:09:13 +00:00
nshmyrev 7c89bb8a64 scripts_pl folder is named just scripts now. Less perl, more python here for better
consistency with modern trends and other ASR frameworks.



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/SphinxTrain@11327 94700074-3cef-4d97-a70e-9c8c206c02f5
2012-01-29 12:11:41 +00:00