Entirely reworked text handling pipeline

This breaks the text pipeline into the following pieces:

- input parsing (split known bracketed phoneme sections out first)

- normalization (fix spacing, numbers, abbreviations, etc.)

- phonetization (do CMUdict lookup or g2p prediction)

- sequencing (convert the symbolic representation to integer
  representation)

Each "pipeline" can have a unique combination of operations for each of
these pipeline steps.

This will grow over time.
This commit is contained in:
Brandon Thomas
2022-06-28 03:25:02 -04:00
parent 6a43c8d7ca
commit 76c6dbf12b
33 changed files with 135390 additions and 1 deletions
@@ -0,0 +1,15 @@
from text.pipeline_components.phonetization.cmudict import CMUDict
from text.pipeline_components.phonetization.cmudict import DEFAULT_CMUDICT_PATH
class TestCmuDict:
def test_lookup_success(self):
cmudict = CMUDict(DEFAULT_CMUDICT_PATH)
arpabet_sequence = cmudict.lookup('word')
assert arpabet_sequence == ['W ER1 D'] # TODO: Split the phonemes!
def test_lookup_failure(self):
cmudict = CMUDict(DEFAULT_CMUDICT_PATH)
arpabet_sequence = cmudict.lookup('fakeword')
assert arpabet_sequence == None