mirror of
https://github.com/storytold/artcraft.git
synced 2026-11-03 00:47:53 +00:00
79 lines
2.6 KiB
ReStructuredText
79 lines
2.6 KiB
ReStructuredText
NeMo Text Processing Deployment
|
|
===============================================
|
|
|
|
This tool deploys :doc:`NeMo Inverse Text Normalization (ITN) <../nemo_text_processing/inverse_text_normalization>` and :doc:`NeMo Text Normalization (TN) <../nemo_text_processing/text_normalization>` for production :cite:`tools-itn_deploy-zhang2021nemo`.
|
|
It uses `Sparrowhawk <https://github.com/google/sparrowhawk>`_ :cite:`tools-itn_deploy-sparrowhawk` -- an open-source version of Google Kestrel :cite:`tools-itn_deploy-ebden2015kestrel`.
|
|
|
|
Requirements
|
|
------------------------
|
|
|
|
:doc:`nemo_text_processing <../nemo_text_processing/intro>` package
|
|
|
|
|
|
Usage
|
|
------------
|
|
|
|
Starts docker container with production backend with plugged in grammars. This is entry point script.
|
|
|
|
Arguments:
|
|
^^^^^^^^^
|
|
* ``GRAMMARS`` - ``tn_grammars`` or ``itn_grammars`` to export either TN or ITN grammars from :doc:`nemo_text_processing <../nemo_text_processing/intro>`.
|
|
* ``INPUT_CASE`` - ``cased`` or ``lower_cased`` (lower_cased is supported only in TN grammars).
|
|
* ``MODE`` - choose ``test`` to run test on the grammars inside the container.
|
|
|
|
For example:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
# to export ITN grammars
|
|
bash export_grammar.sh --GRAMMARS=itn_grammars
|
|
|
|
# to export and test TN grammars
|
|
bash export_grammar.sh --GRAMMARS=itn_grammars --INPUT_CASE=cased --MODE=test
|
|
|
|
This script runs the following steps in sequence:
|
|
|
|
Exports grammar `ClassifyFst` and `VerbalizeFst` from :doc:`nemo_text_processing <../nemo_text_processing/intro>` to `OUTPUT_DIR/classify/tokenize_and_classify.far` and `OUTPUT_DIR/verbalize/verbalize.far` respectively.
|
|
|
|
.. code-block:: bash
|
|
|
|
python pynini_export.py <--output_dir OUTPUT_DIR> <--grammars GRAMMARS> <--input_case INPUT_CASE>
|
|
|
|
Builds C++ production backend docker
|
|
|
|
.. code-block:: bash
|
|
|
|
bash docker/build.sh
|
|
|
|
|
|
Plugs in grammars into production backend by mounting grammar directory `classify/` and `verbalize/` with sparrowhawk grammar directory inside docker. Returns docker prompt
|
|
|
|
.. code-block:: bash
|
|
|
|
# to launch container with the exported grammars
|
|
bash docker/launch.sh
|
|
|
|
# to launch container with the exported grammars and run tests on TN grammars
|
|
bash docker/launch.sh test_tn_grammars
|
|
|
|
# to launch container with the exported grammars and run tests on ITN grammars
|
|
bash docker/launch.sh test_itn_grammars
|
|
|
|
|
|
Runs TN or ITN in docker container:
|
|
|
|
.. code-block:: bash
|
|
|
|
echo "two dollars fifty" | ../../src/bin/normalizer_main --config=sparrowhawk_configuration.ascii_proto
|
|
|
|
This returns $2.50 for ITN.
|
|
|
|
References
|
|
----------
|
|
|
|
.. bibliography:: tools_all.bib
|
|
:style: plain
|
|
:labelprefix: TOOLS-ITN_DEPLOY
|
|
:keyprefix: tools-itn_deploy-
|