Remote Grammar Compiler

Server providing a generic API for grammar compilation

Installation

The RGC server will run if there is a missing compiler. A warning will be logged and the requests to use this compiler will be skipped.

Cerence

• Download /home/resource/cerence/csdk/tools/5.2.8/cerence_asr_embedded_tools_win_x86_64_v5_2_8.zip from FTP and install it.

• Create an environment variable RGC_CERENCE_SCRIPT_PATH with value C:\ProgramData\cerence\tools\cerence_asr_embedded_5_2_8\venv\Scripts. By default, the installation directory of cerence asr is C:\ProgramData\cerence\tools\cerence_asr_embedded_5_2_8 if it's not the same make sure to update the installation folder base in the environment variable.

• Download the clc files from FTP and set the environment variable RGC_CERENCE_CLC_PATH to the containing directory. Example of clc file: clc_enu_mpthreevadml_v6_0_1.dat.

• Download the acmod files from FTP and set the environment variable RGC_CERENCE_ACMOD_PATH to the containing directory. Example of acmod file: acmod6_6000_enu_gen_car_f16_v1_0_1.dat.

Sensory

• Download /home/resource/sensory/packages/sensory_tnl_6.17.0_windows.zip from FTP and extract it.

• Create an environment variable RGC_SENSORY_SCRIPT_PATH with value C:\sensory_tnl_6.17.0_windows\bin. In my case I extract it in C:\ if you extract it in any other place make sure to update the path.

• Download the snsr files from FTP and set the environment variable RGC_SENSORY_SNSR_PATH to the containing directory. Example of snsr file: lvcsr-build-enUS-2.3.0.snsr. The language ressources for sensory can be found here: /home/resource/sensory/packages/language_packs.

• Download the tpl files from FTP and set the environment variable RGC_SENSORY_TPL_PATH to the containing directory. Example of tpl file: tpl-vad-lvcsr-3.5.0.snsr. The TPL ressources files for sensory can be found here: /home/resource/sensory/packages/models.

Vivoka

• Open cmd and goto the directory where to install vasr-compiler in our case C:\vasr-compiler.

• Install the vivoka compiler using conan install vasr-compiler/1.0.5@vivoka/stable -o enable_bin=True -o phonetisaurus:shared=False --build missing.

• Create an environment variable RGC_VIVOKA_SCRIPT_PATH with value C:\vasr-compiler\bin. Make sure to update the path if you use different installation directory.

• Download the G2P files from FTP and set the environment variable RGC_VIVOKA_G2P_MODEL_PATH to the containing directory. The directory must have this architecture ${lang}\model.fst. Example of G2P file: eng-US\model.fst. The G2P ressources for vivoka can be found here: /home/resource/vdk/vasr/g2p_models.

• Download the model files from FTP and set the environment variable RGC_VIVOKA_MODEL_PATH to the containing directory. Example of language model directory: final_${lang}/. The models ressources files for vivoka can be found here: /home/resource/vdk/vasr/final_models.

Authentification server

Create an environment variable RGC_AUTHENTICATION_URL and set it the the remote x-api-key server checker URL. For test we can use the fake apikey server located under tools/ and set RGC_AUTHENTICATION_URL to http://127.0.0.1:14181/v1/users. If the authentification server is not correctly configured any incoming requests will be refused.

Configuration

The configuration file is located in config/rgc.conf.json. If the server cannot load the configuration file it will stop.

Environment variables

Queries

The rgc have one unique route /v1/grammar/compile. This route allow grammar compilation for different providers.

Headers

Body

Request

{
    "provider": "vivoka/cerence/sensory",
    "grammars": ["I0JORitFTVYyLjE7CiFncmFtbWFyIHJnYzsKIXN0YXJ0IDxTVEFSVD47CjxTVEFSVD46IGhlbGxvOw=="],
    "verbosity": 1,
    "options": {
        "lang": "eng-US",
        "tpl": "tpl-vad-lvcsr-3.5.0.snsr",
        "oov_rejection": 0.17,
        "nlu": true
    }
}

Response

Return Code

Successful response

{
    "output_files": {
        "fcf": "I0JORitFTVYyLjE7CiFncmFtbWFyIHJnYzsKIXN0YXJ0IDxTVEFSVD47CjxTVEFSVD46IGhlbGxvOw==",
        "s3c": "I0JORitFTVYyLjE7CiFncmFtbWFyIHJnYzsKIXN0YXJ0IDxTVEFSVD47CjxTVEFSVD46IGhlbGxvOw==",
        "snsr": "I0JORitFTVYyLjE7CiFncmFtbWFyIHJnYzsKIXN0YXJ0IDxTVEFSVD47CjxTVEFSVD46IGhlbGxvOw==",
        "fst": "I0JORitFTVYyLjE7CiFncmFtbWFyIHJnYzsKIXN0YXJ0IDxTVEFSVD47CjxTVEFSVD46IGhlbGxvOw=="
    },
    "tasks": [
        {
            "command": null,
            "exit_code": 0,
            "grammar_index": 0,
            "name": "BnfParser"
        },
        {
            "command": null,
            "exit_code": 0,
            "name": "SensoryCompile"
        }
    ]
}

Error response

Example 1

{
    "error": "BnfParser failed with exit code 1: [lexer] token recognition error at: '<songName;\\n'",
    "tasks": [
        {
            "command": null,
            "errors": [
                {
                    "location": {
                        "end": {
                            "column": 8,
                            "line": 7
                        },
                        "start": {
                            "column": 7,
                            "line": 7
                        }
                    },
                    "message": "token recognition error at: '<songName;\\n'",
                    "offending_symbol": "",
                    "type": "lexer"
                },
                {
                    "location": {
                        "end": {
                            "column": 9,
                            "line": 9
                        },
                        "start": {
                            "column": 8,
                            "line": 9
                        }
                    },
                    "message": "mismatched input ':' expecting ';'",
                    "offending_symbol": ":",
                    "type": "parser"
                }
            ],
            "exit_code": 1,
            "grammar_index": 0,
            "name": "BnfParser"
        }
    ]
}

Example 2

{
    "error": "SensoryCompile failed with exit code 1: [subprocess] execve failed: Permission denied",
    "tasks": [
        {
            "command": null,
            "exit_code": 0,
            "grammar_index": 0,
            "name": "BnfParser"
        },
        {
            "command": [
                "/home/me/vivoka/tools/new-rgc/fileTest/snsr-edit.exe",
                "-t",
                "/home/me/vivoka/tools/new-rgc/fileTest/lvcsr-build-enUS-2.3.0.snsr",
                "-o",
                "/tmp/rgc/sensory_M2EdZhkZnsXfPQpVu89x2A9zIrFvs46C/recognizer.snsr",
                "-f",
                "grammar-stream",
                "/tmp/rgc/sensory_M2EdZhkZnsXfPQpVu89x2A9zIrFvs46C/input_grammar_6KqdzYCmOiQn8FH2MOxOW4Movbjndq0j",
                "-s",
                "search.frame-nota=0.170000"
            ],
            "errors": [
                {
                    "location": null,
                    "message": "execve failed: Permission denied",
                    "type": "subprocess"
                }
            ],
            "exit_code": 1,
            "name": "SensoryCompile"
        }
    ]
}