Difference between revisions of "LangID"
Jump to navigation
Jump to search
| (2 intermediate revisions by one other user not shown) | |||
| Line 8: | Line 8: | ||
|- | |- | ||
| Percent Complete | | Percent Complete | ||
| − | | style="text-align:center;" | | + | | style="text-align:center;" | 48% |
|- | |- | ||
| Message Queue | | Message Queue | ||
| Line 47: | Line 47: | ||
! Description | ! Description | ||
|- | |- | ||
| − | | langid_min_probability | + | | <code>langid_min_probability</code> |
| float | | float | ||
| The minimum threshold for a positive language identification. Between 0.0 and 1.0. Default is 0.333. | | The minimum threshold for a positive language identification. Between 0.0 and 1.0. Default is 0.333. | ||
|- | |- | ||
| − | | langid_max_bytes_to_interpret | + | | <code>langid_max_bytes_to_interpret</code> |
| integer | | integer | ||
| The maximum bytes to scan for language identification. Default is 1000. | | The maximum bytes to scan for language identification. Default is 1000. | ||
|} | |} | ||
| + | |||
| + | E.G. for <code>[[Configuration_System#Truxton_Settings_File|TruxtonSettings.xml]]</code> | ||
| + | <source lang="xml"> | ||
| + | <truxton_options> | ||
| + | <!-- ... | ||
| + | other configs | ||
| + | ... | ||
| + | --> | ||
| + | <langid_min_probability>0.75</langid_min_probability> | ||
| + | <langid_max_bytes_to_interpret>12000</langid_max_bytes_to_interpret> | ||
| + | </truxton_options> | ||
| + | </source> | ||
Latest revision as of 08:11, 10 May 2022
| Executable | LangID.exe
|
| Stage | 18 |
| Percent Complete | 48% |
| Message Queue | langid
|
The LangID ETL scans ascii and unicode files and identifies non-english language. If it is over a certain confidence threshold it will then tag the document with the language it found.
File Types
LangID processes the following types of files.
| File Type | Produces |
|---|---|
| Type_ASCII_Text | Tags |
| Type_UTF8_Encoded_Text | Tags |
| Type_Extracted_ASCII | Tags |
| Type_Extracted_Unicode | Tags |
Note: The types Type_Extracted_ASCII and Type_Extracted_Unicode are typically provided by the TextExtract ETL.
Configuration
| Name | Data Type | Description |
|---|---|---|
langid_min_probability
|
float | The minimum threshold for a positive language identification. Between 0.0 and 1.0. Default is 0.333. |
langid_max_bytes_to_interpret
|
integer | The maximum bytes to scan for language identification. Default is 1000. |
E.G. for TruxtonSettings.xml
<truxton_options>
<!-- ...
other configs
...
-->
<langid_min_probability>0.75</langid_min_probability>
<langid_max_bytes_to_interpret>12000</langid_max_bytes_to_interpret>
</truxton_options>
