option to require only specific languages?

[maia]

Suggestion: as the language files will grow (abbreviations, contractions,…) and will use more memory, it would be nice to let users require only specific languages, so that when e.g. someone only parses french text, constants and language specific rules of 20+ other languages won't be kept in memory.

[diasks2]

Yes, this is a great idea. Any suggestions on how to best accomplish this? Or would it make sense just to have a separate "lite" gem that doesn't include any languages and the user passes in constants?

[maia]

I've asked around, the suggested solution is neither to use Kernel#load (threading issues) nor to use YAML files, but to give the users an option to manually require languages they need.

So instead of using requireon all languages, one could default to english only and let people either use require 'pragmatic_tokenizer/languages/all' or require 'pragmatic_tokenizer/languages/french' plus whatever other languages one needs.