Add UTF-8 encoding to file operations in multiple modules

[IndigoCarmine]

SasView/sasview#4095

For this issue, I added the encoding="utf-8" argument so that model source files are explicitly read as UTF-8.

This fixes the UnicodeDecodeError that occurs when building plugin models on Windows systems using a non-UTF-8 locale, such as the Japanese locale.

[codescene-access]

No quality gates enabled for this code.

See analysis details in CodeScene

Quality Gate Profile: Custom Configuration
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.

[pkienzle]

I can believe this fixes the problem, but having to modify every open statement in every every python module in every application is an icky solution.

Python 3.15 will fix the issue by making UTF-8 the default when it is released in October. It'll probably take a couple of months for dependencies (numpy, scipy, etc.) to be updated.

The CP932 encoding shares code points with ASCII, so decoding is not a problem if only ASCII characters are used in the model definition.

Since the problem is only with adaptive.c, I suggest removing the non-ASCII characters from that file and wait for the problem to fix itself. The documentation describing how to implement adaptive integration belongs in doc/guide/plugin.rst so we should move it anyway.

Another option is to modify the SasView startup on Windows to set PYTHONUTF8=1 in the environment before starting. This should be equivalent to the python 3.15 behaviour.

[IndigoCarmine]

Thank you for the reply.

I also considered removing the non-ASCII chars from adaptive.c, and it is the efficient and correct way to fix this. However, it takes expressiveness away just because of a character encoding error. The same applies to user-written plugin models, where a non-ASCII description would hit exactly the same crash.

I think we should instead make the environment able to handle any Unicode character. Your second option, setting PYTHONUTF8=1 on Windows startup, looks like the better way. End users who just download the installer can't set it themselves - so it would mean adding python_options=['X utf8'] in installers/sasview.spec.

In the meantime, affected users can work around it with setx PYTHONUTF8 1.

On the "modifying every open statement is icky" point - within sasmodels there are only about six open() calls without an explicit encoding=, and they are all one-line edits, so that route may be more tractable than it sounds. (Though I well understand not wanting to write the same thing over and over.)

I'd like to hear other opinions too.