[Libosinfo] [PATCH] force a UTF-8 locale for python3 to avoid broken ascii codec

[Fabiano Fidêncio]

On Wed, Mar 27, 2019 at 10:57 AM Daniel P. Berrangé <berrange at redhat.com> wrote:
>
> The python3 ascii codec violates POSIX C locale requirements by not being
> 8-bit clean in its text handling. It raises an error for any byte with
> top bit set
>
>   >       return codecs.ascii_decode(input, self.errors)[0]
>   E       UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 419: ordinal not in range(128)
>
> To avoid this python bug we must force use of a UTF-8 locale. Ideally we
> would use the C.UTF-8 locale, however, that is not portable across OS,
> only existing on certain Linux distros. Instead we use the en_us.UTF-8
> locale, but only for the character set data.
>
> Signed-off-by: Daniel P. Berrangé <berrange at redhat.com>
> ---
>
> Pushed as a CI build fix for FreeBSD distros
>
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index c63cb6e..9d7f109 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -123,4 +123,4 @@ update-po:
>          done
>
>  check: $(DATA_FILES) $(SCHEMA_FILES)
> -       $(PYTHON) -m pytest $(PYTEST_LOG_LEVEL)
> +       LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 $(PYTHON) -m pytest $(PYTEST_LOG_LEVEL)
> --
> 2.20.1
>
> _______________________________________________
> Libosinfo mailing list
> Libosinfo at redhat.com
> https://www.redhat.com/mailman/listinfo/libosinfo

Daniel,

This commit is the reason of the following breakage (in my personal
gitlab account):
https://gitlab.com/fidencio/osinfo-db/-/jobs/288707257

It seems to happen because both debian & fedora (30+) containers do
not have the required locale.
I'd like to ask your suggestion on how to proceed here:
- Shall we explicitly include glibc-langpack-en as part of the base packages?
  - Its dependencies are: glibc, glibc-commonl
  - Its size is: 6.0 M (on Fedora 30);
- Shall we work around osinfo-db tests in a way that we can make it
work without setting the locale?

Best Regards,
-- 
Fabiano Fidêncio