bitbake: utils: Force bitbake to en_US.UTF-8 locale setting everywhere

Under python 3, if we spawn python processes, we need to have a UTF-8
locale, else python's file access methods will use ascii. You can't
change that mode once the interpreter is started so we have to ensure
a locale is set. Ideally we'd use C.UTF-8 since OE already forces the
C locale but not all distros support that and we need to set something.
Was tempted to choose en_GB so colour gets spelt correctly :).

This is in some ways pretty nasty, forcing it into the environment
everywhere however we only have a limited number of ways of making
everything work correctly and this beats having to add utf-8 encoding
to every file access command.

A similar change will be needed to bitbake.conf in OE.

(Bitbake rev: 8902c29638411d312e6fc4a197707e5742652e15)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake/lib/bb/utils.py

@@ -568,6 +568,7 @@ def preserved_envvars_exported():
         'SHELL',
         'TERM',
         'USER',
+        'LC_ALL',
     ]
 
 def preserved_envvars():
@@ -595,6 +596,12 @@ def filter_environment(good_vars):
         os.unsetenv(key)
         del os.environ[key]
 
+    # If we spawn a python process, we need to have a UTF-8 locale, else python's file
+    # access methods will use ascii. You can't change that mode once the interpreter is
+    # started so we have to ensure a locale is set. Ideally we'd use C.UTF-8 but not all
+    # distros support that and we need to set something.
+    os.environ["LC_ALL"] = "en_US.UTF-8"
+
     if removed_vars:
         logger.debug(1, "Removed the following variables from the environment: %s", ", ".join(removed_vars.keys()))