externalsrc: do not use do_configure[nostamp] for git srctrees

Be a bit more intelligent than mindlessly re-compiling every time.
Instead of always using 'nostamp' flag for do_compile run a python
function to get a list of files to add as 'file-checksums' flag. The
intention is to only re-run do_compile if something in the source tree
content changes.

This python function, srctree_hash_files(), works differently, depending
if the source tree is a git repository clone or not. If the source tree
is a git repository, the function runs 'git add .' and 'git write tree'
to get a hash of the working tree and writes this hash into a file under
the .git directory. This file containing the hash is then returned as
the file for the task to depend on. Hash is changed if any changes are
made in the source tree causing the task to be re-run. A trick is used
to parse the recipe every time so that the hash file gets updated.

If the source tree is not a git repository behaviour remains the same.
In this case srctree_hash_files() currently sets the 'nostamp' flag for
do_compile causing it to be re-run every time.

This method of tracking changes source tree changes to determine if
re-build is needed does not work perofectly, though. Many packages are
built under ${S} which effectively changes the source tree causing some
unwanted re-compilations.  However, if do_compile of the recipe does not
produce new/different artefacts on every run (as commonly is and should
be the case) the re-compilation loop stops. Thus, you should usually see
only one re-compilation (if any) after which the source tree is
"stabilized" and no more re-compilations happen.

[YOCTO #8853]

(From OE-Core rev: a26becdf981b35d7ef8524f9e65c25a74b842f1d)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

meta/classes/externalsrc.bbclass

@@ -85,8 +85,10 @@ python () {
         d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
         d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ")
 
-        # Ensure compilation happens every time
-        d.setVarFlag('do_compile', 'nostamp', '1')
+        # Force the recipe to be always re-parsed so that the file_checksums
+        # function is run every time
+        d.setVar('BB_DONT_CACHE', '1')
+        d.setVarFlag('do_compile', 'file-checksums', '${@srctree_hash_files(d)}')
 
         # We don't want the workdir to go away
         d.appendVar('RM_WORK_EXCLUDE', ' ' + d.getVar('PN', True))
@@ -124,3 +126,28 @@ python externalsrc_compile_prefunc() {
     # Make it obvious that this is happening, since forgetting about it could lead to much confusion
     bb.plain('NOTE: %s: compiling from external source tree %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
 }
+
+def srctree_hash_files(d):
+    import shutil
+    import subprocess
+
+    s_dir = d.getVar('EXTERNALSRC', True)
+    git_dir = os.path.join(s_dir, '.git')
+    oe_index_file = os.path.join(git_dir, 'oe-devtool-index')
+    oe_hash_file = os.path.join(git_dir, 'oe-devtool-tree-sha1')
+
+    ret = " "
+    if os.path.exists(git_dir):
+        # Clone index
+        shutil.copy2(os.path.join(git_dir, 'index'), oe_index_file)
+        # Update our custom index
+        env = os.environ.copy()
+        env['GIT_INDEX_FILE'] = oe_index_file
+        subprocess.check_output(['git', 'add', '.'], cwd=s_dir, env=env)
+        sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env)
+        with open(oe_hash_file, 'w') as fobj:
+            fobj.write(sha1)
+        ret = oe_hash_file + ':True'
+    else:
+        d.setVarFlag('do_compile', 'nostamp', '1')
+    return ret