It was reported that do_validate_branches was failing with the following
error:
Log data follows:
| DEBUG: Executing shell function do_validate_branches
| HEAD is now at fe0fb8d Merge tag 'v4.10.9' into standard/base
| mkdir: cannot create directory .: File exists
|
| [ERROR] Can't find patch dir at ./patches/standard/base
| usage: kgit s2q
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_validate_branches
This was triggered by the execution of 'kgit-s2q --clean' after forcing
the SRCREV to something other than the tip of the branch. --clean is
being run to remove any sentinel files from previous kernel builds to
ensure that the tree is in a consistent state.
There were two bugs, --clean was being executed and not exiting the
script as it was supposed to. Hence validation for applying patches
was done, and threw the error that eventually makes it to the console.
And the second bug is that since do_validate_branches actually calls
kgit-s2q --clean, the dependency on kern-tools-native needs to be on
that function (versus do_kernel_metadata which runs later).
With the tweaked kern-tool + the dependency fix, we no longer see this
error.
(From OE-Core rev: 4d5890b54cbdac01ee748759578b7b22ed8e61a2)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases it is seen that kernel_checkout and validate_branches are
run again in simultaneous builds. During do_patch the kgit-s2q mechanism
looks for a sentinel file inside the .git directory, finds a fence post
and starts picking up patches after that.
This can create trouble as validate_branches checks out the HEAD of the
branch and so the patches should be reapplied rather than skipped due to
finding of the fence post.
We can call kgit-s2q --clean to remove the sentinel file when the
branches are checked out.
(From OE-Core rev: d57aeafdd2b49010a9fa6d1cd9d10f3cfd5754a5)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ability to merge two branches directly from a .scc file was
dropped during the streamlining of the tools.
As was pointed out by David Vincent <freesilicon@gmail.com>, there is
once again a valid use case for this functionality, so we restore the
capability.
(From OE-Core rev: a0059ebbb52c659282e355664bba1a2fa282170e)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Options with spacs around = signs will be droped/ignored by the
kernel.
The audit phase can detect this and warn the user:
% kconf_check --report -o .kernel-meta/cfg/ \
linux-qemux86-standard-build/.config `pwd` $cfgs
[errors (3): .kernel-meta/cfg/fragment_errors.txt
There are errors withing the config fragments.
% cat .kernel-meta/cfg/fragment_errors.txt
Warning: Ignoring "CONFIG_PARAVIRTT_CLOCK =y" -- spaces around equals are invalid
Warning: Ignoring "CONFIG_PARAVIRTTT_CLOCK = y" -- spaces around equals are invalid
Warning: Ignoring "CONFIG_PARAVIRTTTT_CLOCK= y" -- spaces around equals are invalid
(From OE-Core rev: 952a455c09eb88755b2d24ae05c2d6b886922b63)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 626ceac135fa66277c2fa53197be33cc9d4d7614 broke the error
check in process_file by adding in three lines that stomp on $? which
print the output file when verbose is set.
Move output file on verbose print to an elif after the error check.
Signed-off-by: George McCollister <george.mccollister@gmail.com>
(From OE-Core rev: 01657e8023ee535420ab5ba4a5d1952e13bce0cd)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to avoid using shared/common directories for any files that are
part of specific build, since permissions issues in multi user
environments will cause issues.
Integrating the following commit to solve the issue:
scc: move unused patch queue under output dir
(From OE-Core rev: cad65cc0eef2e06cb5ae08062ffae7a4d43a51ad)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Lernel meta-data that has patches, but no branches, can trigger an
error due to no branch specific patch queue.
This error then cascades to more issues since the tools are using
a named file in /tmp to store and display error messages to the
user.
We fix both issues though the following kern tools tweaks:
commit bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Fri Dec 2 13:09:40 2016 -0500
scc: use mktemp for consolidated output capture
To provide useful error messages the tools dump pre-processed
files and messages to a temporary file. If multiple users are
doing builds, this means they either race, or can have permissions
issues.
By creating the temporary file via mktemp, we avoid both issues.
(We also make sure to clean these up on exit, or /tmp will get
polluted quickly).
commit a287da4bfe0b4acb8f2b0627bd8e7abd1a1dde26
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Fri Dec 2 13:08:08 2016 -0500
patch: do not assume a branch specific patch queue is needed
When processing input files per-branch and global patch queues are
generated. If the meta-data has not created any branches in the
repo, no branch specific queue is required.
The tools assumed that one is always valid, and hence would throw a
non-zero exit code and stop processing.
By testing for a named per-branch queue, we avoid this issue.
(From OE-Core rev: 0fd7da7375f0dcc59b56791fd482de557507c04c)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During processing of the kernel meta data the kern tools were
not properly exiting on syntax errors or invalid commands.
Noticing and debugging these issues wasn't trivial. To make this
easier, we now trap the error and dump the offending meta-data
for the user to see.
There was also an issue with creating branches during tree
generation, which is resolved by always switching to the
active branch.
The following are the commit logs of the changes themselves:
[
commit b36f6f9a5695f2084b83823393e13ca42284bed9
Author: Paul Gortmaker <paul.gortmaker@windriver.com>
Date: Sat Oct 22 17:23:25 2016 -0400
kgit-scc: dont mention meta-repo in help ; it doesnt exist
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
commit 08463d684c1952e74c25344cddace4c3f24c739d
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Mon Oct 31 14:30:12 2016 -0400
scc: exit on error
If there is an error in the processing of the input files, scc
should exit and inform the user.
scc is executed on a combined/preprocessed file and as a result
it doesn't have the granularity to see each input file individually.
Rather than moving preprocessing into scc (from spp), we can trap
the line number of the error and dump context around the line.
This gives the user a pointer to the input file and the specific
line that caused the problem.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
commit bf99953e8ac14cee653e559f2d4a6022c847a182
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Fri Oct 28 21:23:27 2016 -0400
kgit-meta: always checkout branches on branch commands
During a tree generation we must always make the branch active when
we see any kind of branch command. This ensures that any subsequent
patches are applied in the proper context.
Previously, only branch creation was changing the active branch, and
this mean that tree generation was not determinstic and relied
on the order of processing to generate a correct tree.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
]
(From OE-Core rev: 83d10e2acef936b1f38804988f10eafa48db36f9)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a patch contains characters like ; or (), it can cause processing
errors in the patch queue.
We had previously fixed this issue by renaming the patches to not
include invalid characters, but with this change to the kern tools
that ensures patch names are wrapped in quotes, we avoid shell
processing and hence they are valid.
(From OE-Core rev: 727afe9e6fb9ef4a24a36cc907ebb6336d12184e)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've been running with a set of kern-tools that were designed to work
with build systems that knew nothing about git, trees, commits, etc.
As such, there's been a set of shims/wrappers in place to work with
within bitbake/oe-core. These were the *me scripts: createme, updateme,
patchme and configme.
With this commit, we strip that legacy code and use the tools directly.
This means less complexity, fewer corner cases .. and no surprises
when the tools are arunning. As another benefit, the tools consume
much less time during a typical build and have no noticeable impact
on the overall build time.
Existing .scc files, features, and processing are not impacted as
these tools are compatible with existing feature descriptions and
kerne configuration fragments.
The audit of kernel configuration fragments is now detached
from the linux-yocto build structure and process. This means that
they can eventually be tweaked to offer kernel audit to any type of
kernel build and configuration process.
Additionally, the kernel symbol audit phase can now resolve symbol
dependencies and offer guidance when a symbol is missing:
WARNING: linux-yocto-4.4.15+gitAUTOINC+b030d96c7b_f5e2c49d58-r0 do_kernel_configcheck: [kernel config]: specified values did not make it into the kernel's final configuration:
---------- CONFIG_BT_6LOWPAN -----------------
Config: CONFIG_BT_6LOWPAN
From: /home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel-source/.kernel-meta/configs/standard/features/bluetooth/bluetooth.cfg
Requested value: CONFIG_BT_6LOWPAN=y
Actual value:
Config 'BT_6LOWPAN' has the following conditionals:
BT_LE && 6LOWPAN (value: "n")
Dependency values are:
BT_LE [y] 6LOWPAN [n]
(From OE-Core rev: 0f698dfd1c8bbc0d53ae7977e26685a7a3df52a3)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Robert P. J. Day reported that configuration fragments and kernel
features were not being found when organized in a particular manner:
linux
- $BOARD
- mm.patch
- mm.scc
- ssd_sil.cfg
- ssd_sil.patch
- ssd_sil.scc
- uio.cfg
.. etc
There was a bug in the tools that did not handle the mix of subdirs
properly and ended up leaving a trailing / on the elements *not* in
the $BOARD subdir. As a result, the configuration fragments were not
properly found when searching the include paths, and a configuration
failure was triggered (due to missing files).
This change tweaks the tools to always check a path with and without
a trailing / when processing config fragments so they can be later
found when processing the configuration of the kernel.
Reported-by: "Robert P. J. Day" <rpjday@crashcourse.ca>
(From OE-Core rev: 92ba77bea59a33b0ddbd5db36e2a1b42e8fd7190)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the recent changes to improve patch processing times, the ability
to skip already applied patches is not active by default.
The automatic detection and resume was hiding issues with the include
files generated by scripts like yocto-bsp.
If a .scc file that contains a patch is included twice, the patch is
applied twice, and the second appliation fails for obvious reasons.
We can partially fix this by ensuring that already included
configuration fragments are not forced into the meta-series.
.scc files that are explicitly listed twice will continue to fail, and
recipes must be modified to avoid this.
[YOCTO: #8486]
(From OE-Core rev: ed2da98bf3ac798009f58a53b91285b4dac69d5a)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-tools SRCREV to import the following fix:
kgit-meta: resume after last applied patch
When the auto-resume (resume point detection) was removed from the
processing of a meta-series, it ignored the fact that a single patch
series may in fact be processed a number of times.
Two layers patching a kernel will generate two different runs on the
same branch, which always start at patch one. This will obviously
break with duplicate patches.
To avoid this, we simply track the last patch applied, and
explicitly
tell the patch scripts where to start. This gets us resume
functionality, without the overhead of resume point detection.
(From OE-Core rev: 692f1333e257556e7462b2436dd60e865869349c)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-tools SRCREV to integrat the following commit:
patching: only validate user supplied patches by default
Previously the patching tools would consider both system and user
supplied patches in the same manner .. they are simply a series of
patches to be applied to a branch, and that the scripts should determine
where in the series to start (based on what is already on the
branch).
This detection was causing a few problems:
- time consuming
- starting in the middle of a series when intermediate patches
were merged to a branch.
To solve both the performance and start detection, we instead simply
note the transition from system (i.e. already defined features and
series) and user/recipe supplied patches. When the transition is noted,
the system will start pushing ALL patches without doing autoresume
detection.
Control in keeping the series up to date is passed to the user, and
consistent behaviour/performance is achieved.
(From OE-Core rev: 440ad49e53359ea800c179df105ab885873d7691)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-tools SRCREV to import the following changes:
cbd4b7102668 patchme/updateme: unify meta directory handling
b65075997152 configme: standalone operation
The change of note is [configme: standalone operation], which makes the
kernel configuration script free from dependencies on other parts of the
kern-tools.
With this change, we set the stage to extend kernel configuration
fragments and auditing to arbitrary trees.
(From OE-Core rev: 17f071dea78a08648eda71829c845104338510b9)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is possible that recipe specific tasks, or build processes drop
files into the kernel source directory. These files can cause problems
with the meta data detection in the kern-tools.
With this change, we have a single unified meta data detection routine,
that logs the result in a new file ".metadir", which subsequent scripts
can find, and use, thereby avoid repeating the same check many times.
We also enhance the check to look for a sentinel file in a proper meta
directory, to avoid false positives when an unexpected kernel process
leaves an uncommitted directory in the kernel dir.
[YOCTO: #7441]
(From OE-Core rev: 6b04ae2c0439b83c0445fd1b8cb9cba5cee6b9bc)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When fixing a kernel configuration warning, it is often necessary to
modify the kernel's meta-data and re-run the tools to update and
re-audit the config. This implies that the patch, config and audit
steps are run multiple times.
The tools had a bug that would incorrectly restore old meta-data
versus using updated configuration. Updating the kern-tools SRCREV
to fix the issue.
(From OE-Core rev: b903559daa847d2c56bf729fc5ca885113d0eecc)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the SRCREV for the following incremental improvement in patch
processing time:
kgit-meta: skip patches on non-leaf nodes
In a similar way as commit 0768d697 [kgit-meta: dont run kgit-s2q
for
non-leaf nodes], we can save even more processing time by not even
analysing and linking patches if we aren't on the leaf node of the
tree.
This early exit can save nearly 95% of the time required to "patch"
a tree when no changes are actually applied.
(From OE-Core rev: 148c78e0f5de2689de3ad9beaa9f6de618d87758)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After a linux-yocto style kernel is configured, a kernel configuration
audit is executed to detect common errors or issues with the config.
This output used to be visible, but was made less obvious to not alarm
users unnecessarily (since some configuration issues are acceptable).
There are some classes of configuration issue that are worth being
visible, and that is specified configuration values that do not make the
final .config. These dropped options can result in any number of runtime
failures, so flagging them at build time makes sense.
The visibility of auditing is controlled by KCONF_AUDIT_LEVEL:
0: no reporting
1: report options that are specified, but not in the final config
2: report options that are not hardware related, but set by a BSP
The default level is 1, with level 2 and above being for BSP development
only.
If these conditions are detected, warnings will be generated as follows:
WARNING: [kernel config]: specified values did not make it into the
kernel's final configuration:
Value requested for CONFIG_SND_PCSP not in final ".config"
Requested value: "CONFIG_SND_PCSP=y"
Actual value set: ""
or
WARNING: [kernel config]: BSP specified non-hw configuration:
CONFIG_BLOCK
CONFIG_CFG80211_WEXT
CONFIG_CORDIC
CONFIG_CRC8
CONFIG_EFIVAR_FS
CONFIG_EFI_PARTITION
CONFIG_NET
CONFIG_NETDEVICES
CONFIG_PARTITION_ADVANCED
CONFIG_WEXT_CORE
CONFIG_WEXT_PROC
CONFIG_WIRELESS
At this point thse are only a warnings, since there needs to be time for
layers and configuration fragments to be validated against this new
check.
[YOCTO: #6943]
(From OE-Core rev: ad4d59495194b37bc510e9891bd14c0a2ac30dba)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the SRCREV to import the following kern-tools patch:
kgit-meta: always clear series file on branch transitions
This was triggered by the patch optimization changes, that no longer
run do_patch if a leaf/final branch is not being processed.
Without this change, invalid patches, or already applied patches in
an existig series file will be re-used which leads to missing files,
or patch errors.
(From OE-Core rev: 762cf3beea5ff374e2ddf491e541f07129443af3)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the SRCREV for the following commits:
4822d22b65c2 kgit-meta: dont run kgit-s2q for non-leaf nodes
3e3de1b9cdec createme: remove meta branch checks
With these, we save 10 seconds on the average patch phase, and
significantly more if very long patch queues are used.
(From OE-Core rev: 04e600d933878f3d104cf734d437e6baffb983d8)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During patch processing a consolidated set of configs, patches and directives
is created under the kernel source tree being modified. During that processing,
absolutely paths are converted to relative. It has been found that if directories
are sufficiently similar, like so:
/path/to/my-linux
/path/to/my-linux-3.16
The processing will chop to much of some paths, resulting in invalid relative
directories (like -3.16 in the above example).
Importing the following two kern tools fixes for the issue:
23345b8846fe kgit: retain trailing / in directory processing
a8cf93a3bc94 kgit-s2q: move subject and diffstat mismatch to 'fuzzy' matching
[YOCTO: #6753]
(From OE-Core rev: 660c90458e8b4114e4a8deb920e44263e03a1ec6)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
From the kern-tools commit:
tools: allow meta directories that are not the same as the branch name
With this change it is now possible to have a meta branch with meta data
in a directory that is not the same name as the branch.
The changes to three parts of the build are required to discover the name
of the meta directory by relying on the fact that in a clean/proper build
the meta directory is the only untracked, top level directory in the build.
As such, we can restore a checkpoint and then examine the build directory
to determine the meta directory name .. avoiding any new variables to
indicate this to the scripts and build system.
(From OE-Core rev: 36823f7aff5c8e28900997c96a97c302947981b0)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to generate and support kernel trees with full history, we need
to modify the kernel tools
e914d570232a kgit-checkpoint: ensure that full meta-data artifacts are maintained
192be836d318 kgit-scc: allow meta-data history to be maintained
(From OE-Core rev: f2015ead17c875ae37a9ad496fdafef2b931f771)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A lot of our recipes had short one-line DESCRIPTION values and no
SUMMARY value set. In this case it's much better to just set SUMMARY
since DESCRIPTION is defaulted from SUMMARY anyway and then the SUMMARY
is at least useful. I also took the opportunity to fix up a lot of the
new SUMMARY values, making them concisely explain the function of the
recipe / package where possible.
(From OE-Core rev: b8feee3cf21f70ba4ec3b822d2f596d4fc02a292)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-tools SRCREV to import the following change:
kgit-s2q: always update ORIG_HEAD after applying changes
In situations where git am fails to apply patches, and git apply is used,
we must update ORIG_HEAD as well as HEAD. This is required, since if the
next patch in the queue also fails git am application, it will reset to
ORIG_HEAD before using git apply. If we haven't updated ORIG_HEAD, we'll
end up warping back to the top of the branch each time.
This problem can only be seen in very specific situations, in particular if
a generated BSP branches from qemuppc, and has a series of non git "am able"
patches. We fail, since all of the qemuppc patches are not applied due to
the branch head constantly being reset.
(From OE-Core rev: 5126ac0aeb3154d31769dc20a46b6b1a6b2e3d9b)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-toosl SRCREV to pick up the following fix:
previous versions of the kern-tools supported the ability to import a bare
patch, with no From: Subject: or other identifying fields that are typically
in a full commit.
The same type of commit with kgit-s2q will prompt for a author ID, just
as git-quilt-import does. In build system environment that leads to an
infinite loop and the commit is never pushed.
To fix this issue, we add an interactive flag (-i), that when passed the
prompt based behaviour is used. When it isn't passed (the default), the following
name and email will be used for the git author:
GIT_AUTHOR_NAME="invalid_git config"
GIT_AUTHOR_EMAIL="<unknown@unknown>"
And a bare/incomplete header patch will be applied.
[YOCTO #5100]
(From OE-Core rev: cb0d8f8b9c59b351d11eef9c4951c4ce5601acb8)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's not necessary to specify the protocol parameter when it's the
default protocol for the fetcher, e.g. the default protocol for
git fetcher it git, "protocol=git" isn't needed.
(From OE-Core rev: a2bab241c64428d5109c3c5ac5de4463fbad70c5)
Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previous changes to the kern-tools improved functionality to ensure that
as a series is considered, it is checked against the tree to confirm that
all patches are really applied.
There was a bug in the subject based detection, such that the first matching
patch was take, and not the last. This change ensures that we start from
the end of a series, not the start.
(From OE-Core rev: 6357657ec5b5687defaf1acdd94c1cf89aa06541)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-tools SRCREV to pick up the following fixes:
60a894e kgit-s2q: add proper commit ID handling for mixed am/apply usage
3b08257 kgit-s2q: delete pruning of path support.
c5868b4 kgit-s2q: Restore implicit exit status to "git apply" section
1bd00b9 kgit-scc: mask warnings from cleanup phase 5
bb75299 kgit-s2q: fix commit warp when running "git am --abort"
ef9571b kgit-scc: cleanup git rebase-apply dir
fdb7d21 kgit-scc: ensure treegen stops if a meta series fails
008987b config: add kconfig cleaning options
69ff569 kgit-s2q: strip blank lines and comments
e7b4540 kgit-init: disable garbage collection on a new tree
417eaed kgit-s2q: delete old LTSI patch dir finding code
21f2200 kgit-scc: better error checking on resume
ad5084c kern-tools: use .meta as meta data container
1deb5d8 kgit-meta: don't push patches without a series file
eb431a1 kgit-s2q: aid patch reject resolution via helper scripts
f859c40 kgit-s2q: only use patch annotations when explicitly asked
333ae18 kgit: speed patch application by batching patches
bf6991d kgit: teach tools about non-default meta dirs
bcfc712 kgit-s2q: usability improvements
cb28803 kgit-s2q: fix patch prefix stripping.
37f40e1 kgit-s2q: warn/exit with error if patch not in series
f4704d2 kgit-s2q: consistent rm usage
e11819c kgit-s2q: standardize on use of git mailinfo
36a5eda kgit: remove guilt dependency
c461a4f spp/scc: export mark commands to meta-series
5311162 updateme: ensure that generated features are only used once
4f7a263 kgit-checkpoint: clear .gitignore for meta branch
21ee6f2 updateme: enforce a matching machine
b08749d kgit-scc: remove -meta files after consruction
These are bug fixes, usability changes as well as the removal of the
guilt dependency. During the uprev of the guilt package, the amount of
circumvention of the typical guilt workflow and checks meant that using
it as a series -> branch manager was no longer appropriate. As a result
a new tools kgit-s2q (series 2 queue) was created based on git-quiltimport,
git am, and the LTSI tree generation scripts.
The result is better series to branch validation, faster application and
a simpler management model. This tool is backwards compatible with any
tree previously constructed with guilt. We are now "guilt free"
(From OE-Core rev: 983bff587b60fdd0244ad00f238df5ed50cc1e1a)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-tools SRCREV to pick up the following fix:
When a feature is passed to the kernel configuration scripts, and that
feature is a directory name, it is a shortcut for:
$DIR/$DIR.scc
This expansion is not commonly used, and should be avoided. But for the
purposes of backwards compatibility, updateme can expand the feature into
a .scc file before passing it to the next set of configuration scripts.
(From OE-Core rev: bec0a48c34695827d70cbbde7795b5a56fc13d56)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The separately packaged merge_config.sh in the kern-tools package was
missing upstream fixes, and in particular a change that ensures it is
dash compatible.
By grabbing that upstream commit and rebasing the existing patches on
top of the new baseline, we are up to date and working on systems
where /bin/sh is dash.
[YOCTO #4473]
(From OE-Core rev: 681bcd2783e100dd2882273f28f16ef118161e89)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-tools SRCREV with the following fix:
Updateme is responsible for updating an existing meta-series with new patches,
configs and tree manipulations. To do this, it first checks for an existing
board description and generates one if required. It then searches for features
and fragments to be applied for the tree.
There were two problems:
- A top level board description is detected via the presence of "define"
directives that indicate the board name, the arch and kernel type. The
test for define would match on patches or fragments with 'define' in their
name, and would incorrectly use that file as the top level board description.
This is fixed by ensuring that only defines at the start of a line, or preceded
by whitepace match.
- When searching for features that were indicated as 'addon' or 'optional', the
search would find, and apply, any feature with the passed name as substring
versus an exact match.
This is fixed by ensuring that the matched feature name is /<feature name>
versus <feature name>
(From OE-Core rev: 57ae1e412a35d827f84bf9b1f48747bf703f84b7)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the SCRCREV to pick up the following fix
updateme: use absolute path for generated BSP descriptions
When a custom BSP is used, a top level BSP is generated by the tools and fed
to the build system just as a user defined BSP would be located and
passed. The location of the generated file is placed in the top_tgt file,
which is used by subsequent stages. A relative path was being placed into
top_tgt, which binds the build to a particular directory structure and
working directory.
The location of parts of the build have changed, and this relative path is
no longer accurate. Changing it to an absolute path solve the build issues
related to custom BSPs.
(From OE-Core rev: 2d7b2478a3d48a5686afde790c378ee2f69b8e59)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A regression was introduced when implementing the ability to restrict
configuration values via include directives. Only patch and config files that
were local to a feature directory could be found. While this doesn't impact
most users of the tools, it is an issue that needs to be fixed.
Additionally, the regex that detected flags passed to includes was not
specific enough, and unfortunately named feature files would match. This
resulted in features like standard-nocfg.scc inhibiting all configuration
items, even base configs.
This change also bumps the linux-yocto 3.4 and 3.8 PR values to ensure
that kernels will be rebuilt once this change is active.
(From OE-Core rev: ddce9f375c626ef2c86f48612b3d7a24e3111b0b)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Variables defined in .scc files have two purposes:
- Documentation in the meta-series
- Variables that can be tested in sub sections and other features
The second part of this functionality was broken when fixing configuration
for tiny/small systems. As a result, arch tests were failing and configs were
dropped. This restores the existing functionality.
(From OE-Core rev: 4170e458e0f700319f4e1023c0c6c2d803449566)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One of the features introduced early on in the 1.4 release cycle was the
ability to include a kernel feature, but only get its patches and not configs
(and vice versa).
As it turns out, this only was exercised recently and once a single include
with dropped configs was started, ALL configuration values following the
commit were dropped.
To fix the problem, the processing of kernel features has been split into
two. Where the features are preprocessed and the assembled/complete file is
used to generate the meta-series (which is later applied to the tree). The
logic of the tools is the same, but the two phases of processing allows
configuration values to be excluded properly and simply, while keeping the
logic for modifying the tree in a separate step.
All changes are invisible to the user, and are done within the existing
scripts and build system bindings. Output series and manipulations to
the tree are the same as they were before this change.
Updating the kern-tools SRCREV to pickup the kern-tools changes for this.
(From OE-Core rev: 961ab0ac53de317c22409d90244a313998959714)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bumping the kern-tools SRCREV to pickup the following fixes and documentation
updates:
d484e3f kgit-meta: remove hardcoded meta directory name
affad20 yocto-kernel-tools: Typoes, "fragement", "depreciated"
142ed49 kgit-init: update tools list
(From OE-Core rev: 65113af811afcf53d3056d372861cd4d1a6bff07)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the kern-tools SRCREV to import the following fixes:
commit 7f91d198d32fc90260e52724ef4aac0b997c1e8b
kconf_check: fix new Kconfig detection
One of the functions of the kernel configuration audit is to notify
the user if Kconfig* files have been removed from the kernel, and
also to notify of new Kconfig files.
New Kconfig files should be classified as hardware or non-hardware to
allow BSP audits to notify if boards are setting values that they
shouldn't, hence why notifying about new "buckets" is important.
commit c4f26a3296e0e1c3dbdd5ec8e2947d5443a9ffc2
updateme/scc: allow config fragment exclusion
It is common to need the features (patches, git operations) of a
branch, but not want the kernel configuration fragments of a given
branch. To allow this, we provide a new include flag "nocfg".
When this flag is used, all of the configuration fragments included
by the targetted feature will not be applied to the current build,
with one exception, a base/critical fragment can force it's config
values, since without them, the system would not be functional.
Example:
include ktypes/standard/standard.scc nocfg
commit c7ec19d55aca6c4b17073c5362fce5be61a89d82
scc: wrap git merge
To allow for parameter validation and sanity checking, wrap "git merge"
as a dedicated "merge" command instead of using the raw git fallback.
This also makes it consistent with existing top level commands such
as 'tag', 'branch', 'patch', etc.
There are no changes to arguments, and existing 'git merge' commands
continue to work with this change.
[YOCTO #3419]
[YOCTO #3421]
(From OE-Core rev: faf042b2c87874153a6b689479ab86e49804af8c)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To promote the reuse and sharing of configuration fragments this change
allows any kernel-yocto based recipe to have multiple alternate git repositories
which provide kernel feature directory trees listed on the SRC_URI.
These feature directories are in addition to any in-tree kernel meta data branches
that may be available (described via the KMETA variable in linux-yocto recipes).
Features found within these directories can be used from recipes via the
KERNEL_FEATURES variable. Features found within a feature directory are free
to include any other features that are available in any directories. In both
cases the path to a feature description (a .scc file) is relative to the
root of a given feature directory (which is how existing .scc files work)
The search order for features is determined by the order that repositories
appear on the SRC_URI.
Normal SRC_URI rules apply to any repository that is added as a kernel
feature container. A SRCREV must be supplied and it must be unpacked to
a unique directory, which is controlled via the "destsuffic" url parameter.
In addition to these standard requirements, any kernel feature repository
reference should identify itself via the "type=kmeta" url parameter. If
type=kmeta is not supplied, the repository will not be processed for
kernel features.
As an example, the following in a linux-yocto bbappend makes two additional
feature directories available to KERNEL_FEATURES and fragments.
SRC_URI += "git://git.yoctoproject.org/yocto-kernel-cache;protocol=git;branch=master;type=kmeta;name=feat1;destsuffix=kernel-cache/"
SRC_URI += "git://${KSRC_linux_yocto_3_4};protocol=file;branch=meta;name=feat2;type=kmeta;destsuffix=kernel-features-experimental/"
SRCREV_feat1 = "${AUTOREV}"
SRCREV_feat2 = "${AUTOREV}"
(From OE-Core rev: 02ad603a104b70ab74548c8018e738bfbb3c59db)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a configuration fragment was missing, the previous error output
was not clear about the error:
| [INFO] doing kernel configme
| [INFO] Configuring target/machine combo: "standard/atom-pc"
| [INFO] collecting configs in ./meta/meta-series
| ERROR: could not sanitize configuration fragments
| errors are logged in ... linux/meta/cfg/standard/atom-pc/config.log
but we know the name of the missing fragment and can improve the error
message to be this:
| [ERROR] kernel configuration fragment fragment 'virto.cfg' cannot be found
| ERROR. A meta series could not be created for branch yocto/standard/common-pc/atom-pc
| ERROR. Could not locate meta series for atom-pc
| ERROR. Could not apply patches for atom-pc.
| Patch failures can be resolved in the devshell (bitbake -c devshell linux-yocto)
[YOCTO #3473]
(From OE-Core rev: 8e5dc511ffce4f4b512457dbc5d3dd510f6e4a95)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 46cc0d0a2f1486bf541c1a1b11075de3da396cc2 since
the revision in question isn't in the repository.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a configuration fragment was missing, the previous error output
was not clear about the error:
| [INFO] doing kernel configme
| [INFO] Configuring target/machine combo: "standard/atom-pc"
| [INFO] collecting configs in ./meta/meta-series
| ERROR: could not sanitize configuration fragments
| errors are logged in ... linux/meta/cfg/standard/atom-pc/config.log
but we know the name of the missing fragment and can improve the error
message to be this:
| [ERROR] kernel configuration fragment fragment 'virto.cfg' cannot be found
| ERROR. A meta series could not be created for branch yocto/standard/common-pc/atom-pc
| ERROR. Could not locate meta series for atom-pc
| ERROR. Could not apply patches for atom-pc.
| Patch failures can be resolved in the devshell (bitbake -c devshell linux-yocto)
[YOCTO #3473]
(From OE-Core rev: 46cc0d0a2f1486bf541c1a1b11075de3da396cc2)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the SRCREV to import the following changes.
[updateme: find the board description with the highest score]
This removes the requirement that a custom linux-yocto .scc file have
define KTYPE <foo>, where <foo> is typically "standard". The tools can
now match on a .scc file that only matches the board, but will still
chose one that matches the board and kernel type, if available.
[updateme: allow for tabs or spaces in defines]
define KMACHINE<tab>$MACHINE was missed by the regex.
[scc/kgit-meta: detect and avoid duplicating patching]
To allow feature description to be included multiple times, they were
previously split into -enable and 'patch' descriptions. With this change
the patches will be detected as already included, and skipped
automatically. Removing the need to do this split. It also cleans up
the ability to warn about multiple includes.
[kconf_check: add "verify" configuration fragment type]
This adds the ability for a BSP to have a kernel configuration
fragment that lists options that must be present. If they are not
present it is a hard error. "required" is a similar fragment, but
it adds them to the build, and audits them at the end, but does
not abort the build if they are present. This is a minor distinction,
but one that is useful when creating flexible, shared kernel config
structures.
[kconf_check: improve kernel audit report formatting]
[kconf_check: perform validity checks on non-hardware options]
[kconf_check: cleanups and verbose flag]
The existing output was verbose and not always useful to the reader.
This change makes the output more compact, audits non-hardware options
and gives information
[invalid (54)]: meta/cfg/preempt-rt/common-pc/invalid.cfg
This BSP sets config options that are not offered anywhere within this kernel
(From OE-Core rev: 2d328dc0f7dd763c45444394b681d2726b4f6c83)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bumping the kern-tools SRCREV to pickup the following change:
[
kconf_check: fix find warning
When searching for all available Kconfig files, kconf_check was using
$meta_dir instead of $META_DIR. This resulted in a truncated path and
the following warning:
find: warning: -path $oe-path/linux/ will not match anything because it ends with /.
Using the proper variable removes the warning and make sure that we
do actually search all relevant directories.
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
]
[YOCTO #3226]
(From OE-Core rev: 5999ccebc7b071737f82709467e2a2ec152240f6)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was reported that the kernel configuration checks for custom yocto
kernels had the following output:
NOTE: validating kernel configuration
grep: /meta-series: No such file or directory
grep: /meta-series: No such file or directory
WARNING: Can't find any BSP hardware or required configuration fragments.
WARNING: Looked at //cfg///hdw_frags.txt and //cfg///required_frags.txt in directory: //cfg//
NOTE: Tasks Summary: Attempted 375 tasks of which 367 didn't need to be rerun and all succeeded.
which is not inspire confidence in the output of the process.
Completely inhibiting the check is one option to remove the messages,
but that removes the ability see output, which can help move users to
a better or more fully configured linux-yocto based kernel.
To fix this, we have to ensure that the path to the meta-series is
always valid, and that the tools can deal with not all files existing
in the audit directory.
Since custom yocto kernels do not set KMETA (they don't have a meta branch),
we ensure that a default of 'meta' is passed to the audit ('meta' is always
valid), and that kconf_check itself can deal with an incomplete set of
input audit files.
The net result is output like this (using a defconfig with invalid options
for the kernel being built):
NOTE: validating kernel configuration
This BSP sets 19 invalid/obsolete kernel options.
These config options are not offered anywhere within this kernel.
The full list can be found in your kernel src dir at:
meta/cfg/standard/qemux86/invalid.cfg
There were 1 instances of config fragment errors.
The full list can be found in your kernel src dir at:
meta/cfg/standard/qemux86/fragment_errors.txt
The full list can be found in your kernel src dir at:
meta/cfg/standard/qemux86/missing_required.cfg
(From OE-Core rev: 4d1b7dae063ee4c35c426306d0e22f11ce112c72)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Importing the following tools SRCREV:
kgit-meta: exclude explicit branches from name calculations
kernel branches are constructed during patching of the tree by
constructing a '/' based hierarchy of names as each branch
directive is encountered.
But if a "branch $name $branchpoint" is used, the entire branch
name is supplied so no additions to the hierarchy should
happen. As such, that type of branch command should not be part
of branch name calculation and preparation.
(From OE-Core rev: a3884938233c8a2d6861b1d4e6be5b9824d3b131)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When patches fail to apply, the status of all pending patches should
be exported to the logs and to the user. Currently, a missing export
of GUILT_BASE makes it look more like an internal error, than a 'normal'
patch failure:
| [ERROR] unable to complete push
| pending patches are:
| Patches directory doesn't exist, try guilt-init
With this variable exported, we have this:
| [INFO] validating against known patches (qemux86-standard-meta)
| error: patch failed: Makefile:2
| error: Makefile: patch does not apply
| To force apply this patch, use 'guilt push -f'
| [ERROR] unable to complete push
| pending patches are:
| links/files/0002-makefile-patch.patch
(From OE-Core rev: b2b2512cbc4196fa0f814be3677517dab30e5b52)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
commit 7a79f7412 [linux-yocto: make KBRANCH the exception and not the rule]
ensures that a request branch is always built. The implementation of this
guarantee is a branch switch before the build starts. But that switch may
be before all patches are applied. If the proper routines are not called,
no patches can be applied to the tree.
Updating the SRCREV to pickup this fix:
updateme: use branch command when forcing branch switches
When forcing a branch switch to the desired branch we should be
using the proper 'branch' command. Since without this call, the
proper variables will not be set, and patches can't be applied
to the tree.
(From OE-Core rev: bede3086cb9ab0f79fb7775528b646817b2b9af0)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To support custom repositories that set a SRCREV and that only have
a single master branch, do_validate_branches needs a special case
for 'master'. We can't delete and recreate the branch, since you
cannot delete the current branch, instead we must reset the branch
to the proper SRCREV.
(From OE-Core rev: de5bb5879fa3282c46dc1ede36af34eaab8f647f)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bruce Ashfield
Awais Belal
Paul Eggleton
Jackie Huang
Richard Purdie