This chapter describes how to accomplish tasks involving the kernel's tree structure. The information covers the following: Tree construction Build strategies Workflow examples

The Yocto Project kernel repository, as shipped with the product, is created by compiling and executing the set of feature descriptions for every BSP/feature in the product. Those feature descriptions list all necessary patches, configuration, branching, tagging and feature divisions found in the kernel. You can find the files used to describe all the valid features and BSPs in the Yocto Project kernel in any clone of the kernel git tree. The directory wrs/cfg/kernel-cache/ is a snapshot of all the kernel configuration and feature descriptions (.scc) used to build the kernel repository. You should realize, however, that browsing the snapshot of feature descriptions and patches is not an effective way to determine what is in a particular kernel branch. Instead, you should use git directly to discover the changes in a branch. Using git is a efficient and flexible way to inspect changes to the kernel. For examples showing how to use git to inspect kernel commits, see the following sections in this chapter. Ground up reconstruction of the complete kernel tree is an action only taken by the Yocto Project team during an active development cycle. Creating a project takes advantage of this complete tree in order to place efficiently a git tree within the project. The general flow for constructing a project-specific kernel tree is as follows: A top-level kernel feature is passed to the kernel build subsystem. Normally, this is a BSP for a particular kernel type. The file that describes the top-level feature is located by searching these system directories: The kernel-cache under linux/wrs/cfg/kernel-cache Recipe SRC_URIs For a typical build a feature description of the format: <bsp name>-<kernel type>.scc is the target of the search. Once located, the feature description is either compiled into a simple script of actions, or an existing equivalent script that was part of the shipped kernel is located. Extra features are appended to the top-level feature description. These features can come from the KERNEL_FEATURES variable in recipes. Each extra feature is located, compiled and appended to the script from step #3 The script is executed, and a meta-series is produced. The meta-series is a description of all the branches, tags, patches and configuration that needs to be applied to the base git repository to completely create the "bsp_name-kernel_type". The base repository is cloned, and the actions listed in the meta-series are applied to the tree. The git repository is left with the desired branch checked out and any required branching, patching and tagging has been performed. The tree is now ready for configuration and compilation. The end-user generated meta-series adds to the kernel as shipped with the Yocto Project release. Any add-ons and configuration data are applied to the end of an existing branch. The full repository generation that is found in the linux-2.6-windriver.git is the combination of all supported boards and configurations. This technique is flexible and allows the seamless blending of an immutable history with additional deployment specific patches. Any additions to the kernel become an integrated part of the branches.

There are some prerequisites that must be met before starting the compilation phase of the kernel build system: There must be a kernel git repository indicated in the SRC_URI. There must be a branch <bsp name>-<kernel type>. You can typically meet these prerequisites by running the tree construction/patching phase of the build system. However, other means do exist. For examples of alternate workflows such as bootstrapping a BSP, see the Workflow Examples section in this manual. Before building a kernel it is configured by processing all of the configuration "fragments" specified by the scc feature descriptions. As the features are compiled, associated kernel configuration fragments are noted and recorded in the meta-series in their compilation order. The fragments are migrated, pre-processed and passed to the Linux Kernel Configuration subsystem (lkc) as raw input in the form of a .config file. The lkc uses its own internal dependency constraints to do the final processing of that information and generates the final .config file that is used during compilation. Using the board's architecture and other relevant values from the board's template the Kernel compilation is started and a kernel image is produced. The other thing that you will first see once you configure a kernel is that it will generate a build tree that is separate from your git source tree. This build tree has the name using the following form: linux-<BSPname>-<kerntype>-build "kerntype" is one of the standard kernel types. The existing support in the kernel.org tree achieves this default functionality. What this means, is that all the generated files for a particular BSP are now in this directory. The files include the final .config, all the .o files, the .a files, and so forth. Since each BSP has its own separate build directory in its own separate branch of the git tree you can easily switch between different BSP builds.

As previously noted, the Yocto Project kernel has built in git/guilt integration. However, these utilities are not the only way to work with the kernel repository. Yocto Project has not made changes to git or to other tools that would invalidate alternate workflows. Additionally, the way the kernel repository is constructed results in using only core git functionality thus allowing any number of tools or front ends to use the resulting tree. This section contains several workflow examples.

A common question when working with a BSP or kernel is: "What changes have been applied to this tree?" In projects that have a collection of directories that contain patches to the kernel it is possible to inspect or "grep" the contents of the directories to get a general feel for the changes. This sort of patch inspection is not an efficient way to determine what has been done to the kernel. The reason it is inefficient is because there are many optional patches that are selected based on the kernel type and the feature description. Additionally, patches could exist in directories that are not included in the search. A more efficient way to determine what has changed in the kernel is to use git and inspect or search the kernel tree. This method gives you a full view of not only the source code modifications, but also provides the reasons for the changes.

Following are a few examples that show how to use git to examine changes. Note that because the Yocto Project git repository does not break existing git functionality and because there exists many permutations of these types of commands there are many more methods to discover changes. Unless you provide a commit range (<kernel-type>..<bsp>-<kernel-type>), kernel.org history is blended with Yocto Project changes. # full description of the changes > git whatchanged <kernel type>..<bsp>-<kernel type> > eg: git whatchanged standard..common_pc-standard # summary of the changes > git log --pretty=oneline --;abbrev-commit <kernel type>..<bsp>-<kernel type> # source code changes (one combined diff) > git diff <kernel type>..<bsp>-<kernel type> > git show <kernel type>..<bsp>-<kernel type> # dump individual patches per commit > git format-patch -o <dir> <kernel type>..<bsp>-<kernel type> # determine the change history of a particular file > git whatchanged <path to file> # determine the commits which touch each line in a file > git blame <path to file>

Significant features or branches are tagged in the Yocto Project tree to divide changes. Remember to first determine (or add) the tag of interest. Because BSP branch, kernel.org, and feature tags are all present, there are many tags. # show the changes tagged by a feature > git show <tag> > eg: git show yaffs2 # determine which branches contain a feature > git branch --contains <tag> # show the changes in a kernel type > git whatchanged wrs_base..<kernel type> > eg: git whatchanged wrs_base..standard You can use many other comparisons to isolate BSP changes. For example, you can compare against kernel.org tags (e.g. v2.6.27.18, etc), or you can compare agains subsystems (e.g. git whatchanged mm).

Another common operation is to build a BSP supplied by Yocto Project, make some changes, rebuild and then test. Those local changes often need to be exported, shared or otherwise maintained. Since the Yocto Project kernel source tree is backed by git, this activity is much easier as compared to with previous releases. Because git tracks file modifications, additions and deletions, it is easy to modify the code and later realize that the changes should be saved. It is also easy to determine what has changed. This method also provides many tools to commit, undo and export those modifications. There are many ways to save kernel modifications. The technique employed depends on the destination for the patches: Bulk storage Internal sharing either through patches or by using git External submissions Exporting for integration into another SCM Because of the following list of issues, the destination of the patches also influences the method for gathering them: Bisectability Commit headers Division of subsystems for separate submission or review

This section describes how you can export in "bulk" changes that have not been separated or divided. This situation works well when you are simply storing patches outside of the kernel source repository, either permanently or temporarily, and you are not committing incremental changes during development. This technique is not appropriate for full integration of upstream submission because changes are not properly divided and do not provide an avenue for per-change commit messages. Therefore, this example assumes that changes have not been committed incrementally during development and that you simply must gather and export them. # bulk export of ALL modifications without separation or division # of the changes > git add . > git commit -s -a -m >commit message< or > git commit -s -a # and interact with $EDITOR The previous operations capture all the local changes in the project source tree in a single git commit. And, that commit is also stored in the project's source tree. Once the changes are exported, you can restore them manually using a template or through integration with the default_kernel.

This section describes how to save modifications when you are making incremental commits or practicing planned sharing. The examples in this section assume that changes have been incrementally committed to the tree during development and now need to be exported. The sections that follow describe how you can export your changes internally through either patches or by using git commands. During development the following commands are of interest. For full git documentation, refer to the git man pages or to an online resource such as . # edit a file > vi >path</file # stage the change > git add >path</file # commit the change > git commit -s # remove a file > git rm >path</file # commit the change > git commit -s ... etc. Distributed development with git is possible when you use a universally agreed-upon unique commit identifier (set by the creator of the commit) that maps to a specific changeset with a specific parent. This identifier is created for you when you create a commit, and is re-created when you amend, alter or re-apply a commit. As an individual in isolation, this is of no interest. However, if you intend to share your tree with normal git push and pull operations for distributed development, you should consider the ramifications of changing a commit that you have already shared with others. Assuming that the changes have not been pushed upstream, or pulled into another repository, you can update both the commit content and commit messages associated with development by using the following commands: > git add >path</file > git commit --amend > git rebase or git rebase -i Again, assuming that the changes have not been pushed upstream, and that no pending works-in-progress exist (use "git status" to check) then you can revert (undo) commits by using the following commands: # remove the commit, update working tree and remove all # traces of the change > git reset --hard HEAD^ # remove the commit, but leave the files changed and staged for re-commit > git reset --soft HEAD^ # remove the commit, leave file change, but not staged for commit > git reset --mixed HEAD^ You can create branches, "cherry-pick" changes or perform any number of git operations until the commits are in good order for pushing upstream or for pull requests. After a push or pull, commits are normally considered "permanent" and you should not modify them. If they need to be changed you can incrementally do so with new commits. These practices follow the standard "git" workflow and the kernel.org best practices, which Yocto Project recommends. It is recommend to tag or branch before adding changes to a Yocto Project BSP or before creating a new one. The reason for this recommendation is because the branch or tag provides a reference point to facilitate locating and exporting local changes.

This section describes how you can extract committed changes from a working directory by exporting them as patches. Once extracted, you can use the patches for upstream submission, place them in a Yocto Project template for automatic kernel patching, or apply them in many other common uses. This example shows how to create a directory with sequentially numbered patches. Once the directory is created, you can apply it to a repository using the git am command to reproduce the original commit and all the related information such as author, date, commit log, and so forth. The new commit identifiers (ID) will be generated upon re-application. This action reflects that the commit is now applied to an underlying commit with a different ID. # <first-commit> can be a tag if one was created before development # began. It can also be the parent branch if a branch was created # before development began. > git format-patch -o <dir> <first commit>..<last commit> In other words: # Identify commits of interest. # If the tree was tagged before development > git format-patch -o <save dir> <tag> # If no tags are available > git format-patch -o <save dir> HEAD^ # last commit > git format-patch -o <save dir> HEAD^^ # last 2 commits > git whatchanged # identify last commit > git format-patch -o <save dir> <commit id> > git format-patch -o <save dir> <rev-list>

This section describes how you can export changes from a working directory by pushing the changes into a master repository or by making a pull request. Once you have pushed the changes in the master repository you can then pull those same changes into a new kernel build at a later time. Use this command form to push the changes: git push ssh://<master server>/<path to repo> <local branch>:<remote branch> For example, the following command pushes the changes from your local branch common_pc-standard to the remote branch with the same name in the master repository //git.mycompany.com/pub/git/kernel-2.6.27. > push ssh://git.mycompany.com/pub/git/kernel-2.6.27 common_pc-standard:common_pc-standard A pull request entails using "git request-pull" to compose an email to the maintainer requesting that a branch be pulled into the master repository, see for an example. Other commands such as 'git stash' or branching can also be used to save changes, but are not covered in this document.

If patches are to be sent for external submission, they can be done via a pull request if the patch series is large or the maintainer prefers to pull changes. But commonly, patches are sent as email series for easy review and integration. Before sending patches for review ensure that you understand the standard of the community in question and follow their best practices. For example, kernel patches should follow standards such as: Documentation/SubmittingPatches (in any linux kernel source tree) The messages used to commit changes are a large part of these standards, so ensure that the headers for each commit have the required information. If the initial commits were not properly documented or don't meet those standards rebasing via git rebase -i offer an opportunity to manipulate the commits and get them into the required format. Other techniques such as branching and cherry picking commits are also viable options. Once complete, patches are sent via email to the maintainer(s) or lists that review and integrate changes. "git send-email" is commonly used to ensure that patches are properly formatted for easy application and avoid mailer induced patch damage. An example of dumping patches for external submission follows: # dump the last 4 commits > git format-patch --thread -n -o ~/rr/ HEAD^^^^ > git send-email --compose --subject '[RFC 0/N] <patch series summary>' \ --to foo@yoctoproject.org --to bar@yoctoproject.org \ --cc list@yoctoproject.org ~/rr # the editor is invoked for the 0/N patch, and when complete the entire # series is sent via email for review

Using any one of the previously discussed techniques, commits can be exported as patches for import into another SCM. Note however, that if those patches are manually applied to a secondary tree and then that secondary tree is checked into the SCM, then it often results in lost information (like commit logs) and so it is not recommended. Many SCMs can directly import git commits, or can translate git patches to not lose information. Those facilities are SCM dependent and should be used whenever possible.

This is not the same as the exporting of patches to another SCM, but instead is concerned with kernel development that is done completely in another environment, but built with the Yocto Project build system. In this scenario two things must happen: The delivered Yocto Project kernel must be exported into the second SCM. Development must be exported from that secondary SCM into a format that can be used by the Yocto Project build system.

Depending on the SCM it may be possible to export the entire Yocto Project kernel git repository, branches and all, into a new environment. This is the preferred method, since it has the most flexibility and potential to maintain the meta data associated with each commit. When a direct import mechanism is not available, it is still possible to export a branch (or series of branches) and check them into a new repository. The following commands illustrate some of the steps that could be used to import the common_pc-standard kernel into a secondary SCM > git checkout common_pc-standard > cd .. ; echo linux/.git > .cvsignore > cvs import -m "initial import" linux MY_COMPANY start The CVS repo could now be relocated and used in a centralized manner. The following commands illustrate how two BSPs could be condensed and merged into a second SCM: > git checkout common_pc-standard > git merge cav_ebt5800-standard # resolve any conflicts and commit them > cd .. ; echo linux/.git > .cvsignore > cvs import -m "initial import" linux MY_COMPANY start

Once development has reached a suitable point in the second development environment, changes can either be exported as patches or imported into git directly (if a conversion/import mechanism is available for the SCM). If changes are exported as patches, they can be placed in a recipe and automatically applied to the kernel during patching.

This section provides an example for creating a BSP based on an existing, and hopefully, similar one. Follow these steps and keep in mind your particular situation and differences: Get a machine configuration file that matches your machine. You can start with something in meta/conf/machine. Or, meta-emenlow/conf/machine has an example in its own layer. The most up-to-date machines that are probably most similar to yours and that you might want to look at are meta/conf/machine/atom-pc.conf and meta-emenlow/conf/machine/emenlow.conf. Both of these were either just added or upgraded to use the Yocto Project kernel at . The main difference between them is that "emenlow" is in its own layer. It is in its own layer because it needs extra machine-specific packages such as its own video driver and other supporting packages. The "atom-pc" is simpler and does not need any special packages - everything it needs can be specified in the configuration file. The "atom-pc" machine also supports all of Asus eee901, Acer Aspire One, Toshiba NB305, and the Intel® Embedded Development Board 1-N450 with no changes. If you want to make minor changes to support a slightly different machine, you can create a new configuration file for it and add it alongside the others. You might consider keeping the common stuff separate and including it. Similarly, you can also use multiple configuration files for different machines even if you do it as a separate layer like meta-emenlow. As an example consider this: Copy meta-emenlow Fix or remove anything you do not need. For this example the only thing left was the kernel directory with a linux-yocto_git.bbappend file (linux-yocto is the kernel listed in meta-crownbay/conf/machine/crownbay.conf. Finally, a new entry to the build/donf/bblayers.conf was added so the new layer could be found by Bitbake. Get an image with a working kernel built. For the kernel to compile successfully, you need to create a branch in the git repository specifically named for your machine. So first create a bare clone of the Yocto Project git repository, and then create a local clone of that: $ git clone --bare git://git.pokylinux.org/linux-2.6-windriver.git linux-2.6-windriver.git $ git clone linux-2.6-windriver.git linux-2.6-windriver Now create a branch in the local clone and push it to the bare clone: $ git checkout -b crownbay-standard origin/standard $ git push origin crownbay-standard:crownbay-standard At this point, your git tree should be set up well enough to compile. Point the build at the new kernel git tree. You can do this by commenting out the SRC_URI variable in meta/recipes-kernel/linux/linux-yocto_git.bb and using a SRC_URI that points to your new bare git tree. You should also be able to do this in linux-yocto_git.bbappend in the layer: # To use a staged, on-disk bare clone of a Wind River Kernel, use a variant of the # below SRC_URI = "git://///path/to/kernel/default_kernel.git;fullclone=1" # SRC_URI = "git://git.pokylinux.org/linux-2.6-windriver.git;protocol=git;fullclone=1;branch=${KBRANCH};name=machine \ git://git.pokylinux.org/linux-2.6-windriver.git;protocol=git;noclone=1;branch=wrs_meta;name=meta" After doing that, select the machine in build/conf/local.conf: # MACHINE ?= "crownbay" # You should now be able to build and boot an image with the new kernel: $ bitbake poky-image-sato-live Of course, that will give you a kernel with the default config, which is probably not what you want. If you just want to set some kernel config options, you can do that by putting them in a files. For example inserting the following into some .cfg file: CONFIG_NETDEV_1000=y CONFIG_E1000E=y And, another .cfg file would contain: CONFIG_LOG_BUF_SHIFT=18 http://git.pokylinux.org/cgit/cgit.cgi/linux-2.6-windriver/ SRC_URI_append_crownbay = " file://some.cfg \ file://other.cfg \ " You could also add these directly to the git repo's wrs_meta branch as well. However, the former method is probably easier. If you're also adding patches to the kernel, you can do the same thing. Put your patches in the SRC_URI as well (plus .cfg for their kernel config options if needed). Practically speaking, to generate the patches, you'd go to the source in the build tree: build/tmp/work/crownbay-poky-linux/linux-yocto-2.6.34+git0+d1cd5c80ee97e81e130be8c3de3965b770f320d6_0+ 0431115c9d720fee5bb105f6a7411efb4f851d26-r13/linux Then, modify the code there, using quilt to save the changes, and recompile (bitbake -c compile -f) until it works. Once you have the final patch from quilt, copy it to the SRC_URI location, and it should be applied the next time you do a clean build. Of course, since you have a branch for the BSP in git, it would be better to put it there instead. For example, in this case, commit the patch to the crownbay-standard branch, and during the next build it will be applied from there.

If kernel images are being built with -dirty on the end of the version string, this simply means that there are modification in the source directory that haven't been committed. > git status The above git command will indicate modified, removed or added files. Those changes should be committed to the tree (even if they will never be saved, or exported for future use) and the kernel rebuilt. To brute force pickup and commit all such pending changes enter the following: > git add . > git commit -s -a -m "getting rid of -dirty" And then rebuild the kernel

In order to temporarily use a different base kernel in Yocto Project Linux 3.0 you need to do the following: Create a custom kernel layer. Create a git repository of the transition kernel. Once those requirements are met multiple boards and kernels can be built. The cost of setup is only paid once and then additional BSPs and options can be added. This creates a transition kernel layer to evaluate functionality of some other kernel with the goal of easing transition to an integrated and validated Yocto Project kernel.