------------------------------ My poor Portage deconstruction ------------------------------ Here's what I have so far in my "walk" through the Portage code (specifically, portage HEAD sometime around Feb. 2005). Right now I've enumerated ebuild_, the effects of `importing portage`_, emerge_, the Portage `portage.doebuild()`_ function, and the `ebuild.sh`_ script. ebuild ------ The ebuild script is a simple interface to portage.py. It's a rather simple python script: * Import needed modules * Add ``/usr/lib/portage/pym`` to the python path so that portage can be found. * Define the ``getroot()`` function to ensure that ``ROOT`` is set correctly. * Set the ``PORTAGE_CALLER`` environment variable to ``ebuild``. * Parse options using getopt: * Look for ``--debug``, set ``debug=1`` if found * Look for ``merge``, add ``-noauto`` to ``FEATURES`` if found. * Import portage (and doing all of the setup that entails). :: Example: ebuild --debug /path/to/foo.ebuild fetch unpack Script loops over arguments after the ebuild name: - store (clone) portage.config to be passed to doebuild - Set ``cleanup=1`` if running either ``clean`` or ``config`` - execute ``a=portage.doebuild(ebuild name, argument, ROOT, config, debug=debug,cleanup=cleanup)`` - fail (exit) if ``a`` exists (or user interrupt or I/O error) importing portage ----------------- Quite a few things happen when portage is imported. Most of portage.py goes about defining a boatload of classes and functions, but there's also a fair amount of information that is assembled and made available by the import process. Here's a (probably incomplete) list of those items: * Define constants: portage version number, a list of variables that are "incremental" (USE, FEATURES, etc.), and a list of variables that are "sticky" (KEYWORDS_ACCEPT, USE, ...). * Import a lot of python modules that will be used by portage. Core python modules are imported first, followed by various portage modules. Along the way, ``catpkgsplit``, ``pkgsplit``, and ``pkgcmp`` from ``portage_versions`` are added to the portage namespace. * Establish a signal handler to catch SIGCHLD, SIG_DFL, SIGINT, and SIGTERM. * Define a *lot* of functions and classes. * Define the "VDB" path (traditionally ``/var/db/pkg``) * Identify ostype from uname, set some os-related defaults, export USERLAND to the environment * Check to make sure user is in the portage group (or root) * List of "incrementals" and "stickies" keys * List possible ebuild version suffixes ("pre", "alpha", ...) * List ebuild keys ("DEPEND", "LICENSE", ...) * Check environment for ROOT, use "/" if not found * Create tmp and var/tmp directories (below ROOT) * Set 022 umask * Check to see if the current profile is deprecated * Set some global settings (PID, bash env, cache dir) * Read use.defaults and set variables appropriately * Get OVERLAY * Lock global settings * Check for selinux * Set up some cachedir stuff * mtime db stuff??? * Get FEATURES * Set up porttree and bintree dbs * Get thirdpartymirrors * Perform PORTAGE_CACHEDIR and PORTAGE_TMPDIR sanity checks * Get category list from file * Get package.mask, /etc/portage/package.* * Get packages file * Get unmask list, mask list, "revmask"? * Get ACCEPT_KEYWORDS * Check that /etc/make.profile is a symlink * Check for prelink emerge ------ This script is much more complicated than the ebuild script (portage.py is just a bit more than 2x as many lines as this script). * Set environment variable ``PORTAGE_CALLER="emerge"`` * Import portage <-- portage.settings and other vars now exposed * Check if output should be colored (if not, call ``nocolor()`` ). * Check portage.settings for ``PORTAGE_NICENESS``; if so, nice portage * Freeze portdb api (don't know what this does just yet) * Turn off "noauto" if set in ``FEATURES`` * Set # of merged ebuilds = 0 * Set up lists of valid emerge parameters (params), actions, and options (all hard-coded lists); also map short-form of options to long-form versions. * Process command line (ugly!) - process short-form to convert to long-form - Process long-form arguments - Create list of options (myopts), actions (myactions), & files (myfiles) - minor fixes ( ``-U`` implies ``-u``, etcetera) - handle ``--noconfmem`` and ``--debug`` flags * Define log, exit, countdown, signal handling, help, and "emerge info" helper functions. * Check permissions; exit if invalid. * Start logging (unless ``--pretend`` was set) * Decide what type of dependencies to use (self, selective, recurse, deep, or empty) * Define spinner func * Define the search class * Build package digraph - Define getlist function for world or system - Define genericdict function - Define depgraph class (huge) * Define unmerge function * Define post_emerge function (finish logging, handle info stuff, config protection) * If --debug set, edebug=1 (not sure why two different debug flags set) * Handle ``emerge sync`` (big if, elif, else construction) * Handle ``emerge regen`` * Handle ``emerge config`` * Handle ``emerge info`` * Handle ``emerge search`` * Handle ``emerge inject`` * Handle ``emerge unmerge``, ``emerge prune``, and ``emerge clean`` - Call ``unmerge(myaction, myfiles)`` * Handle ``emerge depclean`` * Else: Handle update, system, or just process files - Do some setup stuff if ``--pretend`` set - If ``--resume`` and portage.mtimedb has a "resume" key, then update opts, create a "resume" depgraph - Else, create depgraph, error out if something is wrong - If ``--pretend``, show depgraph - Else + If ``--buildpkgonly``, check that deps are satisfied + If --resume, do merge using ``mydepgraph.merge(portage.mtimedb["resume"]["mergelist"])`` + Else, handle ``--digest`` or perform merge, the latter with ``mydepgraph.merge(mydepgraph.altlist())`` - Run autoclean - Run ``post_emerge()`` portage.doebuild() ------------------ Much of what happens during the actual merge of a package happens in portage's doebuild function. The interface is:: doebuild(myebuild, mydo, myroot, debug=0, listonly=0, fetchonly=0) where ``myebuild`` is the path to the ebuild to be merged, ``mydo`` is the action to be performed (one of the commands that follows ``ebuild foo.ebuild ...``), and ``debug``, ``listonly``, and ``fetchonly`` are flags that can be true (1) or false (0). Here's what happens: * Check to make sure the input is valid * Set a boatload of keys in the ``settings`` instance of the config class - PORTAGE_DEBUG, ROOT, STARTDIR, EBUILD, O, CATEGORY, FILESDIR, PF, ECLASSDIR, SANDBOX_LOG, P, PN, PV, PR, PVR, SLOT, BUILD_PREFIX, PKG_TMPDIR, PORTAGE_BUILDDIR, KV, KVERS * If the action is ``depend``, then return dependency info (eclasses?). If ``--debug`` was also set, then the deps are printed on the cmd line. * If necessary (meaning not ``fetch``, ``digest``, or ``manifest``), create the build directory (``/var/tmp/portage/whatever``) and the temp directory (and add T to ``settings``). Also create a ccache directory, if necessary, along with WORKDIR and D. * If ``unmerge``, call ``unmerge()`` and return. * Setup logging if requested * If ``help``, ``clean``, ``setup``, ``prerm``, ``postrm``, ``preinst``, ``postinst``, or ``config``, then pass the action to ``ebuild.sh`` (using "``ebuild.sh mydo,debug,free=1``") and return. * Generate A from SRC_URI * Ensure that DISTDIR exists; change permissions for ${DISTDIR}/cvs-src if necessary. * Run fetch(); return error on failure. * Generate digest if ``digest`` in FEATURES or called explicitly (return in the latter case). * If ``manifest`` then generate manifest and return. * Check manifest and die on failure if ``strict`` in FEATURES * If ``fetch``, return (since already done above, and it must have worked if we're here) * Set up an "``actionmap``" of dependencies (i.e., ``unpack`` requires ``dep`` and ``setup``). * If ``package``, ensure that PKGDIR is created, call ``spawnebuild`` with the current action and the actionmap, and return. * If ``qmerge``, call merge w/o checking for dependencies, and return. * If ``merge``, call ``spawnebuild`` with "``install``" and the ``actionmap``, and then call ``merge`` and return. * Else, die because the action didn't match any of the above. Oops! ebuild.sh --------- The ``ebuild.sh`` bash script predominantly gets called by the ``portage.doebuild`` function to handle the bash functions in an ebuild. Following the bouncing ball.... The first block of code is initialization stuff: * If not ``depend`` or ``clean``, then - clean the "successful" file from ${T} if it exists - Set up logging if desired (including punching a hole in sandbox for the log file) - If ${T}/environment exists, source it * unalias everything * Set "expand_aliases" and set up ``die`` and ``assert`` aliaes * Source ``/etc/profile`` on GNU userland systems * Export proper CC and CXX variables. * Set up the a hardcoded PATH, tack ${ROOTPATH} to the end of it, and add ${PREROOTPATH} to the beginning of the PATH if it exists. * Source helper functions files (``functions.sh`` and ``extra_functions.sh``). Then a great number of functions are defined: * Define a do-nothing ``esyslog()`` override function. * Define a variety of USE flag functions: ``use``, ``has``, ``has_version``, ``best_version``, ``use_with``, and ``use_enable`` functions. * Define ``diefunc`` function. * Set umask and a number of installation defaults. * Define ``check_KV`` function. * Define the ``keepdir`` function to put ".keep" files in directories that portage should not automatically remove if empty. * Define a series of sandbox helper functions. * Define the ``unpack`` function. * Define the ``econf`` convenience function. Notice that ``econf`` automatically dies on error(s). * Define the ``einstall`` convenience function. * Define ``pkg_setup`` as a placebo function to be overridden. * Define ``pkg_nofetch``. * Define minimal, but functional, ``src_unpack`` and ``src_compile`` functions. * Define ``src_install``, ``pkg_preinst``, ``pkg_postinst``, ``pkg_prerm``, and ``pkg_postrm`` skeleton functions that, when needed, should be overridden. * Define the deprecated ``try`` function. * Define the ``gen_wrapper`` function to generate ``lib/cpp`` and ``/usr/bin/cc`` wrappers. * Define some of the ``dyn-*`` functions: ``dyn_setup``, ``dyn_unpack``, ``dyn_clean``. * Define some ``install`` helper functions: ``into``, ``insinto``, ``exeinto``, ``docinto``, ``insopts``, ``diropts``, ``exeopts``, and ``libopts``. Note that a number of additional helper functions, such as the ``do*`` functions, are actual executable scripts in ``/usr/lib/portage/bin``. * Define a number of abort handler functions: ``abort_handler``, ``abort_compile``, ``abort_unpack``, ``abort_package``, and ``abort_install``. * Define the rest of the ``dyn_*`` functions: ``dyn_compile``, ``dyn_package``, ``dyn_install``, ``dyn_spec``, ``dyn_rpm``, and ``dyn_help``. * Define some eclass debugging functions: ``debug-print``, ``debug-print-function``, and ``debug-print-section``. * Define the eclass ``inherit`` function. * Define some eclass helper functions: ``EXPORT_FUNCTIONS``, ``newdepend``, ``newrdepend``, ``newcdepend``, ``newpdepend``, and ``do_newdepend``. Finally, the main part of the script is reached: * If not ``depend`` or ``clean``, then - cd into the ``/var/tmp/portage/whatever`` directory - export USER=portage if the effective user ID is "portage" - Set up distcc and ccache PATHs, environment variables, and sandbox holes * Turn on sandbox flag * Set the S default and unset a handful of variables * Source the ebuild * Set the S default if S not defined in the ebuild. (redundant?) * Set TMP and TMPDIR to be ${T} (otherwise sandbox might error out) * Set RDEPEND is not set, set it to be DEPEND. * Add eclass deps to {R,C,P,}DEPEND. * Loop over arguments passed to ``ebuild.sh``, using a big case statement - ``nofetch`` -- call ``pkg_nofetch`` - ``prerm``, ``postrm``, ``preinst``, ``postinst``, ``config`` -- turn off sandbox, handle debugging, and call the eponymous ``pkg_*`` function. - ``unpack``, ``compile``, ``clean``, ``install`` -- handle sandbox and debugging, then call the eponymous ``dyn_*`` function. - ``help``, ``clean``, ``setup`` -- Turn off sandbox, handle debugging, and call the appropriate ``dyn_*`` function. - ``package``, ``rpm`` -- Turn off sandbox, handle debugging, and call the eponymous ``dyn_*`` function. - ``depend`` -- Turn off sandbox and write dependency info to ``/var/edb/cache/dep/CATEGORY/PF``. - Else, error that the command wasn't valid. * If ``clean``, clean that package's temp directory * Finish by touching "successful" in the package's temp directory.