2022.07.26

Scons

https://scons.org/doc/production/HTML/scons-man.html

Description

An SCons build system is set up by a script, describing

  • things to build (targets)
  • (optional) rules to build those files (actions)

Before reading the SConscript files

  • search site_scons/s
    • (1) evaulate site_init.py, if exists
    • (2) prepended site_tools/ to the default toolpath the path, if exists

Auto build dependencies For example, #include preprocessor directives in C or C++ files. Scons will rebuild dependent files appropriately whenever any "included" input file changes. Scons supports the ability to define new scanners to support additional input file types.

Target Selection

Sconscript File Reference

Environment

Tools

Builder

# character it is top-relative

Predefined builder methods

  • env.CFile()
  • ...

Construction methods

  • env.Action(...)
  • ...
  • env.Builder(action ,[args])

print all builders

env = Environment()
print("Builders:", list(env['BUILDERS']))

TODO

2022.06.11

version: 21.2.1.1, commit: e4fae58da6c044b6efec62392ff99f343ce67947

As a gem5 beginner, I am curious how config script works. For example, gem5/configs/learning_gem5/part1/simple.py,

import m5
import m5.objects import *
system = System()

where do m5, objects, System come from?

Importer

# entrance: src/python/m5/main.py
gem5.opt --pdb simple.py
(pdb) p m5

Like code above, pdb shows the type of m5 is importer.ByteCodeLoader, which is defined in src/python/importer.py.

importer.py add a its own finder to meta_path

sys.meta_path.insert(0, importer)

[TODO] How importer.py works with config script?

Files in psedoImport.py is generated by script below,

find -name SConscript -exec grep sim_objects {}  \; \
| egrep "\w+\.py" -o \
| sort \
| uniq > /tmp/out

SimObject

  • src/SConsript:

    gem5_lib_simobjects = SimObject.all.with_tag(env, 'gem5 lib')

    • SimObject(PySource):

      super().__init__('m5.objects', source, tags, add_tags)

      • src/SConscript:

        class PySource(SourceFile):
  ...
  def __init__(self, package, source, tags=None, add_tags=None):
    super().__init__(source, tags, add_tags)
    ...
    cpp = File(self.filename + '.cc')
    ...

        cpp = X86Decoder.py.cc

        • site_scons/gem5_scons/sources.py:

          SourceFile(object, metaclass=SourceMeta): `__init__`

          • SourceMeta(type): __init__(cls, ...)

            cls.all = SourceList() // class __init__(self, ...) first arg is self!

            Therefore cls.all = ... add all to SimObject.

            • SourceList(list): __getattr__
              • SourceFilter.factories.with_tag
                • with_tag is add by SourceFilter.factories.update(...)

scons debug

add pdb trace to scons script, then run scons,

import pdb
pdb.set_trace()
2022.06.14

FS[TODO] [TEMP]

./build/ARM/gem5.opt configs/example/arm/fs_bigLITTLE.py --caches --bootloader "$IMG_ROOT/binaries/boot.arm64" --kernel "$IMG_ROOT/binaries/vmlinux.arm64" --disk "$IMG_ROOT/../ubuntu-18.04-arm64-docker.img" --bootscript=configs/boot/bbench-gb.rcS

Root

Only one Root instance.

src/sim/Root.py: _the_instance

src/python/m5/simulate.py: root = objects.Root.getInstance()

2022.06.15

Param

src/python/m5/params.py:

Param = ParamFactory(ParamDesc), Param is an instance of ParamFactory, whose param_desc_class is class ParamDesc.

When Param.Int(5, "number of widgets") is executed, there are sevearl processes as below.

  • Param.Int gets the attr Int from Param, which calls ParamFactory.__getattr__(self, attr). This __getattr__ initiates a new ParamFactory instance. self.ptype_str is originally empty. After initiation, self.ptype_str becomes Int. Therefore Param.Int is an instance of ParamFactory, with ptype_str = "Int".

  • Param.Int() invokes ParamFactory.__call__(...). Something is taken from allParams. The initiation of allParams is also quite tricky, which hevaily depends on python's metaclass.

    • Here is the inheritage relations of Int class

      class Int(CheckedInt)
class CheckedInt(NumericParamValue, metaclass=CheckedIntType)
class CheckedIntType(MetaParamValue)
class MetaParamValue(type)

      The inheritage means the Int is derived from a metaclass MetaParamValue. Every time a class, like Int, is created, MetaParamValue.__new__(mcls, name, bases, dct) will be invoked, where mcls is the to-be-created class, Int, name is the name of the to-be-created class, "Int". Then, class Int will be added to allParams["Int"], and the same for every class derived from MetaParamValue.

    Therefore, class Int is taken from allParams, ptype = class Int. As I depicted above param_desc_class is class ParamDesc, so self.param_desc_class(...) is the constructor of ParamDesc, aka ParamDesc.__init__(...).

    Till now, every compilicated problem is cleared. Param.Int(5, "number of widgets") contains two arguments, so 5 is self.default, "number of widgets" is self.desc.

SimObject and its subclass is also can be used in Param, see source code below,

# src/python/m5/params.py
ParamFactory:
    __call__():
        try:
            ptype = allParams[self.ptype_str]
        except KeyError:
            # if name isn't defined yet, assume it's a SimObject, and
            # try to resolve it later
            pass

Scons in Gem5

  • Q: How gem5 binary is compiled, linked by scons?
    • Q: How are isa/main.isa generated files compiled?
    • A: see chapter isa/main.isa

isa/main.isa

  • src/arch/x86/SConscript:

    isa_desc_files = ISADesc('isa/main.isa', tags='x86 isa')

    • src/arch/SConscript:

      def ISADesc(desc, decoder_splits=1, exec_splits=1, tags=None, add_tags=None):
  ...
  IsaDescBuilder(target=gen, source=sources, env=env)
  ...

      Here target are files in build/X86/arch/x86/generated, source are isa/main.isa, micro_asm.py and parser_files.

      • src/arch/SConscript:

        def run_parser(target, source, env):
    # Add the current directory to the system path so we can import files.
    sys.path[0:0] = [ arch_dir.abspath ]
    import isa_parser

    parser = isa_parser.ISAParser(target[0].dir.abspath)
    parser.parse_isa_desc(source[0].abspath)

desc_action = MakeAction(run_parser, Transform("ISA DESC", 1))

IsaDescBuilder = isa_desc_filesBuilder(action=desc_action)

        MakeAction is a wrapper of scons Action, in site_scons/gem5_scons/init.py.

        I guess (output=)Transform(...) is for echo message to terminal.

        run_parser is the builder action for isa file.

        A parser is created. And parser is used to parse isa/main.isa.

        Then, everything else runs as noted in isa.md.

        # These generated files are also top level sources.
def source_gen(name):
  ...
source_gen('decoder.cc')
...

        The generated files are Source, which will be added to all. See SourceFile and SourceMeta. If the tags are None, the default tags are 'gem5 lib'.

binary compilation

Take build/X86/gem5.debug for an example.

  • SConstruct:

    for t in BUILD_TARGETS:
  this_build_root, variant = parse_build_path(t)
  ...

    build_root=build, variant=X86.

    for variant_path in variant_paths:
  ...
  current_vars_file = os.path.join(build_root, 'variables', variant_dir)
  if isfile(current_vars_file):
    sticky_vars.files.append(current_vars_file)
    ...
  sticky_vars.Update(env)
  ...

    current_vars_file=build/variables/X86. Therefore, vars in build/variables/X86 are added to env.

    SConscript('src/SConscript', variant_dir=variant_path, exports=exports)

    If the optional variant_dir argument is present, it causes an effect equivalent to the VariantDir function. VariantDir() sets up an alternate build location.

    These variables are locally exported only to the called SConscript file(s) and do not affect global pool managed by Export. The subsidiary SConscript files must use the Import function to import the variables.

    • src/SConscript:

      class Gem5(Executable):
  '''Base class for the main gem5 executable.'''
  def declare(self, env):
    ...

      Definition of class Gem5. The inheritage of class Gem5, Gem5 -> Executable -> TopLevelBase -> TopLevelMeta.

      # Walk the tree and execute all SConscripts in subdirectories
...

      Gem5('gem5', with_any_tags('gem5 lib', 'main'))

      The default tags for source file is gem5 lib, see SourceFile.

      class Executable(TopLevelBase):
  '''Base class for creating an executable from sources.'''
  def path(self, env):
    return self.dir.File(self.target + '.${ENV_LABEL}')
  ...

      ENV_LABEL=debug/opt/fast.

      for env in (envs[e] for e in needed_envs):
  for cls in TopLevelMeta.all:
    cls.declare_all(env)

      The final part of src/Sconscript will call declare(env) func of everything TopLevelBase, which include Gem5(...).

      • src/SConscript:

        class Gem5(Executable):
  ...
  def declare(self, env):
    ...
    return super().declare(env, objs)

        • src/SConscript:

          class Executable(TopLevelBase):
  ...
  def declare(self, env, objs=None):
    if objs is None:
      objs = self.srcs_to_objs(env, self.sources(env))
    ...
    executable = env.Program(self.path(env).abspath, objs)[0]
    ...