5.1. zeekpkg.manager module

A module defining the main Zeek Package Manager interface which supplies methods to interact with and operate on Zeek packages.

class zeekpkg.manager.Manager(state_dir, script_dir, plugin_dir, zeek_dist='', user_vars=None, bin_dir='')

Bases: object

A package manager object performs various operations on packages.

It uses a state directory and a manifest file within it to keep track of package sources, installed packages and their statuses.

sources

dictionary package sources keyed by the name given to add_source()

Type:

dict of str -> source.Source

installed_pkgs

a dictionary of installed packaged keyed on package names (the last component of the package's git URL)

Type:

dict of str -> package.InstalledPackage

zeek_dist

path to the Zeek source code distribution. This is needed for packages that contain Zeek plugins that need to be built from source code.

Type:

str

state_dir

the directory where the package manager will a maintain manifest file, package/source git clones, and other persistent state the manager needs in order to operate

Type:

str

user_vars

dictionary of key-value pairs where the value will be substituted into package build commands in place of the key.

Type:

dict of str -> str

backup_dir

a directory where the package manager will store backup files (e.g. locally modified package config files)

Type:

str

log_dir

a directory where the package manager will store misc. logs files (e.g. package build logs)

Type:

str

scratch_dir

a directory where the package manager performs miscellaneous/temporary file operations

Type:

str

script_dir

the directory where the package manager will copy each installed package's script_dir (as given by its zkg.meta or bro-pkg.meta). Each package gets a subdirectory within script_dir associated with its name.

Type:

str

plugin_dir

the directory where the package manager will copy each installed package's plugin_dir (as given by its zkg.meta or bro-pkg.meta). Each package gets a subdirectory within plugin_dir associated with its name.

Type:

str

bin_dir

the directory where the package manager will link executables into that are provided by an installed package through executables (as given by its zkg.meta or bro-pkg.meta)

Type:

str

source_clonedir

the directory where the package manager will clone package sources. Each source gets a subdirectory associated with its name.

Type:

str

package_clonedir

the directory where the package manager will clone installed packages. Each package gets a subdirectory associated with its name.

Type:

str

package_testdir

the directory where the package manager will run tests. Each package gets a subdirectory associated with its name.

Type:

str

manifest

the path to the package manager's manifest file. This file maintains a list of installed packages and their status.

Type:

str

autoload_script

path to a Zeek script named packages.zeek that the package manager maintains. It is a list of @load for each installed package that is marked as loaded (see load()).

Type:

str

autoload_package

path to a Zeek __load__.zeek script which is just a symlink to autoload_script. It's always located in a directory named packages, so as long as ZEEKPATH is configured correctly, @load packages will load all installed packages that have been marked as loaded.

Type:

str

class SourceAggregationResults(refresh_error='', package_issues=[])

Bases: object

The return value of a call to Manager.aggregate_source().

refresh_error

an empty string if no overall error occurred in the "refresh" operation, else a description of what wrong

Type:

str

package_issues

a list of reasons for failing to collect metadata per packages/repository. The first tuple element gives the repository URL in which the problem occurred and the second tuple element describes the failure.

Type:

list of (str, str)

add_source(name, git_url)

Add a git repository that acts as a source of packages.

Parameters:
  • name (str) -- a short name that will be used to reference the package source.

  • git_url (str) -- the git URL of the package source

Returns:

empty string if the source is successfully added, else the reason why it failed.

Return type:

str

aggregate_source(name, push=False)

Pull latest git info from a package source and aggregate metadata.

This is like calling refresh_source() with the aggregate arguments set to True.

This makes the latest pre-aggregated package metadata available or performs the aggregation locally in order to push it to the actual package source. Locally aggregated data also takes precedence over the source's pre-aggregated data, so it can be useful in the case the operator of the source does not update their pre-aggregated data at a frequent enough interval.

Parameters:
  • name (str) -- the name of the package source. E.g. the same name used as a key to add_source().

  • push (bool) -- whether to push local changes to the aggregated metadata to the remote package source.

Returns:

the results of the

refresh/aggregation.

Return type:

Manager.SourceAggregationResults

backup_modified_files(backup_subdir, modified_files)

Creates backups of modified config files

Parameters:
  • modified_files (list of (str, str)) -- the return value of modified_config_files().

  • backup_subdir (str) -- the subdir of backup_dir in which

Returns:

paths indicating the backup locations. The order of the returned list corresponds directly to the order of modified_files.

Return type:

list of str

bundle(bundle_file, package_list, prefer_existing_clones=False)

Creates a package bundle.

Parameters:
  • bundle_file (str) -- filesystem path of the zip file to create.

  • package_list (list of (str, str)) -- a list of (git URL, version) string tuples to put in the bundle. If the version string is empty, the latest available version of the package is used.

  • prefer_existing_clones (bool) -- if True and the package list contains a package at a version that is already installed, then the existing git clone of that package is put into the bundle instead of cloning from the remote repository.

Returns:

empty string if the bundle is successfully created, else an error string explaining what failed.

Return type:

str

bundle_info(bundle_file)

Retrieves information on all packages contained in a bundle.

Parameters:

bundle_file (str) -- the path to the bundle to inspect.

Returns:

a tuple with the the first element set to an empty string if the information successfully retrieved, else an error message explaining why the bundle file was invalid. The second element of the tuple is a list containing information on each package contained in the bundle: the exact git URL and version string from the bundle's manifest along with the package info object retrieved by inspecting git repo contained in the bundle.

Return type:

(str, list of (str, str, package.PackageInfo))

discover_builtin_packages()

Discover packages included in Zeek for dependency resolution.

This is using Zeek's --build-info flag and specifically the zkg.provides entry it contains. Requires Zeek 6.0 and later.

Returns:

List of built-in packages.

Return type:

list of package.BuiltinPackage

find_builtin_package(pkg_path)

Find a builtin plugin that matches pkg_path.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source.

Returns:

PackageInfo instance representing a builtin package matching pkg_path.

Return type:

PackageInfo

find_installed_package(pkg_path)

Return an package.InstalledPackage if one matches the name.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

get_installed_package_dependencies(pkg_path)

Return a set of tuples of dependent package names and their version number if pkg_path is an installed package.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

has_plugin(installed_pkg)

Return whether a package.InstalledPackage installed a plugin.

Parameters:

installed_pkg (package.InstalledPackage) -- the installed package to check for whether it has installed a Zeek plugin.

Returns:

True if the package has installed a Zeek plugin.

Return type:

bool

has_scripts(installed_pkg)

Return whether a package.InstalledPackage installed scripts.

Parameters:

installed_pkg (package.InstalledPackage) -- the installed package to check for whether it has installed any Zeek scripts.

Returns:

True if the package has installed Zeek scripts.

Return type:

bool

info(pkg_path, version='', prefer_installed=True)

Retrieves information about a package.

Parameters:
  • pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

  • version (str) -- may be a git version tag, branch name, or commit hash from which metadata will be pulled. If an empty string is given, then the latest git version tag is used (or the default branch like "main" or "master" if no version tags exist).

  • prefer_installed (bool) -- if this is set, then the information from any current installation of the package is returned instead of retrieving the latest information from the package's git repo. The version parameter is also ignored when this is set as it uses whatever version of the package is currently installed.

Returns:

A package.PackageInfo object.

install(pkg_path, version='')

Install a package.

Parameters:
  • pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

  • version (str) -- if not given, then the latest git version tag is installed (or if no version tags exist, the default branch like "main" or "master" is installed). If given, it may be either a git version tag, a git branch name, or a git commit hash.

Returns:

empty string if package installation succeeded else an error string explaining why it failed.

Return type:

str

Raises:

IOError -- if the manifest can't be written

installed_package_dependencies()

Return dict of 'package' -> dict of 'dependency' -> 'version'.

Package-name / dependency-name / and version-requirement values are all strings.

installed_packages()

Return list of package.InstalledPackage.

list_depender_pkgs(pkg_path)

List of depender packages.

If C depends on B and B depends on A, we represent the dependency chain as C -> B -> A. Thus, package C is dependent on A and B, while package B is dependent on just A. Example representation:

{
'A': set(),
'B': set([A, version_of_A])
'C': set([B, version_of_B])
}

Further, package A is a direct dependee for B (and implicitly for C), while B is a direct depender (and C is an implicit depender) for A.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

Returns:

list of depender packages.

Return type:

list

load(pkg_path)

Mark an installed package as being "loaded".

The collection of "loaded" packages is a convenient way for Zeek to more simply load a whole group of packages installed via the package manager.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

Returns:

empty string if the package is successfully marked as loaded, else an explanation of why it failed.

Return type:

str

Raises:

IOError -- if the loader script or manifest can't be written

load_with_dependencies(pkg_name, visited={})

Mark dependent (but previously installed) packages as being "loaded".

Parameters:
  • pkg_name (str) -- name of the package.

  • visited (set(str)) -- set of packages visited along the recursive loading

Returns:

list of tuples containing dependent package name and whether it was marked as loaded or else an explanation of why the loading failed.

Return type:

list(str, str)

loaded_package_states()

Save "loaded" state for all installed packages.

Returns:

dictionary of "loaded" status for installed packages

Return type:

dict

loaded_packages()

Return list of loaded package.InstalledPackage.

match_source_packages(pkg_path)

Return a list of package.Package that match a given path.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

modified_config_files(installed_pkg)

Return a list of package config files that the user has modified.

Parameters:

installed_pkg (package.InstalledPackage) -- the installed package to check for whether it has installed any Zeek scripts.

Returns:

tuples that describe the modified config files. The first element is the config file as specified in the package metadata (a file path relative to the package's root directory). The second element is an absolute file system path to where that config file is currently installed.

Return type:

list of (str, str)

package_build_log(pkg_path)

Return the path to the package manager's build log for a package.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

package_versions(installed_package)

Returns a list of version number tags available for a package.

Parameters:

installed_package (package.InstalledPackage) -- the package for which version number tags will be retrieved.

Returns:

the version number tags.

Return type:

list of str

pin(pkg_path)

Pin a currently installed package to the currently installed version.

Pinned packages are never upgraded when calling upgrade().

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

Returns:

None if no matching installed package could be found, else the installed package that was pinned.

Return type:

package.InstalledPackage

Raises:

IOError -- when the manifest file can't be written

refresh_installed_packages()

Fetch latest git information for installed packages.

This retrieves information about outdated packages, but does not actually upgrade their installations.

Raises:

IOError -- if the package manifest file can't be written

refresh_source(name, aggregate=False, push=False)

Pull latest git information from a package source.

This makes the latest pre-aggregated package metadata available or performs the aggregation locally in order to push it to the actual package source. Locally aggregated data also takes precedence over the source's pre-aggregated data, so it can be useful in the case the operator of the source does not update their pre-aggregated data at a frequent enough interval.

Parameters:
  • name (str) -- the name of the package source. E.g. the same name used as a key to add_source().

  • aggregate (bool) -- whether to perform a local metadata aggregation by crawling all packages listed in the source's index files.

  • push (bool) -- whether to push local changes to the aggregated metadata to the remote package source. If the aggregate flag is set, the data will be pushed after the aggregation is finished.

Returns:

an empty string if no errors occurred, else a description of what went wrong.

Return type:

str

remove(pkg_path)

Remove an installed package.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

Returns:

True if an installed package was removed, else False.

Return type:

bool

Raises:
  • IOError -- if the package manifest file can't be written

  • OSError -- if the installed package's directory can't be deleted

restore_loaded_package_states(saved_state)

Restores state for installed packages.

Parameters:
  • saved_state (dict) -- dictionary of saved "loaded" state for installed

  • packages. --

save_temporary_config_files(installed_pkg)

Return a list of temporary package config file backups.

Parameters:

installed_pkg (package.InstalledPackage) -- the installed package to save temporary config file backups for.

Returns:

tuples that describe the config files backups. The first element is the config file as specified in the package metadata (a file path relative to the package's root directory). The second element is an absolute file system path to where that config file has been copied. It should be considered temporary, so make use of it before doing any further operations on packages.

Return type:

list of (str, str)

source_packages()

Return a list of package.Package within all sources.

test(pkg_path, version='', test_dependencies=False)

Test a package.

Parameters:
  • pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

  • version (str) -- if not given, then the latest git version tag is used (or if no version tags exist, the default branch like "main" or "master" is used). If given, it may be either a git version tag or a git branch name.

  • test_dependencies (bool) -- if True, any dependencies required for the given package will also get tested. Off by default, meaning such dependencies will get locally built and staged, but not tested.

Returns:

a tuple containing an error message string, a boolean indicating whether the tests passed, as well as a path to the directory in which the tests were run. In the case where tests failed, the directory can be inspected to figure out what went wrong. In the case where the error message string is not empty, the error message indicates the reason why tests could not be run. Absence of a test_command in the requested package is considered an error.

Return type:

(str, bool, str)

unbundle(bundle_file)

Installs all packages contained within a bundle.

Parameters:

bundle_file (str) -- the path to the bundle to install.

Returns:

an empty string if the operation was successful, else an error message indicated what went wrong.

Return type:

str

unload(pkg_path)

Unmark an installed package as being "loaded".

The collection of "loaded" packages is a convenient way for Zeek to more simply load a whole group of packages installed via the package manager.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

Returns:

True if a package is successfully unmarked as loaded.

Return type:

bool

Raises:

IOError -- if the loader script or manifest can't be written

unload_with_unused_dependers(pkg_name)

Unmark dependent (but previously installed packages) as being "loaded".

Parameters:

pkg_name (str) -- name of the package.

Returns:

list of tuples containing dependent package name and whether it was marked as unloaded or else an explanation of why the unloading failed.

Return type:

list(str, str)

Raises:

IOError -- if the loader script or manifest can't be written

unpin(pkg_path)

Unpin a currently installed package and allow it to be upgraded.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

Returns:

None if no matching installed package could be found, else the installed package that was unpinned.

Return type:

package.InstalledPackage

Raises:

IOError -- when the manifest file can't be written

upgrade(pkg_path)

Upgrade a package to the latest available version.

Parameters:

pkg_path (str) -- the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in alice/zkg.index, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo".

Returns:

an empty string if package upgrade succeeded else an error string explaining why it failed.

Return type:

str

Raises:

IOError -- if the manifest can't be written

validate_dependencies(requested_packages, ignore_installed_packages=False, ignore_suggestions=False, use_builtin_packages=True)

Validates package dependencies.

Parameters:
  • requested_packages (list of (str, str)) -- a list of (package name or git URL, version) string tuples validate. If the version string is empty, the latest available version of the package is used.

  • ignore_installed_packages (bool) -- whether the dependency analysis should consider installed packages as satisfying dependency requirements.

  • ignore_suggestions (bool) -- whether the dependency analysis should consider installing dependencies that are marked in another package's 'suggests' metadata field.

  • use_builtin_packages (bool) -- whether package information from builtin packages is used for dependency resolution.

Returns:

the first element of the tuple is an empty string if dependency graph was successfully validated, else an error string explaining what is invalid. In the case it was validated, the second element is a list of tuples, each representing a package, where:

  • The first element is a dependency package that would need to be installed in order to satisfy the dependencies of the requested packages.

  • The second element of tuples in the list is a version string of the associated package that satisfies dependency requirements.

  • The third element of the tuples in the list is a boolean value indicating whether the package is included in the list because it's merely suggested by another package.

The list will not include any packages that are already installed or that are in the requested_packages argument. The list is sorted in dependency order: whenever a dependency in turn has dependencies, those are guaranteed to appear in order in the list. This means that reverse iteration of the list guarantees processing of dependencies prior to the depender packages.

Return type:

(str, list of (package.PackageInfo, str, bool))

zeek_plugin_path()

Return the path where installed package plugins are located.

This path can be added to ZEEK_PLUGIN_PATH for interoperability with Zeek.

zeekpath()

Return the path where installed package scripts are located.

This path can be added to ZEEKPATH for interoperability with Zeek.

class zeekpkg.manager.Stage(manager, state_dir=None)

Bases: object

get_subprocess_env()
populate()