# Migrating a `debhelper` plugin to `debputy`

_This is [how-to guide] and is primarily aimed at getting a task done._

<!-- To writers and reviewers: Check the documentation against https://documentation.divio.com/ -->

This document will help you convert a `debhelper` plugin / `debhelper` tool into a `debputy` plugin.
Prerequisites for this how-to guide:

 * You have a `debhelper` tool/plugin that you want to migrate.  Ideally a simple one as not all tools
   can be migrated at this time.
 * Many debhelper tools do not come with test cases, because no one has created a decent test framework
   for them.  Therefore, consider how you intend to validate that the `debputy` plugin does not have any
   (unplanned) regressions compared to `debhelper` tool.
 * You are familiar with setting up a basic python project plus related packaging. You will be expected
   to apply this in Step 2.

Note that during the conversion, you may find that `debputy` cannot support the requirements for your
debhelper tool for now.  Feel free to file an issue for what is holding you back in the
[debputy issue tracker].

Prerequisites
-------------

This guide assumes familiarity with Debian packaging and the debhelper tool stack in general.  Notably,
you are expected to be familiar with the `Dh_Lib.pm` API to the point of recognising references to said
API and how to look up document for methods from said API.

If the debhelper tool is not written in `Dh_Lib.pm`, then you will need to understand how to map the
`Dh_Lib.pm` reference into the language/tool equivalent on your own.

## Step 0: The approach taken

The guide will assume you migrate one tool (a `dh_foo` command) at a time. If you have multiple tools
that need to migrate together, you may want to review "Step 1" below for all tools before migrating to
further steps.

## Step 1: Analyze what features are required by the tools and the concept behind the helper

For the purpose of this guide, we can roughly translate debhelper tools into one or more
of the following categories:

 1. Install `debian/pkg.foo` as it is into directory `Y`.
    - This category uses a mix of `pkgfile` + `install_dir` + `install_file` / `install_prog`
    - Example: `dh_installtmpfiles`
 2. Read `debian/pkg.foo` and do something based on the content of said file.
    - Typically, the category uses a mix of `pkgfile` + `filedoublearray` / `filearray` / `open(...)`.
      The most common case of this is to install a list of files in the `debian/pkg.foo` file.
    - Example: `dh_install`
 3. If some file is installed in or beneath a directory `Y`, (optionally) analyze it and apply metadata
    (substvars, maintscripts, triggers, etc.). Note this does *not* apply to special-case of services.
    While services follow this pattern, `debputy` will have special support for services.
    - Typically, this category uses a bit of glob matching + (optionally) `open` +
      `addsubstvars` / `autoscript` / `autotrigger`
    - Example: `dh_installtmpfiles`, `dh_usrlocal`
    - *Counter* examples: `dh_installsystemd` (due to service rule).
 4. Any tool that manages services like `systemd`, `init.d` or `runit`.
    - Typically, this category uses a bit of glob matching + (optionally) `open` +
      `addsubstvars` / `autoscript` / `autotrigger`
    - Example: `dh_installsystemd`
 5. Based on a set of rules, modify a set of files if certain criteria are met.
    - Example: `dh_strip`, `dh_compress`, `dh_dwz`, `dh_strip_nondeterminism`, `dh_usrlocal`
 6. Run custom build system logic that cannot or has not been fit into the `debhelper` Buildsystem API.
    - Example: `dh_cmake_install`, `dh_raku_build`, etc.
 7. "None of the above". There are also tools that have parts not fitting into any of the above.
    - Which just means the guide has no good help to offer you for migrating.
    - Example: `dh_quilt_patch`

As mentioned, a tool can have multiple categories at the same time.  As an example:

 * The `dh_installtmpfiles` tool from debhelper is a mix between 1 (installing `debian/pkg.tmpfiles` in to
   `usr/lib/tmpfiles.d`) and 3 (Generate a maintscript based on `<prefix>/tmpfiles.d/*.conf` globs).

 * The `dh_usrlocal` tool from debhelper is a mix between 3 (Generate a maintscript to create dirs in
   `usr/local` as necessary on install and clean up on removal) and 4 (Remove any directory from `usr/local`
   installed into the package).


When migrating a tool (or multiple tools), it is important to assert that all categories are supported by
the `debputy` plugin API.  Otherwise, you will end with a half-finished plugin and realize you cannot
complete the migration because you are missing a critical piece that `debputy` currently do not support.

At the time of writing, `debputy` support the following categories: 1 + 3

If your tool does not fit inside those two base categories, you cannot fully migrate the tool. You should
consider whether it makes sense to continue without the missing features.

## Step 2: Provide a basic empty plugin

All `debputy` plugins are written in python, and you will need to prepare your package for handling python
code. How you do this is left to you (as promised in the introduction of this guide). This step assumes you
have done this and is now ready to write the python file(s) that will make up your plugin.

A `debputy` plugin starts with an initialization function like this:

```python
from debputy.plugin.api import DebputyPluginInitializer

def initialize_my_plugin(api: DebputyPluginInitializer):
    pass
```

To register the debputy plugin, you must provide a small JSON file named `myplugin.json` with the following content:


```json
{
  "plugin_version": "<version of your plugin here>",
  "plugin_initializer": "initialize_my_plugin",
  "module": "mypythonmodule",
  "api_compat_level": 1
}
```

Remember to replace the values, so they match your plugin. The keys are:

 * `plugin_version`: The version of your plugin. It will be displayed to users. Note the filename will become
   the plugin name (that is, `foo.json` -> the `foo` plugin)
 * `plugin_initializer`: The function `debputy` should call to initialize your plugin. This is the function we
   just defined in the previous example). The plugin loader requires this initialization function to be a top
   level function of the module (that is, `getattr(module, plugin_initializer)` must return the initializer
   function).
 * `module`: The python module the `plugin_initializer` function is defined in.
 * `api_compat_level`: This is the API compat level of `debputy` required to load the
   plugin. This defines how `debputy` will load the plugin and is to ensure that
   `debputy`'s plugin API can evolve gracefully.  For now, only one version is supported
   and that is `1`.

This file then has to be installed into the `debputy` plugin directory.

With this you have an empty plugin that `debputy` can load, but it does not provide any features.

## Step 3: Provide packager provided files (Category 1 tools)

*This step only applies if the tool in question automatically installs `debian/pkg.foo` in some predefined path
like `dh_installtmpfiles` does.  If not, please skip this section as it is not relevant to your case.*

You can ask `debputy` to automatically detect `debian/pkg.foo` files and install them into a concrete directory
by using the `packager_provided_file` file.  An example would be:

```python
from debputy.plugin.api import DebputyPluginInitializer

def initialize_my_plugin(api: DebputyPluginInitializer):
    api.packager_provided_file(
        "foo",  # This is the "foo" in "debian/pkg.foo"
        "/usr/share/foo",  # This is the directory to install them into.
        filename_format="{name}.conf",  # Optionally, impose a given format like `pkg.conf` rather than `pkg`.
    )
```

This would make `debputy` install `debian/my-pkg.foo` as `/usr/share/foo/my-pkg.conf` provided the plugin is
loaded.

If you are the type that like to provide tests for your code, the following `py.test` snippet can get you started:

```python
from debputy.plugin.api.test_api import initialize_plugin_under_test_from_desc


def test_packager_provided_files():
    # Correct the path to match the current path to the myplugin.json file.
    # Also, make sure that your python module is located in the python search path (PYTHONPATH=somewhere/ might help)
    plugin = initialize_plugin_under_test_from_desc("path/to/myplugin.json")
    ppf_by_stem = plugin.packager_provided_files_by_stem()
    assert ppf_by_stem.keys() == {'foo'}
    foo_file = ppf_by_stem['foo']

    assert foo_file.name_stem == 'foo'
    assert foo_file.compute_dest_basename("g++-3.1") == 'g++-3.1.conf'
    # Note we could test install_into as well. This is omitted because `debputy` translate it into
    # a normalized path and the test would then be dependent on this implementation detail
```

Note this test does *not* rely on python's EntryPoint API to work.

## Step 3: Migrate metadata detection (Category 3 tools)

*This step only applies if the tool in question generates substvars, maintscripts or triggers based on
certain paths being present or having certain content like `dh_installtmpfiles` does.  However,
this section does **NOT** apply to service management tools (such as `dh_installsystemd`). If not, please
skip this section as it is not relevant to your case.*

For dealing with substvars, maintscripts and triggers, the plugin will need to register a function that
can perform the detection. The `debputy` API refers to it as a "detector" and functionally it behaves like
a "callback" or "hook". The "detector" will be run once per package that it applies to with some context and
is expected to register the changes it wants.

A short example is:

```python
import textwrap
from debputy.plugin.api import DebputyPluginInitializer, VirtualPath, BinaryCtrlAccessor,
    MetadataDetectorPackageContext


def initialize_my_plugin(api: DebputyPluginInitializer):
    # ... remember to preserve any existing code here that you may have had from previous steps.
    api.metadata_or_maintscript_detector(
        "foo-detector",  # This is an ID of the detector. It is part of the plugins API and should not change.
        # Packagers see it if it triggers an error and will also be able to disable by this ID.
        detect_foo_files,  # This is the detector (hook) itself.
    )


def detect_foo_files(fs_root: VirtualPath,
                     ctrl: BinaryCtrlAccessor,
                     context: MetadataDetectorPackageContext,
                     ) -> None:
    # If for some reason the hook should not apply to all packages, and `metadata_or_maintscript_detector` does not
    # provide a filter for it, then you just do an `if <should not apply>: return`
    if not context.binary_package.is_arch_all:
        # For some reason, our hook only applies to arch:all packages.
        return

    foo_dir = fs_root.lookup("usr/share/foo")
    if not foo_dir:
        return

    conf_files = [path.absolute for path in foo_dir.iterdir if path.is_file and path.name.endswith(".conf")]
    if not conf_files:
        return
    ctrl.substvars.add_dependency("misc:Depends", "foo-utils")
    conf_files_escaped = ctrl.maintscript.escape_shell_words(*conf_files)
    snippet = textwrap.dedent(f"""\
    foo-analyze --install {conf_files_escaped}
    """)
    ctrl.maintscript.on_configure(snippet)
```

This code would register the `detect_foo_files` function as a metadata hook. It would be run for all regular `deb`
packages processed by `debputy` (`udeb` requires opt-in, auto-generated packages such as `-dbgsym` cannot be
targeted).

The hook conditionally generates a dependency (via the `${misc:Depends}` substvar) on `foo-utils` and a `postinst`
snippet to be run when the package is configured.

An important thing to note is that `debputy` have *NOT* materialized the package anywhere. Instead, `debputy`
provides an in-memory view of the file system (`fs_root`) and related path metadata that the plugin should base its
analysis of. The in-memory view of the file system can have virtual paths that are not backed by any real
path on the file system.  This commonly happens for directories and symlinks - and during tests, also for files.


If you are the type that like to provide tests for your code, the following `py.test` snippet can get you started:

```python
from debputy.plugin.api.test_api import initialize_plugin_under_test_from_desc, build_virtual_file_system, \
    package_metadata_context


def test_packager_provided_files():
    # Correct the path to match the current path to the myplugin.json file.
    # Also, make sure that your python module is located in the python search path (PYTHONPATH=somewhere/ might help)
    plugin = initialize_plugin_under_test_from_desc("path/to/myplugin.json")
    detector_id = 'foo-detector'

    fs_root = build_virtual_file_system([
        '/usr/share/foo/foo.conf'  # Creates a virtual (no-content) file.
        # Use virtual_path(..., fs_path="/path") if your detector needs to read the file
        # NB: You have to create that "/path" yourself.
    ])

    plugin.run_metadata_detector(
        detector_id,
        fs_root,
        # Test with an arch:any package. The test framework will supply a minimum number of fields
        # (e.g., "Package") so you do not *have* to provide them if you do not need them.
        # That is also why providing `Architecture` alone works here.
        context=package_metadata_context(package_fields={'Architecture': 'any'})
    )
    # Per definition of our detector, there should be no dependency added (even though the file is there)
    assert 'misc:Depends' not in plugin.substvars
    # Nor should any maintscripts have been added
    assert plugin.maintscript_registered_by(detector_id) == []

    plugin.run_metadata_detector(
        detector_id,
        fs_root,
        # This time, we test with an arch:all package
        context=package_metadata_context(package_fields={'Architecture': 'all'})
    )

    assert plugin.substvars['misc:Depends'] == 'foo-utils'

    # You could also have added `maintscript='postinst'` to filter by which script it was added to.
    snippets = plugin.maintscript_registered_by(detector_id)
    # There should be exactly one snippet
    assert len(snippets) == 1
    snippet = snippets[0]
    # And we can verify that the snippet is as expected.
    assert snippet.maintscript == 'postinst'
    assert snippet.registration_method == 'on_configure'
    assert 'foo-analyze --install /usr/share/foo/foo.conf' in snippet.detector_provided_script
```

Note this test does *not* rely on python's EntryPoint API to work.

## Closing

You should now either have done all the basic steps of migrating the debhelper tool to `debputy`
or discovered some feature that the guide did not cover. In the latter case, please have a look
at the [debputy issue tracker] and consider whether you should file a feature request for it.

[how-to guide]: https://documentation.divio.com/how-to-guides/
[debputy issue tracker]: https://salsa.debian.org/debian/debputy/-/issues
