Configuration#

This document describes the TOML format used by the Translator.from_config()-method.

Minimal configuration#

Only a fetcher is required, in this file or in one of the extra_fetchers. This file, saved as translation.toml:

[translator]
fmt = "{id}:{name}"

[fetching.SqlFetcher]
connection_string = "postgresql://localhost/db"

is loaded with Translator.from_config("translation.toml").

Sections#

The valid top-level keys are translator, fetching, unknown_ids, and transform (the TOP_LEVEL_KEYS). Only the fetching section is required, though it may be left out of the main configuration file if fetching is configured separately. Any other top-level key will raise a ConfigurationError.

Section: Translator#

Section keys: `[translator]`#
Key	Type	Description
fmt	`Format`	Specify how translated IDs are displayed. Defaults to `{id}:{name}`; use `fmt = "{name}"` for the label alone.
enable_uuid_heuristics	`bool`	Improves matching when `UUID`-like IDs are in use.

Parameters for Name-to-source mapping are specified in a [translator.mapping]-subsection. See: Subsection: Mapping for details (no context).

Section: Unknown IDs#

Section keys: `[unknown_ids]`#
Key	Type	Description	Comments
fmt	`Format`	Specify a format for untranslated IDs.	Can be a plain string `fmt='Unknown'`, or `fmt='{id}'` to leave as-is.

Alternative placeholder-values for unknown IDs can be declared in a [unknown_ids.overrides]-subsection. See: Subsection: Overrides for details (context = source).

Section: Fetching#

The type of the fetcher is determined by the second-level key (mapping, cache and MultiFetcher reserved). For example, a MemoryFetcher would be created by adding a [fetching.MemoryFetcher]-section. A single file may only declare one fetcher this way; a single fetcher commonly serves many sources (e.g. one SqlFetcher across several tables); see Multiple fetchers below if you need to combine fetchers, e.g. a SqlFetcher alongside a MemoryFetcher.

The MemoryFetcher is handy for small, static sources. Give each source one column per placeholder (id and name at minimum):

[fetching.MemoryFetcher.data.customers]
id = [1, 2]
name = ["Alice", "Bob"]

See PlaceholderTranslations.make() for the other accepted data forms. For string-keyed sources, a scalar shorthand (P = "Pending") is often handier; the adoption guide uses it. Avoid it for integer IDs: TOML keys are always strings, so 101 = "Widget" would key the row under the string "101".

Section keys: `[fetching.<Type>]`#
Key	Type	Description	Comments
allow_fetch_all	`bool`	Control access to `fetch_all()`.	Some fetcher types redefine or ignore this key.
selective_fetch_all	`bool`	`fetch_all` skips sources that lack a required placeholder (after mapping).
identifiers	`Sequence[str]`	Hierarchical identifiers for the fetcher.	Based on source file if not given.
optional	`bool`	If `True`, discard on `sources`-resolution crash.	Multi-fetcher mode only. See Optional fetchers for details.

The keys listed above are for the AbstractFetcher class, which all fetchers created by TOML configuration must inherit. Additional parameters vary based on the chosen implementation. See the id_translation.fetching module for choices.

The AbstractFetcher uses a Mapper to bind actual placeholder names in sources to desired placeholder names requested by the calling Translator instance. See: Subsection: Mapping for details. For all mapping operations performed by the AbstractFetcher, context = source.

Hint

Custom fetchers may be initialized by using sections with fully qualified type names in single quotation marks. For example, a [fetching.'my.library.SuperFetcher']-section would import and initialize a SuperFetcher from the my.library module.

Under the hood, this will call get_by_full_name() using name="my.library.SuperFetcher". It must accept the keyword arguments of AbstractFetcherParams and pass them on to AbstractFetcher.

Multiple fetchers#

Complex applications may require multiple fetchers. These may be specified in auxiliary config files, one fetcher per file. These files may only contain fetching and transform sections. If multiple fetchers are defined, a MultiFetcher is created. Fetchers defined this way are hierarchical. The input order determines rank: when several fetchers claim a source, the highest-ranked one serves it if on_source_conflict is 'warn' or 'ignore'. The default, 'raise', rejects the configuration when sources are discovered. For example, for a Translator created by running

>>> from id_translation import Translator
>>> extra_fetchers=["primary-fetcher.toml", "secondary-fetcher.toml"]
>>> Translator.from_config("translation.toml", extra_fetchers=extra_fetchers)

a source claimed by several fetchers is served by the one in translation.toml (if there is one), then primary-fetcher.toml and finally secondary-fetcher.toml, in that order of preference, when conflicts are allowed.

Section keys: `[fetching.MultiFetcher]` (main config only)#
Key	Type	Description
max_workers	`int`	Maximum number of individual child fetchers to call in parallel.
on_source_conflict	raise \| warn \| ignore	Action for disputes during `source discovery`.
fetcher_discarded_log_level	`int \| str`	Log level used when an `optional` fetcher is discarded because `source discovery` failed.

Optional fetchers#

Optional fetchers are allowed to raise when Fetcher.initialize_sources() is called, for example because a database is down. Configuration errors (a ConfigurationError, or a source missing a required placeholder, such as an ID column) still raise. Fetchers should not raise when imported or initialized. To suppress init errors (e.g. ModuleNotFoundError), the config file must specify optional = true in the class init args:

[fetching."my_module.MyFetcher"]
optional = true

The ID_TRANSLATION_SUPPRESS_OPTIONAL_FETCHER_INIT_ERRORS variable must also be true. The TranslatorFactory will always use the ERROR level for fetchers that are discarded this way.

Warning

Using ID_TRANSLATION_SUPPRESS_OPTIONAL_FETCHER_INIT_ERRORS=true can and often will hide configuration errors (e.g. misspelled argument names) or broken packages.

A fetcher discarded because it failed to initialize takes its file’s [transform]-section with it, in the main configuration as well as in an auxiliary one. A fetcher discarded later, when Fetcher.initialize_sources() raises, does not: the section was built when the configuration was read. Its transformers stay registered, and apply to nothing unless a surviving fetcher serves the same source.

Note

optional = true covers an unavailable data source, not an unreadable file: the file must parse, and its top-level sections are still checked. The [transform]-section itself is built after the discard check, so a malformed one is a ConfigurationError only when its fetcher survives: a discarded fetcher takes even a broken section with it.

Caching#

A fetcher can have one CacheAccess, declared in the same file as the fetcher by a [fetching.cache.'<type>'] section. Like fetchers and transformers, the section is keyed by the fully qualified type name, and its keys are passed to the constructor as-is:

[fetching.SqlFetcher]
connection_string = "postgresql://localhost/db"

[fetching.cache.'my.library.MyCacheAccess']
ttl = 3600  # Cache timeout in seconds

This is equivalent to SqlFetcher(connection_string=..., cache_access=MyCacheAccess(ttl=3600)). With Multiple fetchers, each file’s cache applies only to the fetcher declared in that file.

Deprecated since version 1.3.0: The [fetching.cache] + type = "..." form. It still works, with a FutureWarning, and is rejected in id-translation==2.0.0. Giving both forms is an error.

Choosing a caching strategy#

Before implementing a CacheAccess, check whether a simpler built-in mechanism already fits. All three avoid re-fetching translation data; they differ in scope, lifetime, and storage.

Mechanism	Scope	Lifetime	Storage	Use when
`go_offline()`	Whole `Translator`	In-process; discards the fetcher	Memory	Required IDs are known in advance.
`load_persistent_instance()`	Whole `Translator`	Cross-process; reused until it `must be recreated`	Disk (`pickle`)	The cache should be shared or reused between processes.
`CacheAccess`	Per source	User-defined	User-defined	You need per-source control, or want to avoid the others’ trade-offs.

Implementing `CacheAccess`#

This library does not provide any CacheAccess implementations.

Instead, users may implement the CacheAccess interface to define their own caching logic. The AbstractFetcher will then call CacheAccess.load() and CacheAccess.store() when appropriate.

Subsection: Mapping#

Mapping binds names to sources in [translator.mapping], and placeholders to source columns in [fetching.mapping]; * below stands for either. The Mapping primer explains the procedure and the terms it uses (values, candidates and context).

Section keys: `[*.mapping]`#
Key	Type	Description	Comments
score_function	`ScoreFunction`	Compute value/candidate-likeness	See built-in `score_functions`.
min_score	`float`	Minimum score for a match.	Default `0.90`.
on_unmapped	raise \| warn \| ignore	Handle unmatched values.
on_unknown_user_override	raise \| warn \| keep	Handle an override function returning an unknown candidate.
cardinality	`Cardinality`	Determine how many candidates to map a single value to.	E.g. ‘1:1’ or ‘N:1’.

External functions may be used by putting fully qualified names in single quotation marks. Names which do not contain any dot characters ('.') are assumed to refer to functions in the appropriate id_translation.mapping submodule.
[translator.mapping] has scoring disabled unless score_function is set (see Override-only mapping), and heuristics there require one. [fetching.mapping] starts from the fetcher’s default scoring, and heuristics given without a score_function add to it.

Filter functions#

Filters are given in [[*.mapping.filter_functions]] list-subsections. These may be used to remove undesirable matches, for example SQL tables which should not be used or a DataFrame column that should not be translated.

Section keys: `[[*.mapping.filter_functions]]`#
Key	Type	Description	Comments
function	`str`	Function name.	See built-in `filter_functions`.

Note

Additional keys depend on the chosen function implementation.

As an example, the next snippet ensures that only names ending with an '_id'-suffix will be translated by using a filter_names()-filter.

[[translator.mapping.filter_functions]]
function = "filter_names"
regex = ".*_id$"
remove = false  # The default.

Score function#

Some score functions (ScoreFunction) take additional keyword arguments, given in a [*.mapping.score_function.<name>]-subsection instead of the score_function key. At most one may be given:

Arguments for the modified_hamming() scorer.#

[translator.mapping.score_function.modified_hamming]
add_length_ratio_term = false

See id_translation.mapping.score_functions for options.

Score function heuristics#

Heuristics may be used to aid an underlying score_function to make more difficult matches. There are two types of heuristic functions: alias functions (AliasFunction), and short-circuiting functions such as short_circuit(), which are filter functions whose returned candidates win outright (score +∞, others −∞); the score function still runs first.

Heuristics are given in [[*.mapping.score_function_heuristics]] list-subsections (note the double brackets) and are applied in the order in which they are given by the HeuristicScore wrapper class.

Section keys: `[[*.mapping.score_function_heuristics]]`#
Key	Type	Description	Comments
function	`str`	Function name.	See built-in `heuristic_functions`.
mutate	`bool`	Keep changes made by function.	Disabled by default.

Note

Additional keys depend on the chosen function implementation.

As an example, the next snippet lets us match table columns such as animal_id to the id placeholder by using a value_fstring_alias() heuristic.

[[fetching.mapping.score_function_heuristics]]
function = "value_fstring_alias"
fstring = "{context}_{value}"

Hint

For difficult matches, consider using overrides instead of match scores.

Subsection: Overrides#

Overrides pin a mapping by hand, bypassing scoring: a name to a source in [translator.mapping.overrides], or a placeholder to a column in [fetching.mapping.overrides].

[translator.mapping.overrides]       # name -> source
created_by = "staff"

[fetching.mapping.overrides.staff]   # placeholder -> column, for the 'staff' source only
id = "staff_id"

Overrides are implemented by the InheritedKeysDict class. Top-level items in a [*.overrides]-section are shared, while a [*.overrides.<context-name>] subsection holds context-specific items.

Note

The type of context is determined by the class that owns the overrides. For fetchers, it is the source: each source has its own columns, so placeholder overrides may differ per source. The Translator has no context, since a name is matched against every source at once; its overrides must be top-level, and a subsection raises ConfigurationError.

The same structure supplies placeholder values for unknown IDs. This next snippet is from another example. For unknown IDs, the name placeholder is set to ‘Name unknown’ for the ‘name_basics’ source and ‘Title unknown’ for the ‘title_basics’ source. Both inherit the from and to placeholders, which are set to ‘?’.

[unknown_ids.overrides]
from = "?"
to = "?"

[unknown_ids.overrides.name_basics]
name = "Name unknown"
[unknown_ids.overrides.title_basics]
name = "Title unknown"

Warning

Overrides have no fixed keys, and the keys are not validated, so errors may be silent. The mapping process provides detailed information in debug mode, which may be used to discover issues.

Hint

Overrides may also be used to prevent mapping certain values: override them to "_".

Preventing unwanted mappings#

For example, assume that a SQL source table called title_basics has two columns, title and name, with identical contents. We would like to use a format '[{title}. ]{name}' to output translations such as ‘Mr. Astaire’. To avoid output such as ‘Top Hat. Top Hat’ for movies, we may add

[fetching.mapping.overrides.title_basics]
title = "_"

to the fetcher configuration.

An override to "_" blocks the match, so the fetcher does not fetch title for the title_basics source (TOML does not have a null-type). The id cannot be blocked: initialize_sources() raises. Other required placeholders cannot be blocked either: translating raises, and fetch_all() skips the source when selective_fetch_all=True (the default).

Section: Transformations#

Transformers are declared using [transform.'<source>'.'<transformer-type>'] sections. Subsection keys are passed directly to the __init__-method of the chosen transformer type.

For example, to configure a BitmaskTransformer for a permissions source, add a section on the form [transform.'<source>'.BitmaskTransformer] to an appropriate configuration file. The overrides are a list of tables because TOML keys must be strings:

[transform.permissions.BitmaskTransformer]
joiner = " AND "
overrides = [
    { id = 0, override = "NOT_SET" },
    { id = 0b1000, override = "OVERFLOW" },
]

This will create a transform that formats bitmasks such as 0b101 in the following way:

translator.translate((0b000, 0b101, 8), names="permissions")
("NOT_SET", "1:name-of-1 AND 4:name-of-4", "OVERFLOW")

Hint

Custom transformers may be initialized by using sections with fully qualified type names. For example, a [transform.'<source>'.'my.library.SuperTransformer']-section would import and initialize a SuperTransformer from the my.library module.

Chaining transformers#

You may specify any number of Transformers per source. Both the main and auxiliary files may contain [transform.'<source>'.'<transformer-type>'] sections for the same source. This creates a TransformerStack. The Translator owns all transformers, regardless of where they’re defined.

Priority:

Transformers declared in the same file run in declaration order.
Auxiliary fetcher files run before the main file.
Fetcher-provided transformers run before any declared for the same source.
Equal transformers – by identity, or by __eq__ – are the same transformer, so a provided one that is already in the chain is not added again.
Each transformer sees – and may overwrite – the effects of those before it.

Sections are keyed by type, so a transformer type may appear at most once per source and file. To chain two identically-typed transformers, declare them in different files or register them in code.

Programmatic transformer registration#

There are two ways to perform post-initialization transformer registration:

The Translator.register_transformer() method, and
The Fetcher.get_transformer() method.

Extending the create_translator() factory function (see rsundqvist/id-translation-project) to take advantage of register_transformer() to register bitmask sources is simple:

t = Translator(...)
t.initialize_sources()
for source in t.sources:
    if source.endswith("_bitmask"):
        t.register_transformer(source, BitmaskTransformer())
        t.register_transformer(source, CustomTransformer(), on_existing="append")

This assumes that naming is consistent. Custom fetchers may instead prefer to override Fetcher.get_transformer(), e.g. to register IntFlag enums.

The Translator.initialize_sources() method calls get_transformer() for all sources, so fetcher-provided transformers are used automatically.

Meta configuration#

The metaconf.toml-file must be placed next to the main TOML configuration file, and determines how other files are processed by the factory. See Metaconf for internal representation.

You rarely need this file: environment-variable interpolation (${VAR} / ${VAR:default}) is on by default. Add a metaconf.toml only to change that default. See ConfigMetadata for details.

Interpolation replaces ${VAR} in the text of the file before it is parsed, so the value becomes part of the TOML. Put interpolated strings in single quotes, which TOML reads literally:

read_path_format = '${DATA_DIR}/{}.csv'

In double quotes, a backslash or double quote in the value is read as TOML syntax: C:\temp\new would become a tab and a newline, and C:\data fails to parse. Outside quotes, the value is parsed as TOML, so return_all = ${RETURN_ALL:false} gives a boolean. Comments are interpolated too, so an unset variable that only a comment mentions still raises.

The `metaconf.toml` file format.#
Top-level section	Type	Description
`[env]`	`EnvConf`	Control environment-variable interpolation; `${VAR}` or `${VAR:default}`. Keys: `allow_interpolation` (default `true`), `allow_blank` (default `false`: raise if a variable resolves to an empty string, as `${VAR:}` does when `VAR` is unset), and `allow_nested` (default `false`).
`[equivalence]`	`EquivalenceConf`	Determines how equivalence between configuration files is determined (used by e.g. `load_persistent_instance()`).

The metaconf.toml-file is read as-is, without any preprocessing.

Custom TOML initialization#

All TOML configuration is interpreted by the TranslatorFactory class. To customize how different components are created, overwrite the all-caps factory properties of this class. For example, you may overwrite the TranslatorFactory.FETCHER_FACTORY attribute with your own implementation to customize how fetcher instances are created. Likewise, set TranslatorFactory.MAPPER_FACTORY to use custom Mapper implementations.

If your use case is not covered, consider opening an issue in the repository: rsundqvist/id-translation#issues

Configuration#

Minimal configuration#

Sections#

Section: Translator#

Section: Unknown IDs#

Section: Fetching#

Multiple fetchers#

Optional fetchers#

Caching#

Choosing a caching strategy#

Implementing CacheAccess#

Subsection: Mapping#

Filter functions#

Score function#

Score function heuristics#

Subsection: Overrides#

Preventing unwanted mappings#

Section: Transformations#

Chaining transformers#

Programmatic transformer registration#

Meta configuration#

Custom TOML initialization#

Implementing `CacheAccess`#