integration.pandas#

Integration for Pandas types.

Module Attributes

PandasT

Supported pandas types.

Classes

PandasIO(*[, level, missing_as_nan, ...])

Optional IO implementation for pandas types.
class PandasIO(*, level=-1, missing_as_nan=None, as_category=False, ordered='name', observed=False)[source]#

Bases: DataStructureIO[PandasT, NameType, SourceType, IdType]

Optional IO implementation for pandas types.

Parameters:

  • level – Column level to use as names when translating a DataFrame with MultiIndex columns. See pandas.MultiIndex.get_level_values() for details. Ignored otherwise.

  • missing_as_nan – If set, unknown IDs will be NaN. Grouping operations will typically drop NaN values. If False, placeholders such as '<Failed: id=-1>' will be used instead. Default is True if as_category=True, False otherwise.

  • as_category – Set dtype=’category’ in the result. See Categorical translation for details.

  • ordered – Category sort order. Ignored unless as_category=True. See Categorical translation.

  • observed – Keep only categories present in the data. Ignored unless as_category=True.

Categorical translation#

Setting as_category=True converts the resultant translations to a categorical data type, with the categories set to all real translations. If missing_as_nan=False, the categories may also include placeholders.

The ordered argument sets the category order, and whether the returned pandas.CategoricalDtype is ordered:

  • 'name': Sort by translated name. The default.

  • 'id': Sort by ID. Use to keep e.g. numeric IDs in their natural order, rather than the lexicographic order of the translations they produce.

  • False: Sort by translated name, but leave the dtype unordered. Use when ordered dtypes are in the way; pandas refuses to combine ordered categoricals that don’t share their categories.

Certain fetchers, such as the MemoryFetcher(return_all=True), will return more IDs than requested. In this case the categories may also include values not present in the input data. This may also happen if data was prepared with Translator.go_offline, or if multiple columns were mapped to the same source. Set observed=True to drop these; the categories are then determined by the IDs in the input rather than by what the fetcher returned.

Note

Observed categories are data-dependent, so two vectors translated from the same source no longer necessarily share them. Ordered categoricals that do not share categories cannot be combined by pandas.

Arguments are passed using io_kwargs, e.g. of Translator.translate.

>>> import pandas as pd
>>> from id_translation import Translator
>>> animals = {0: "Tarzan", 1: "Morris", 2: "Simba"}
>>> translator = Translator({"animals": animals})
>>> df = pd.DataFrame({"animals": [0, 2, 0, -1]})

Translate as categories.

>>> result = translator.translate(
...     df,
...     io_kwargs={"as_category": True},
... )
>>> result["animals"]
0    0:Tarzan
1     2:Simba
2    0:Tarzan
3         NaN
Name: animals, dtype: category
Categories (3, str): ['0:Tarzan' < '1:Morris' < '2:Simba']

The unknown ID -1 is NaN since missing_as_nan defaults to True when as_category=True. The '1:Morris' category comes from the fetcher rather than the df, so it survives as an empty group.

>>> result.groupby("animals", observed=False).size()
animals
0:Tarzan    2
1:Morris    0
2:Simba     1
dtype: int64

Use observed=True to drop '1:Morris', keeping only the values that appear in the df.

>>> translator.translate(
...     df,
...     io_kwargs={"as_category": True, "observed": True},
... )["animals"].cat.categories.to_list()
['0:Tarzan', '2:Simba']

Categories are sorted by translated name. Use ordered='id' to sort by ID instead, e.g. to stop '10:Morris' from sorting before '2:Simba'. The name-only Format below makes the difference visible; sorting by name would give ['Morris', 'Simba', 'Tarzan'].

>>> result = translator.translate(
...     df,
...     fmt="{name}",
...     io_kwargs={"as_category": True, "ordered": "id"},
... )
>>> result["animals"].cat.categories.to_list()
['Tarzan', 'Morris', 'Simba']

Setting missing_as_nan=False keeps unknown IDs, adding their placeholders to the categories.

>>> translator.translate(
...     df,
...     io_kwargs={"as_category": True, "missing_as_nan": False},
...     copy=False,
... )
>>> df["animals"].cat.categories.to_list()
['0:Tarzan', '1:Morris', '2:Simba', '<Failed: id=-1>']
extract(translatable, names)[source]#

Extract IDs from translatable.

Parameters:

  • translatable – Data to extract IDs from.

  • names – List of names in translatable to extract IDs for.

Returns:

A dict {name: ids}.

classmethod handles_type(arg)[source]#

Return True if the implementation handles data for the type of arg.

insert(translatable, names, tmap, copy)[source]#

Insert translations into translatable.

Parameters:

  • translatable – Data to translate. Modified iff copy=False.

  • names – Names in translatable to translate.

  • tmap – Translations for IDs in translatable.

  • copy – If True, modify contents of the original translatable. Otherwise, returns a copy.

Returns:

A copy of translatable if copy=True, None otherwise.

Raises:

NotInplaceTranslatableError – If copy=False for a type which is not translatable in-place.

names(translatable)[source]#

Extract names from translatable.

Parameters:

translatable – Data to extract names from.

Returns:

A list of names to translate. Returns None if names cannot be extracted.

priority = 1999#

Determines order in which IOs are considered (higher = earlier).

Set priority < 0 to disable.

class PandasT#

Supported pandas types.

alias of TypeVar(‘PandasT’, ~pandas.DataFrame, ~pandas.Series, ~pandas.Index, ~pandas.MultiIndex)