in tensorflow_transform/analyzers.py [0:0]
def vocabulary(
x: common_types.TensorType,
top_k: Optional[int] = None,
frequency_threshold: Optional[int] = None,
vocab_filename: Optional[str] = None,
store_frequency: Optional[bool] = False,
weights: Optional[tf.Tensor] = None,
labels: Optional[tf.Tensor] = None,
use_adjusted_mutual_info: bool = False,
min_diff_from_avg: Optional[int] = None,
coverage_top_k: Optional[int] = None,
coverage_frequency_threshold: Optional[int] = None,
key_fn: Optional[Callable[[Any], Any]] = None,
fingerprint_shuffle: Optional[bool] = False,
file_format: common_types
.VocabularyFileFormatType = DEFAULT_VOCABULARY_FILE_FORMAT,
name: Optional[str] = None) -> common_types.TemporaryAnalyzerOutputType:
r"""Computes the unique values of a `Tensor` over the whole dataset.
Computes The unique values taken by `x`, which can be a `Tensor` or
`CompositeTensor` of any size. The unique values will be aggregated over all
dimensions of `x` and all instances.
In case `file_format` is 'text' and one of the tokens contains the '\n' or
'\r' characters or is empty it will be discarded.
If an integer `Tensor` is provided, its semantic type should be categorical
not a continuous/numeric, since computing a vocabulary over a continuous
feature is not appropriate.
The unique values are sorted by decreasing frequency and then reverse
lexicographical order (e.g. [('a', 5), ('c', 3), ('b', 3)]). This is true even
if `x` is numerical dtype (e.g. [('3', 5), ('2', 3), ('111', 3)]).
For large datasets it is highly recommended to either set frequency_threshold
or top_k to control the size of the output, and also the run time of this
operation.
When labels are provided, we filter the vocabulary based on the relationship
between the token's presence in a record and the label for that record, using
(possibly adjusted) Mutual Information. Note: If labels are provided, the x
input must be a unique set of per record, as the semantics of the mutual
information calculation depend on a multi-hot representation of the input.
Having unique input tokens per row is advisable but not required for a
frequency-based vocabulary.
WARNING: The following is experimental and is still being actively worked on.
Supply `key_fn` if you would like to generate a vocabulary with coverage over
specific keys.
A "coverage vocabulary" is the union of two vocabulary "arms". The "standard
arm" of the vocabulary is equivalent to the one generated by the same function
call with no coverage arguments. Adding coverage only appends additional
entries to the end of the standard vocabulary.
The "coverage arm" of the vocabulary is determined by taking the
`coverage_top_k` most frequent unique terms per key. A term's key is obtained
by applying `key_fn` to the term. Use `coverage_frequency_threshold` to lower
bound the frequency of entries in the coverage arm of the vocabulary.
Note this is currently implemented for the case where the key is contained
within each vocabulary entry (b/117796748).
Args:
x: A categorical/discrete input `Tensor` or `CompositeTensor` with dtype
tf.string or tf.int[8|16|32|64]. The inputs should generally be unique per
row (i.e. a bag of words/ngrams representation).
top_k: Limit the generated vocabulary to the first `top_k` elements. If set
to None, the full vocabulary is generated.
frequency_threshold: Limit the generated vocabulary only to elements whose
absolute frequency is >= to the supplied threshold. If set to None, the
full vocabulary is generated. Absolute frequency means the number of
occurrences of the element in the dataset, as opposed to the proportion of
instances that contain that element.
vocab_filename: The file name for the vocabulary file. If None, a file name
will be chosen based on the current scope. If not None, should be unique
within a given preprocessing function. NOTE To make your pipelines
resilient to implementation details please set `vocab_filename` when you
are using the vocab_filename on a downstream component.
store_frequency: If True, frequency of the words is stored in the vocabulary
file. In the case labels are provided, the mutual information is stored in
the file instead. Each line in the file will be of the form
'frequency word'. NOTE: if this is True then the computed vocabulary
cannot be used with `tft.apply_vocabulary` directly, since frequencies are
added to the beginning of each row of the vocabulary, which the mapper
will not ignore.
weights: (Optional) Weights `Tensor` for the vocabulary. It must have the
same shape as x.
labels: (Optional) Labels dense `Tensor` for the vocabulary. If provided,
the vocabulary is calculated based on mutual information with the label,
rather than frequency. The labels must have the same batch dimension as x.
If x is sparse, labels should be a 1D tensor reflecting row-wise labels.
If x is dense, labels can either be a 1D tensor of row-wise labels, or a
dense tensor of the identical shape as x (i.e. element-wise labels).
Labels should be a discrete integerized tensor (If the label is numeric,
it should first be bucketized; If the label is a string, an integer
vocabulary should first be applied). Note: `CompositeTensor` labels are
not yet supported (b/134931826). WARNING: When labels are provided, the
frequency_threshold argument functions as a mutual information
threshold, which is a float. TODO(b/116308354): Fix confusing naming.
use_adjusted_mutual_info: If true, and labels are provided, calculate
vocabulary using adjusted rather than raw mutual information.
min_diff_from_avg: MI (or AMI) of a feature x label will be adjusted to zero
whenever the difference between count and the expected (average) count is
lower than min_diff_from_average. This can be thought of as a regularizing
parameter that pushes small MI/AMI values to zero. If None, a default
parameter will be selected based on the size of the dataset (see
calculate_recommended_min_diff_from_avg).
coverage_top_k: (Optional), (Experimental) The minimum number of elements
per key to be included in the vocabulary.
coverage_frequency_threshold: (Optional), (Experimental) Limit the coverage
arm of the vocabulary only to elements whose absolute frequency is >= this
threshold for a given key.
key_fn: (Optional), (Experimental) A fn that takes in a single entry of `x`
and returns the corresponding key for coverage calculation. If this is
`None`, no coverage arm is added to the vocabulary.
fingerprint_shuffle: (Optional), (Experimental) Whether to sort the
vocabularies by fingerprint instead of counts. This is useful for load
balancing on the training parameter servers. Shuffle only happens while
writing the files, so all the filters above (top_k, frequency_threshold,
etc) will still take effect.
file_format: (Optional) A str. The format of the resulting vocabulary file.
Accepted formats are: 'tfrecord_gzip', 'text'. 'tfrecord_gzip' requires
tensorflow>=2.4. The default value is 'text'.
name: (Optional) A name for this operation.
Returns:
The path name for the vocabulary file containing the unique values of `x`.
Raises:
ValueError: If `top_k` or `frequency_threshold` is negative.
If `coverage_top_k` or `coverage_frequency_threshold` is negative.
If either `coverage_top_k` or `coverage_frequency_threshold` is specified
and `key_fn` is not.
If `key_fn` is specified and neither `coverage_top_k`, nor
"""
top_k, frequency_threshold = _get_top_k_and_frequency_threshold(
top_k, frequency_threshold)
if (coverage_top_k or coverage_frequency_threshold) and not key_fn:
raise ValueError('You must specify `key_fn` if you specify `coverage_top_k'
' or `coverage_frequency_threshold` in `vocabulary`.')
if key_fn and not (coverage_top_k or coverage_frequency_threshold):
raise ValueError('You must specify `coverage_top_k` or '
'`coverage_frequency_threshold` if you specify `key_fn` in'
' `vocabulary`.')
if file_format not in ALLOWED_VOCABULARY_FILE_FORMATS:
raise ValueError(
'"{}" is not an accepted file_format. It should be one of: {}'.format(
file_format, ALLOWED_VOCABULARY_FILE_FORMATS))
coverage_top_k, coverage_frequency_threshold = (
_get_top_k_and_frequency_threshold(
coverage_top_k, coverage_frequency_threshold))
if x.dtype != tf.string and not x.dtype.is_integer:
raise ValueError('expected tf.string or integer but got %r' % x.dtype)
if labels is not None and not labels.dtype.is_integer:
raise ValueError('expected integer labels but got %r' % labels.dtype)
if (frequency_threshold is None and labels is None and key_fn is None and
not fingerprint_shuffle and top_k is not None and
top_k <= LARGE_VOCAB_TOP_K):
logging.info('If the number of unique tokens is smaller than the provided '
'top_k or approximation error is acceptable, consider using '
'tft.experimental.approximate_vocabulary for a potentially '
'more efficient implementation.')
with tf.compat.v1.name_scope(name, 'vocabulary'):
vocabulary_key = vocab_filename
vocab_filename = _get_vocab_filename(vocab_filename, store_frequency)
informativeness_threshold = float('-inf')
coverage_informativeness_threshold = float('-inf')
if labels is not None:
if weights is not None:
vocab_ordering_type = _VocabOrderingType.WEIGHTED_MUTUAL_INFORMATION
else:
vocab_ordering_type = _VocabOrderingType.MUTUAL_INFORMATION
# Correct for the overloaded `frequency_threshold` API.
if frequency_threshold is not None:
informativeness_threshold = frequency_threshold
frequency_threshold = 0.0
if coverage_frequency_threshold is not None:
coverage_informativeness_threshold = coverage_frequency_threshold
coverage_frequency_threshold = 0.0
elif weights is not None:
vocab_ordering_type = _VocabOrderingType.WEIGHTED_FREQUENCY
else:
vocab_ordering_type = _VocabOrderingType.FREQUENCY
analyzer_inputs = _get_vocabulary_analyzer_inputs(
vocab_ordering_type=vocab_ordering_type,
x=x,
file_format=file_format,
labels=labels,
weights=weights)
return _vocabulary_analyzer_nodes(
analyzer_inputs=analyzer_inputs,
input_dtype=x.dtype.name,
vocab_ordering_type=vocab_ordering_type,
vocab_filename=vocab_filename,
top_k=top_k,
frequency_threshold=frequency_threshold or 0,
informativeness_threshold=informativeness_threshold,
use_adjusted_mutual_info=use_adjusted_mutual_info,
min_diff_from_avg=min_diff_from_avg,
fingerprint_shuffle=fingerprint_shuffle,
store_frequency=store_frequency,
key_fn=key_fn,
coverage_top_k=coverage_top_k,
coverage_frequency_threshold=coverage_frequency_threshold or 0,
coverage_informativeness_threshold=coverage_informativeness_threshold,
file_format=file_format,
vocabulary_key=vocabulary_key)