concepts.benchmark.common.vocab.Vocab#

Bases: object

A simple vocabulary class.

Methods

`add`(word)	Add a word to the vocabulary.
`add_word`(word)	Add a word to the vocabulary.
`check_json_consistency`(json_file)	Check whether the vocabulary is consistent with a json file.
`dump_json`(json_file)	Dump the vocabulary to a json file.
`from_dataset`(dataset, keys[, extra_words, ...])	Generate a vocabulary from a dataset.
`from_json`(json_file)	Load a vocabulary from a json file.
`from_list`(dataset[, extra_words, single_word])	Generate a vocabulary from a list of strings.
`invmap_sequence`(sequence[, proc_be])	Map a sequence of indices to a sequence of words.
`map`(word)	Map a word to its index.
`map_fields`(feed_dict, fields)	Map the content in a specified set of fields in a dictionary to indices.
`map_sequence`(sequence[, add_be])	Map a sequence of words to a sequence of indices.
`words`()

Attributes

idx2word

A dictionary mapping indices to words.

__init__(word2idx=None)[source]#

Initialize the vocabulary.

Parameters:: word2idx – a dictionary mapping words to indices. If not specified, the vocabulary will be empty.

__iter__()[source]#

Return an iterator over the words in the vocabulary.

Return type:: Iterable[str]

__len__()[source]#

Return the size of the vocabulary.

Return type:: int

__new__(**kwargs)#

add(word)[source]#

Add a word to the vocabulary. Alias of add_word().

Parameters:: word (str)

add_word(word)[source]#

Add a word to the vocabulary.

Parameters:: word (str)

check_json_consistency(json_file)[source]#

Check whether the vocabulary is consistent with a json file.

Parameters:: json_file (str)
Return type:: bool

dump_json(json_file)[source]#

Dump the vocabulary to a json file.

Parameters:: json_file (str)

classmethod from_dataset(dataset, keys, extra_words=None, single_word=False)[source]#

Generate a vocabulary from a dataset.

Parameters:

dataset – the dataset to generate the vocabulary from.
keys (Sequence[str]) – the keys to retrieve from the dataset items.
extra_words (Sequence[str] | None) – additional words to add to the vocabulary.
single_word (bool) – whether to treat the values of the keys as single words.

Return type:

Vocab

classmethod from_json(json_file)[source]#

Load a vocabulary from a json file.

Parameters:: json_file (str)
Return type:: Vocab

classmethod from_list(dataset, extra_words=None, single_word=False)[source]#

Generate a vocabulary from a list of strings.

Parameters:

dataset (list) – the list of strings to generate the vocabulary from.
extra_words (Sequence[str] | None) – additional words to add to the vocabulary.
single_word (bool) – whether to treat the values of the keys as single words.

Return type:

Vocab

invmap_sequence(sequence, proc_be=False)[source]#

Map a sequence of indices to a sequence of words. If the argument proc_be is True, the begin-of-sentence and end-of-sentence tokens will be removed from the sequence.

Parameters:

sequence (Sequence[int] | Tensor) – the sequence of indices to map.
proc_be (bool) – whether to remove the begin-of-sentence and end-of-sentence tokens from the sequence.

Returns:

a list of words.

Return type:

List[str]

map(word)[source]#

Map a word to its index. If the word is not in the vocabulary, return the index of the unknown token.

Parameters:: word (str)
Return type:: int

map_fields(feed_dict, fields)[source]#

Map the content in a specified set of fields in a dictionary to indices. The argument fields is a list of keys in the dictionary to map. This function will modify the dictionary in-place.

Parameters:

feed_dict (dict) – the dictionary of fields to map.
fields (Sequence[str]) – the list of keys to map.

Returns:

a dictionary of mapped fields.

Return type:

dict

map_sequence(sequence, add_be=False)[source]#

Map a sequence of words to a sequence of indices. If the argument add_be is True, the begin-of-sentence and end-of-sentence tokens will be added to the sequence.

Parameters:

sequence (Sequence[str]) – the sequence of words to map.
add_be (bool) – whether to add the begin-of-sentence and end-of-sentence tokens to the sequence.

Returns:

a list of indices.

Return type:

List[int]

words()[source]#

Return type:: Iterable[str]

property idx2word: dict#: A dictionary mapping indices to words. This is a lazy property. It will be automatically recomputed when the length of the vocabulary changes.