langml.baselines.contrastive.simcse.dataloder

Module Contents

Classes

DataLoader

TFDataLoader

class langml.baselines.contrastive.simcse.dataloder.DataLoader(data: List, tokenizer: object, batch_size: int = 32)[source]

Bases: langml.baselines.BaseDataLoader

__len__(self) → int[source]
static load_data(fpath: str, apply_aeda: bool = True, aeda_tokenize: Callable = whitespace_tokenize, aeda_language: str = 'EN') → Tuple[List[Tuple[str, str]], List[Tuple[str, str, int]]][source]
Parameters

  • fpath – str, path of data

  • apply_aeda – bool, whether to apply the AEDA technique to augment data, default True

  • aeda_tokenize – Callable, specify aeda tokenize function, it works when set apply_aeda=True

  • aeda_language – str, specifying the language, it works when set apply_aeda=True

make_iter(self, random: bool = False)[source]
class langml.baselines.contrastive.simcse.dataloder.TFDataLoader(data: List, tokenizer: object, batch_size: int = 32)[source]

Bases: DataLoader

make_iter(self, random: bool = False)[source]
__call__(self, random: bool = False)[source]