`langml.plm`

Submodules

Package Contents

Classes

`TokenEmbedding`
`EmbeddingMatching`
`Masked`	Generate output mask based on the given mask.

Functions

`load_bert`(config_path: str, checkpoint_path: str, seq_len: Optional[int] = None, pretraining: bool = False, with_mlm: bool = True, with_nsp: bool = True, lazy_restore: bool = False, weight_prefix: Optional[str] = None, dropout_rate: float = 0.0, **kwargs) → Union[Tuple[langml.tensor_typing.Models, Callable], Tuple[langml.tensor_typing.Models, Callable, Callable]]	Load pretrained BERT/RoBERTa
`load_albert`(config_path: str, checkpoint_path: str, seq_len: Optional[int] = None, pretraining: bool = False, with_mlm: bool = True, with_nsp: bool = True, lazy_restore: bool = False, weight_prefix: Optional[str] = None, dropout_rate: float = 0.0, **kwargs) → Union[Tuple[langml.tensor_typing.Models, Callable], Tuple[langml.tensor_typing.Models, Callable, Callable]]	Load pretrained ALBERT

Attributes

custom_objects

class langml.plm.TokenEmbedding[source]

Bases: tensorflow.keras.layers.Embedding

static get_custom_objects() → dict

compute_mask(self, inputs: langml.tensor_typing.Tensors, mask: Optional[langml.tensor_typing.Tensors] = None) → List[Union[langml.tensor_typing.Tensors, None]]

call(self, inputs: langml.tensor_typing.Tensors) → List[langml.tensor_typing.Tensors]

compute_output_shape(self, input_shape: langml.tensor_typing.Tensors) → List[langml.tensor_typing.Tensors]

class langml.plm.EmbeddingMatching(initializer: langml.tensor_typing.Initializer = 'zeros', regularizer: Optional[langml.tensor_typing.Regularizer] = None, constraint: Optional[langml.tensor_typing.Constraint] = None, use_bias: bool = True, use_softmax: bool = True, **kwargs)[source]

Bases: tensorflow.keras.layers.Layer

get_config(self) → dict

build(self, input_shape: langml.tensor_typing.Tensors)

compute_mask(self, inputs: langml.tensor_typing.Tensors, mask: Optional[langml.tensor_typing.Tensors] = None) → langml.tensor_typing.Tensors

call(self, inputs: langml.tensor_typing.Tensors, mask: Optional[langml.tensor_typing.Tensors] = None, **kwargs) → langml.tensor_typing.Tensors

static get_custom_objects() → dict

compute_output_shape(self, input_shape: langml.tensor_typing.Tensors) → langml.tensor_typing.Tensors

class langml.plm.Masked(return_masked: bool = False, **kwargs)[source]

Bases: tensorflow.keras.layers.Layer

Generate output mask based on the given mask. https://arxiv.org/pdf/1810.04805.pdf

static get_custom_objects() → dict

get_config(self) → dict

compute_mask(self, inputs: langml.tensor_typing.Tensors, mask: Optional[langml.tensor_typing.Tensors] = None) → Union[List[Union[langml.tensor_typing.Tensors, None]], langml.tensor_typing.Tensors]

call(self, inputs: langml.tensor_typing.Tensors, mask: Optional[langml.tensor_typing.Tensors] = None, **kwargs) → langml.tensor_typing.Tensors

compute_output_shape(self, input_shape: langml.tensor_typing.Tensors) → Union[List[langml.tensor_typing.Tensors], langml.tensor_typing.Tensors]

langml.plm.load_bert(config_path: str, checkpoint_path: str, seq_len: Optional[int] = None, pretraining: bool = False, with_mlm: bool = True, with_nsp: bool = True, lazy_restore: bool = False, weight_prefix: Optional[str] = None, dropout_rate: float = 0.0, **kwargs) → Union[Tuple[langml.tensor_typing.Models, Callable], Tuple[langml.tensor_typing.Models, Callable, Callable]][source]

Load pretrained BERT/RoBERTa :param - config_path: str, path of albert config :param - checkpoint_path: str, path of albert checkpoint :param - seq_len: Optional[int], specify fixed input sequence length, default None :param - pretraining: bool, pretraining mode, default False :param - with_mlm: bool, whether to use mlm task in pretraining, default True :param - with_nsp: bool, whether to use nsp task in pretraining, default True :param - lazy_restore: bool, whether to restore pretrained weights lazily, default False.

Set it as True for distributed training.

Parameters

weight_prefix (-) – Optional[str], prefix name of weights, default None. You can set a prefix name in unshared siamese networks.
dropout_rate (-) – float, dropout rate, default 0.

Returns

keras model - bert: bert instance - restore: conditionally, it will return when lazy_restore=True

Return type

model

langml.plm.load_albert(config_path: str, checkpoint_path: str, seq_len: Optional[int] = None, pretraining: bool = False, with_mlm: bool = True, with_nsp: bool = True, lazy_restore: bool = False, weight_prefix: Optional[str] = None, dropout_rate: float = 0.0, **kwargs) → Union[Tuple[langml.tensor_typing.Models, Callable], Tuple[langml.tensor_typing.Models, Callable, Callable]][source]

Load pretrained ALBERT :param - config_path: str, path of albert config :param - checkpoint_path: str, path of albert checkpoint :param - seq_len: Optional[int], specify fixed input sequence length, default None :param - pretraining: bool, pretraining mode, default False :param - with_mlm: bool, whether to use mlm task in pretraining, default True :param - with_nsp: bool, whether to use nsp/sop task in pretraining, default True :param - lazy_restore: bool, whether to restore pretrained weights lazily, default False.