SPBERT MLM (Scratch)

Para más detalles, consulte nuestro repositorio de Github. Aquí hay un ejemplo en PyTorch:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('razent/spbert-mlm-zero')
model = AutoModel.from_pretrained('razent/spbert-mlm-zero')
text = 'select * where brack_open var_a var_b var_c sep_dot brack_close'
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

O utilizando TensorFlow:

from transformers import AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained('razent/spbert-mlm-zero')
model = TFAutoModel.from_pretrained('razent/spbert-mlm-zero')
text = 'select * where brack_open var_a var_b var_c sep_dot brack_close'
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

Citation:

@misc{tran2021spbert,
title={SPBERT: An Efficient Pre-training BERT on SPARQL Queries for Question Answering over Knowledge Graphs},
author={Hieu Tran and Long Phan and James Anibal and Binh T. Nguyen and Truong-Son Nguyen},
year={2021},
eprint={2106.09997},
archivePrefix={arXiv},
primaryClass={cs.CL}
}