def encode()

in scripts_mgenre/preprocess_sentencepiece.py [0:0]

• 8 lines of code
• 6 McCabe index (conditional complexity)

    def encode(self, line):
        global sp, old2new
        ids = sp.encode_as_pieces(line)
        if old2new:
            ids = [x for old in ids for x in old2new[old]]
        if self.args.offset > 0:
            ids = [x + self.args.offset for x in ids]
        return list(map(str, ids))