def tokenize()

in src/evaluation.py [0:0]

• 7 lines of code
• 5 McCabe index (conditional complexity)

    def tokenize(self, text, uncased=False):
        matches = [m for m in self._regexp.finditer(text)]
        if uncased:
            tokens = [m.group().lower() for m in matches]
        else:
            tokens = [m.group() for m in matches]
        return tokens