def above_lang_threshold()

in fineweb-2-pipeline.py [0:0]

• 2 lines of code
• 1 McCabe index (conditional complexity)

    def above_lang_threshold(doc, threshold):
        return doc.metadata["language_score"] >= threshold