sotabench: Benchmarking Every Open Source Model

Each public benchmark has its own instructions on how to use. For example, to use the Image Classification on ImageNet benchmark on your model in a framework-independent way, create a sotabench.py file like this: Example sotabench.py structure from sotabencheval.image_classification import ImageNetEvaluator evaluator = ImageNetEvaluator( # automatically compare to this paper model_name='ResNeXt-10

[Ryobot]

GitHubのレポジトリに各ベンチマークのevaluatorをimportしたsotabench.pyを追加すると，性能や速度のスコアを自動算出してpaperの記録と比較してくれる画期的なサービス．ところでWMT'14のfairseq Trm-Bigが全くpaperの性能が出てない