"ENVIRONMENTAL AND FINANCIAL COST
Strubell et al. recently benchmarked model training and develop-
ment costs in terms of dollars and estimated CO2 emissions [129].
While the average human is responsible for an estimated 5t CO2
per year, the authors trained a Transformer (big) model [136] with
neural architecture search and estimated that the training procedure
emitted 284t of CO2. Training a single BERT base model (without
hyperparameter tuning) on GPUs was estimated to require as much
energy as a trans-American flight.""