Cost Comparison of Deep Learning Hardware: Google TPUv2 vs. Nvidia Tesla V100

Here is a larger-scale comparison of Cloud TPU and Google Cloud GPU performance and cost (focused on Cloud TPU Pods): <a href="https:&#x2F;&#x2F;cloud.google.com&#x2F;blog&#x2F;products&#x2F;ai-machine-learning&#x2F;now-you-can-train-ml-models-faster-and-lower-cost-cloud-tpu-pods" rel="nofollow">https:&#x2F;&#x2F;cloud.google.com&#x2F;blog&#x2F;products&#x2F;ai-machine-learning&#x2F;n...</a><p>All the code used in that comparison is open source, and there is a detailed methodology page with instructions that you can follow if you want to reproduce the results: <a href="https:&#x2F;&#x2F;github.com&#x2F;tensorflow&#x2F;tpu&#x2F;blob&#x2F;master&#x2F;benchmarks&#x2F;ResNet-50_v1.5_Performance_Comparison_TensorFlow_1.12_GCP.md" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;tensorflow&#x2F;tpu&#x2F;blob&#x2F;master&#x2F;benchmarks&#x2F;Res...</a><p>Also, Cloud TPUs are available to everyone for free via Colab. Here is a sample Colab that shows how to train a Keras model on the Fashion MNIST dataset using the Adam optimizer: <a href="https:&#x2F;&#x2F;colab.research.google.com&#x2F;github&#x2F;tensorflow&#x2F;tpu&#x2F;blob&#x2F;master&#x2F;tools&#x2F;colab&#x2F;fashion_mnist.ipynb" rel="nofollow">https:&#x2F;&#x2F;colab.research.google.com&#x2F;github&#x2F;tensorflow&#x2F;tpu&#x2F;blob...</a><p>(I work on Cloud TPUs)

This doesn&#x27;t seem like a very informative benchmark to me. They don&#x27;t mention how&#x2F;whether they tuned the learning rates and batch sizes to optimize for each different device. Like they mention, they also use a very small network that isn&#x27;t something you need the power of a TPU to train quickly and may scale differently than a large network.<p>They also don&#x27;t post their code so I can&#x27;t check that their problems with ADAM aren&#x27;t due to using L2 regularization, which <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1711.05101" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1711.05101</a> shows leads to worse performance than SGD and you should use weight decay instead.

Looks like riseml has shut down and taken their comparison post down. I was hoping to compare the results.

Hmm interesting, is it possible that the batch size for the tpu is larger? I&#x27;m guessing they might be using some sort of large batches to populate their giant GEMM cores

My understanding is that Teslas have scientific accuracy that is not needed for deep learning... my machine has nvidia 1080s.