Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SimCLR with temperature schedule #1319

Open
4 tasks
guarin opened this issue Jul 14, 2023 · 1 comment
Open
4 tasks

Add SimCLR with temperature schedule #1319

guarin opened this issue Jul 14, 2023 · 1 comment
Labels

Comments

@guarin
Copy link
Contributor

guarin commented Jul 14, 2023

https://arxiv.org/abs/2303.13664 shows that varying the temperature in the contrastive loss during pretraining can help models learn better representations for datasets with long-tail distributions and doesn't hurt performance for datasets with uniform distributions. This could be an interesting method to add to the package.

I would start with running a benchmark on a normal dataset (ImageNet) as we don't have standard long-tail datasets in our benchmarks. It would also be interesting to see if it also gives good performance on full ImageNet as the paper only reports results for ImageNet100-LT. If it also works on default ImageNet it could be a good default method to add to most contrastive models.

TODO:

  • Add optional period to cosine_schedule
  • Add benchmark with SimCLR and temperature schedule
  • Run benchmark and report results
  • Add example and docs
@guarin
Copy link
Contributor Author

guarin commented Jul 14, 2023

Added it to #1172

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: No status
Development

No branches or pull requests

1 participant