We wanted to inform the community about new rate limiting measures implemented on vsx.aavso.org due to increased automated traffic from scrapers and AI bot indexing.
Current Implementation:
CAPTCHA verification may be triggered after sustained high-frequency requests
We are currently experimenting with thresholds to determine what is reasonable for legitimate use
Why This Was Necessary: The volume of automated requests has grown significantly, impacting site performance for legitimate users. Rate limiting helps ensure the site remains accessible and responsive for actual astronomical research and observations.
What This Means for You:
Normal browsing and research activities should be unaffected for most users
If you encounter CAPTCHAs frequently during regular use, please let us know so we can adjust the limits
Automated tools and scripts will need to respect these limits
Feedback Requested: We’re actively monitoring how this affects the community. If you find yourself hitting rate limits during legitimate research activities, please share your experience so we can fine-tune the system.
Thanks for your understanding as we work to maintain a stable and accessible resource for the variable star community.
@orionlee all resources served by vsx.aavso.org are now subject to a uniform rate limit. If you encounter a CAPTCHA, please let us know by either replying to this forum or emailing webmaster@aavso.org.
While I don’t expect the usage of my webapp [*] would hit rate limits, what’s the guidance for detecting CAPTCHA from API access?
In my case, the app gets the result in JSON from VSX. How should the codes be adjusted to detect CAPTCHA, e.g., via some specific HTTP status code?
The app won’t solve the CAPTCHA ( ), but I could log it so as to be informed CAPTCHA was hit (as opposed to some other intermittent or unexpected errors).
[*] The webapp lets users review data for individual TESS target (TIC), aggregating data over TESS, VSX, and a few other sources, e.g., Gaia. The volume is low (both in terms of frequency and number of requests each time).
At the moment, we are using AWS WAF to impose rate limiting. This may change in the future. If the request is within rate limits, you will get a standard 202 message and the communication will continue as normal. If you hit the CAPTCHA, you will receive a 405 with the x-amzn-waf-action ith a value of captcha set.
These kinds of changes are unglamorous, but valuable! I appreciate the work which goes into maintaining stability for VSX and the rest of the AAVSO’s services.
It might be nice to add some info about the rate limit to VSX’s “About” or “FAQ” webpages. Whenever I write scripts which query external resources, I tend to ask these questions:
What is the maximum rate? (If I have a list of stars, should I wait 0.1 seconds or 1 second between queries?)
Is there a minimum query volume? (If I’m only checking 3 stars, do I need to be careful to prevent my script from issuing the queries too fast?)
What is the cooldown? (If I accidentally trigger a captcha, how long should I wait before issuing more API queries?)
Thank you for those suggestions. We intend on releasing such guidance, but until we have a better idea of what rates are reasonable, we’ll remain silent on those topics.
kudos to Aru, Brian and the VSX team. In general, queries have been speedier this year!
I have not been able to login to VSX for about a week: After I login, it still says “Welcome Guest. You are not logged in.” Is it some known issue to be resolved?
Intermittently, I faced the same login issue in the past few months, but not for an extended period like this time.
), but I could log it so as to be informed CAPTCHA was hit (as opposed to some other intermittent or unexpected errors).