We wanted to inform the community about new rate limiting measures implemented on vsx.aavso.org due to increased automated traffic from scrapers and AI bot indexing.
Current Implementation:
CAPTCHA verification may be triggered after sustained high-frequency requests
We are currently experimenting with thresholds to determine what is reasonable for legitimate use
Why This Was Necessary: The volume of automated requests has grown significantly, impacting site performance for legitimate users. Rate limiting helps ensure the site remains accessible and responsive for actual astronomical research and observations.
What This Means for You:
Normal browsing and research activities should be unaffected for most users
If you encounter CAPTCHAs frequently during regular use, please let us know so we can adjust the limits
Automated tools and scripts will need to respect these limits
Feedback Requested: We’re actively monitoring how this affects the community. If you find yourself hitting rate limits during legitimate research activities, please share your experience so we can fine-tune the system.
Thanks for your understanding as we work to maintain a stable and accessible resource for the variable star community.
@orionlee all resources served by vsx.aavso.org are now subject to a uniform rate limit. If you encounter a CAPTCHA, please let us know by either replying to this forum or emailing webmaster@aavso.org.
While I don’t expect the usage of my webapp [*] would hit rate limits, what’s the guidance for detecting CAPTCHA from API access?
In my case, the app gets the result in JSON from VSX. How should the codes be adjusted to detect CAPTCHA, e.g., via some specific HTTP status code?
The app won’t solve the CAPTCHA ( ), but I could log it so as to be informed CAPTCHA was hit (as opposed to some other intermittent or unexpected errors).
[*] The webapp lets users review data for individual TESS target (TIC), aggregating data over TESS, VSX, and a few other sources, e.g., Gaia. The volume is low (both in terms of frequency and number of requests each time).
At the moment, we are using AWS WAF to impose rate limiting. This may change in the future. If the request is within rate limits, you will get a standard 202 message and the communication will continue as normal. If you hit the CAPTCHA, you will receive a 405 with the x-amzn-waf-action ith a value of captcha set.
These kinds of changes are unglamorous, but valuable! I appreciate the work which goes into maintaining stability for VSX and the rest of the AAVSO’s services.
It might be nice to add some info about the rate limit to VSX’s “About” or “FAQ” webpages. Whenever I write scripts which query external resources, I tend to ask these questions:
What is the maximum rate? (If I have a list of stars, should I wait 0.1 seconds or 1 second between queries?)
Is there a minimum query volume? (If I’m only checking 3 stars, do I need to be careful to prevent my script from issuing the queries too fast?)
What is the cooldown? (If I accidentally trigger a captcha, how long should I wait before issuing more API queries?)
Thank you for those suggestions. We intend on releasing such guidance, but until we have a better idea of what rates are reasonable, we’ll remain silent on those topics.
kudos to Aru, Brian and the VSX team. In general, queries have been speedier this year!
I have not been able to login to VSX for about a week: After I login, it still says “Welcome Guest. You are not logged in.” Is it some known issue to be resolved?
Intermittently, I faced the same login issue in the past few months, but not for an extended period like this time.
First kudos to AAVSO and all of you who work to maintain systems and keep them running. I have not used the API for a while and noticed today that these limits where applied when a script I have written and used for years started having problems.
My use case is probably not unusual: I have an observation program comprising about 300 stars. Every once in while, specially at the start of my observing season (I live at 60°N so I start observing in late July when the sky gets somehow dark again) I plot light curves for all the stars in my program. This helps me understand what is going on in each star and also asses and identify possible observation errors I made and submitted. My script requests all observations of each star in the program in the last N days (N = variable depending star type - can be from 6 months to 3 years), I then plot these highlighting my observations.
The script has never had problems before requesting these observations. Today I found that the script stops at varying number of requests (30..50). After a stop, it seems that I remain blocked for a few minutes (approximate observation). I have been playing with sleep function, telling the script to wait between 2 to 5 seconds before making a new request but to no avail and this seems to work.
I recall that I had specifically asked in the old forums about which etiquette was expected on API request rate when I started doing this. I got the recommendation of using the “&caller=” tag. So, starting from today I use this tag passing my observer code along in it so AAVSO can identify these calls.
Hope this information helps you get a better idea of who is requesting data and for what purpose.
Yes, we’ve had to implement rate limiting on our API’s to reduce unauthorized access. We plan to move towards modern practices such as API keys in the future to better regulate access. Please send us an email if you would be interested in being one of our early adopters of the API.
), but I could log it so as to be informed CAPTCHA was hit (as opposed to some other intermittent or unexpected errors).