Is there any website maintenance going on? I’m having APASS calls timing out. For example https://www.aavso.org/cgi-bin/apass_dr10_download.pl?ra=359.6&dec=61.2&radius=0.5&outtype=1
thx
Cliff
Greetings,
I would like to provide you with an update regarding the issues we’ve been experiencing during the last two weeks.
Over the weekend, I was able to identify, patch, test, and deploy a fix for the GCVS name resolution issue mentioned above. This was a fairly arduous bug to track down because it was part of a cron job running on a server that is scheduled for decommissioning. The code on that machine had been frozen in place in order to retain functionality while the corresponding repository moved forward considerably. This specific issue had been identified and fixed upstream, but the patch was never backported and applied to this machine.
Issues like this are fairly difficult for us to track down because AAVSO’s IT systems are extremely complex for an organization of our size. At present, AAVSO operates 25 databases that are accessed by 75 applications written in 7 different programming languages spread across 8 servers. Many of these systems were put in place without much documentation, so they have interdependencies that we’re still uncovering. Our Strategic Goals for 2025 - 2029 will have us reduce this complexity, but with a budget that only allows for us to have one full-time software developer on staff, it will take considerable time before things are improved.
To @MikeWiles question about planned maintenance outages: We normally deploy new / updated software every Thursday. If we anticipate outages, we announce them in advance using a banner on www.aavso.org as well as the Community forum here.
I appreciate your patience and understanding and we identify and resolve these issues.
Kind regards,
Brian Kloppenborg
Executive Director, AAVSO
Also VSX looks down, I’m unable to get data or access the search page
Also I’m unable to download charts
I’m also having issues with the LCG, not returning anything when I hit “Send”.
Yes, some website maintenance might be undergoing, or something else is failing.
The VPhot Transform Applier tool is taking forever to complete.
Ari
Greetings,
After reading the forum posts here I spent most of my evening diagnosing the issue. I’ve tracked the issue down to a series of five queries that are overwhelming the database server. I’ve implemented a patch to the most expensive query that improves its performance by approximately 40x. I have a general idea where the other queries are located in code, but it will take me a while to track those down.
I’ll try to deploy the fixes this evening, but I don’t have the relevant software and development environment installed, so it isn’t clear I’ll be able to do this. If not, I’ve asked Aru to pull my code, test it, and deploy it when he wakes up tomorrow morning.
Brian Kloppenborg
Executive Director, AAVSO
Hi, Brian,
Thank you for your efforts.
I just wanted to inform you, and Aru, that the issue with the Transform Applier persists.
Cheers,
Ari
Hi Ari,
Vphot is a separate platform from VSX and other AAVSO databases so some service can be ok but other down at the same time.
By the way I just checked the transform applier in Vphot and it’s working fine with my files, do you are trying to convert a very big dataset?
Greetings,
Last night Aru and I were able to track down the source of the five queries I mentioned above.
One query was the result of a long-standing bug in our GCVS object name resolution code. I’ve implemented a patch for this, but we haven’t deployed it as the same issue appears in multiple locations in our code. We’re refactoring the code right now.
The other four queries were caused by bots indexing the (paginated) data search application, resulting in hundred of queries against the AID per bot. Aru has made a change that mitigates this issue.
This morning I also found the VSX server’s network connection was down. This machine hosts several of our APIs, so it likely caused VStar and TA to not function correctly. This is a really odd issue because VSX is a virtualized server in a major datacenter. Nevertheless, a reboot got it up and running.
At the moment, the database, www, apps, and vsx are all functional. Please let me know if any further issues persist.
Again, I apologize for the service outage and appreciate your understanding regarding these matters.
Brian
Hi,
I found my way here from the previous “Problems with VSX?” thread. It appears to be down again, or there’s some sort of maintenance going on. Is there a place somewhere on the AAVSO site that indicates planned maintenance outages for the various applications?
Thanks,
Mike Wiles
WMEB
Unfortunately, I’m experiencing issues with the Variable Star Plotter at present, with persistent Internal Server Error messages.
I am also getting errors from VSP trying to get comps from Vphot
Good morning. I wonder if someone has been changing web service call permissions? The AAVSOnet server began getting 403 responses to our query of the the telescopes https://aavso.org/apps/aavsonet/telescopes2/ late yesterday EST.
Greetings,
Over the last 48 hours, Aru, Bert, Matt Craig, Patrick Wils, George Silvis, and I have been busy tracking down problematic sections of code, profiling long running queries, and implementing a variety of changes to resolves these problems. While there is still more work to do, I believe the most egregious issues have been resolved.
The specific changes we’ve made include adding local caching to some applications, introducing additional indexes to the AID and VSX databases to accelerate queries, modified our approach to certain problems to reduce the load on the database server, rewrote some queries to utilize new database indexes, and changed our approach to bot detection. Most of our applications now include software that automatically logs error conditions and provides user experience metrics. This additional tool will let us identify and concentrate our efforts on the worst portions of our website.
We’ve been monitoring our servers closely for the last 24 hours and things are looking good. As a result of our efforts, the average number of active sessions on our database has decreased from ~150 to 5. This difference is due almost exclusively to the team’s work on a single section of very heavily used code. We’ve even received reports that users are seeing significantly better response times to VSX queries!
Please let us know if you experience any additional issues. We’ll work as quickly as we can to resolve them.
Brian
Thanks for your work… just did some analysis and everything ran smoothly