Hi! Status update here from the AAVSO Smart Telescope Working Group.
This group (about 12 people) was created 14 months ago with three broad goals:
Figure out what kind of photometric quality is possible with Smart Telescopes; is it good enough to include in AAVSO historical data?
Can the photometric process be made simple enough for a typical smart telescope owner to perform variable star measurement?
Can we make the image and data analysis pipeline efficient enough that the AAVSO can afford to process smart telescope data?
At this point, we’ve got favorable answers to all three questions:
With appropriate averaging of data from multiple images, we can consistently get photometric error below 0.02 magnitudes (measured against Landolt standard stars) from the Seestar S50 (definitely) and the other smart telescopes (probably).
Most of the data reduction process can be automated (it’s actually an important part of getting that photometric quality), with people providing important quality control oversight.
By converting smart telescope images into “starlists” as quickly as possible after the image has been captured (preferably, inside the smart telescope), data transmission and storage costs to the AAVSO drop by as much as a factor of 100x.
Along the way, we’ve developed a set of Python tools that are currently being used to convert smart telescope images into starlists (involves identifying stars in the images, plate-solving to extract sky location for each star, and performing photometry to get instrumental magnitudes for each star), and then to do quality analysis (including averaging) of the resulting starlists. (We don’t yet have a tool for measuring variable star brightness from starlists.) Those tools allowed us to build graphs (like the following one that shows brightest and faintest usable stars from the Seestar S50 as a function of camera gain setting – the default is a setting of 80).
The tools have also given us an ability to identify the sources of many types of photometric measurement error, and adjust our algorithms to take advantage of that insight. (For example, the Bayer color filter introduces a centroiding error when estimating star centers that has a significant effect on properly centering the photometry measurement aperture on the star in de-Bayered images – at least for now, we’ve substituted a different technique for separating flux by color instead of traditional de-Bayering.)
We are currently identifying and removing code execution bottlenecks in our existing Python tools to speed them up. We are also deep in the planning and architecture of the AAVSO’s starlist processing pipeline which will be used to process starlists uploaded to the AAVSO from smart telescopes.
… and if anyone has at least several hours a week available to give us a hand, please reach out. We are looking for people with Python experience (with or without exposure to Django), someone to help us design a testing approach for our processing pipeline, someone to help with project management, someone to help design some browser-based GUIs, someone with experience building database schemas, someone with experience interpreting executing time profiling data, someone who’d like to write some documentation, …
Is the existing Python code available, possibly in GitHub? I’m particularly interested in the centroiding algorithm as I worked on that area in my guiding software some 20 years ago. Has any testing been done with the Seestar S30? My wife has an S50 and I have an S30 and I’m interested what can be done with the S30.
Steve:
The source code for our tools is currently in github, but it’s not in a public repository (it’s private to AAVSO). We’ve been actively talking about splitting it into two pieces, one of which – the bigger piece – would be public, but it hasn’t happened yet. (We need to put some kind of a protection in place so that the interface specification for starlists can’t be hijacked and changed by some other organization.) I’d say, “stay tuned.”
The new stuff we’ve done with centroiding is all specific to one-shot-color sensors, and has two parts: what you do with all the pixels (all 4 color channels), and what you do one color channel at a time. (Our goal is to calculate centroids once per image, and to propagate those centroids to each of the color channels so that we don’t calculate centroids from scratch for each color channel – that was the biggest error source.)
I think we have an S30 owner in the working group, although I haven’t seen any S30 data yet. We have to be a little cautious as we move to the larger field of view of the S30 – there are some field-of-view thresholds that have given us some trouble in the past few months, mostly to do with the way we query external catalogs to support the processing of the starlists. (Some external catalog servers have field-of-view limits because of the number of stars that would be returned from a query. There are several ways to deal with that, but we’ve been putting it off simply because we’re overloaded with things to do.)
I’ve contributed thousands of observations to the AID already over the past 12-18 months using a Seestar S50. Many others have as well. I’m wondering if the Group has considered as to whether the 2nd and 3rd goals may have already been satisfied by existing third party software. In my experience the answer is yes. I have a very simple and efficient workflow and there’s no burden on AAVSO systems to process the data. I just upload text files of data. I’ve also been satisfied with the data quality as per the first goal, although I will be interested to see the output from the Group in terms of tools that I could use to check on that quality.
We are aware of (and regularly use) third-party tools as well as VPhot. And you’re correct that third-party tools provide data with an absolutely minimal cost to the AAVSO.
There are two broad reasons that we are pursing an alternative analysis approach.
First, the average smart telescope user’s knowledge and experience is quite different from that of the average AAVSO observer today. The AAVSO has no intention of forcing anyone to use the alternative analysis approach we’re building; if you want to analyze your images with VPhot or a third-party tool, please do so. But we know that many smart telescope users know little about bias, dark, and flat images, have trouble pulling raw images out of their telescopes, don’t understand what color correction is all about, and don’t recognize the significance of airmass. By working with the smart telescope vendors, we aim to give these newcomers a path that gives good results despite a lack of experience.
Second, there are technical reasons for us to offer an alternative to the third-party tools:
The smart telescopes don’t all behave the same way. For example, we know that the Dwarf III is currently putting an incorrect value into the DATE-OBS field of its raw image FITS headers. Our software recognizes this and corrects the observation time.
As another example, each vendor has a different definition of a “raw” image. The Celestron Origin, for instance, stores its “raw” images with a white-balance color correction factor already applied. This gives Origin bias images a unique appearance and means that Origin image calibration requires some special care; it also affects the way we find star centroids in Origin images.
We know (from the AAVSO Data Quality Task Force) that most AAVSO observers are not applying color corrections, even when using third-party tools. We also know that it’s impossible to achieve the overall photometric accuracy I quoted in my first message without color corrections. Our software automatically takes care of this for the observer.
One of the other findings from the AAVSO Data Quality Task Force is a frequent mismatch between observers’ measurement cadence and the behavior of the stars that they observe. Our software fixes this in a way that adjusts reporting cadence for each reportable star in an image, even when secondary variables have a need for a different measurement cadence(s) than the primary variable.
–Mark (MMU)
Could you expand a little on what exactly you mean by “color correction” in this context? For my Seestar images I just debayer the individual frames and then stack and do aperture photometry on the TG images (and report the results as untransformed TG filter observations). I use Tycho Tracker to do the work.
Hi, Brian:
We use a variety of phrases to refer to the process of adjusting a measured magnitude from the color system of the instrument that made the measurement into a “standard color” system. Sometimes we say “color corrections”; sometimes we say “color transformations”; sometimes we just say “transform”.
Because stars have different spectral shapes (different relative amounts of energy at different colors, caused – in part – by having different black body temperatures), and because everyone’s cameras (and filters) are just a little bit different from everyone else’s cameras (and filters), when one person says that star A and star B have exactly the same brightness, few other people will agree, particularly if star A and star B have different colors.
In order to get people to agree on the brightness of stars that have different colors, each observer must apply a correction to each magnitude measurement they make. That correction “transforms” their measurement from their instrumental color system to a “standard color” system. The transformation equation is different for every combination of telescope, filter, camera, and atmospheric condition. The transformation equation is derived by observing many (hundreds or thousands) of stars of known brightness and color.
The description of your workflow suggests that you are not making this color correction. (And that puts you in with the majority of AAVSO observers today.) By submitting your observations as “untransformed”, you’re letting us (and all the researchers who pull your data) know that you haven’t applied color corrections.
The STWG has measured the error associated with these color corrections (or, more precisely, with the lack of these color corrections). That error combines with other sources of error to become the total error inherent in your measurement. Depending on how small you’ve succeeded in getting your other errors, the color-related error can either be totally insignificant or can be the dominant remaining error source. (Errors don’t combine in a simple additive way; most error sources combine as the square root of the sum of the squares of the individual errors.)
I have no idea what your total measurement error is (particularly since it varies star-by-star). And so I can’t tell you how much you would gain by adding color transformations. I do know that once the STWG got through applying all our adjustments and optimizations to the photometry process, we reduced residual measurement error to the point where color-related errors were larger than the other remaining (uncorrected) errors in our Seestar S50 photometry. And so it became very important for us to include color corrections in our processing workflow. (Although I will openly admit to our own confusion in the beginning: color corrections didn’t seem to make any difference at all. Eventually, we realized that there were other error sources far bigger than color corrections that were effectively swamping the benefits of the color-related stuff. Once we got those other things under control in a repeatable way, the value of the color corrections became more visible and we breathed a sigh of relief. )
Thanks for the clarification; I thought you might be referring to transformation but I wasn’t sure. I haven’t tried transforming my data but it is something that I’ve been meaning to try to do. Do many people transform TG observations?
Very interesting!
I have a S50 and i give some results to AAVSO without color transformation for now. Concerning S50 performances, can you remind me the conditions for determining magnitude limits, please?
I imagine that saturation or SNR comes into play, but what limits you use?
Saturation :ADU Max (individual Subs) >60000 ?
60000 is the approximativ limit I find for a good linearity inst vs standard.
Faint: SNR (stacked subs)> 3 for captor limit ? 100 or perhaps less for measuring?
That magnitude limit graph (in my first post in this thread) was kind of fun to make. George Silvis (SGEO) spent a while with his S50 one night making about 250 images of the field surrounding V503 Her. He varied the gain setting of the Seestar from 0 up to 100 as he grabbed 10-second images.
We then used our image-to-starlist converter tool to create a starlist for each image. (It actually created four starlists for each image: a blue, red, green, and luminance starlist, but we only used the luminance starlist for that graph.) We fed the starlists into another of our tools which cross-referenced (correlated) all of the stars in the image against several different star catalogs. We used the information from the APASS-10 catalog to build the graph.
At the top of the graph, we did a search across all the starlists for stars that were saturated. (Detecting saturation is somewhat different in each of the different smart telescope models. We chose the S50 for this particular test because saturation in the S50 is really straightforward: for all settings of the camera, saturation coincides with a peak pixel level of 65535. Yes, I know the S50 sensor uses a 12-bit A/D converter that’s incapable of counting to 65535. This is all part of the way smart telescope vendors save their “raw” images in a format that isn’t particularly “raw”.) All the stars in all the images were put into three categories:
Stars that never saturated in any image (several hundred stars).
Stars that saturated in every image (one star).
Stars that saturated in some, but not all images (15 stars).
We ignored stars in categories 1 and 2. For stars in category 3, we calculated the fraction of the time that the star saturated (e.g., Star X saturated in 16 of 23 images at a gain setting of 40). We plotted those fractions against star magnitude, and from those plots we could pretty easily find the magnitude at which stars have a 50/50 probability of saturating. Those magnitudes became our upper limit as a function of the gain setting. (By the way, this creates a huge issue when doing photometry with stacked images. If any single image contributing to the stack has a particular star saturate, then you can’t trust that star in the final stacked image. But because the pixel value from that one saturated image gets averaged with all the other non-saturated pixel values, it’s really, really hard to tell which stars in a stacked image can’t be trusted, unless you also have the raw images (or starlists) that contributed to the stack.)
For the lower limit, we used signal-to-noise ratio. We set the lower threshold to be an SNR of 10 (which gives a single-image magnitude uncertainty of around 0.1 mag, something of a worst-case limit for uncertainty). We created a plot with an X axis of SNR and a vertical axis of magnitude (which we pulled from the APASS catalog for each star in the image). We just looked at where that plot crossed the SNR=10 threshold, and used the corresponding magnitude as the faint limit.
Hi, Guy:
We have started testing with the Dwarf 3, but it’s still a work in progress. (We’ve been somewhat confused by what we see in the “raw” images from the Dwarf 3. Several STWG people are looking at these images, but there are two (related) symptoms that have our attention: we see high false-alarm rates during star detection (many detected “stars” aren’t actually stars), and instead of finding “hot pixels” in the raw images (something we’d expect), we seem to be finding “hot clumps” in the raw images.)
I feel pretty confident that we’ll figure out what’s going on fairly soon.
But the other side of your question is about availability of our analysis tools. And I cringe a little bit at the question, because it forces me to acknowledge that my earlier opinion on this was quite misguided.
As background, we have three Python tool “groups”:
Some tools that help us turn images into starlists,
A set of tools to support experiments that measure (and visualize) photometric accuracy
A production tool pipeline that will eventually accept starlists as input, identify variable stars in the starlists, and extract and archive variable star magnitudes.
Tool group 1 (image-to-starlist conversion) is fairly stable (plus or minus things like the Dwarf 3). Tool group 3 is in top-level design and architecture right now. It is many months away from being usable.
But I take your question as being about group 2 tools (accuracy analysis). I originally thought that these tools would be used by at most 5 to 10 people, and that the tools were actually “tools” and not finished programs. I predicted that once design of the production pipeline was stable, these analysis tools would no longer be needed. Well, I’ve had that original prediction pretty much squashed for several reasons:
Every time the smart telescope vendors add new features, we’ll need to revisit our analyses of their images and the resulting starlists. This will never end.
The development of new algorithms for teasing apart error sources will provide a constant pressure to repeat prior analyses.
There’s a significant number of AAVSO observers today who are insanely curious about their errors. We don’t have a good suite of tools to help them measure errors or to understand error sources. This creates a demand – a pull – for tools similar to our Calibrator/Analyzer; we can’t ignore that demand. And these tools are different from tools that measure variable star brightness. VPhot was never intended to help answer the question, “How much of your measurement uncertainty is due to systemic error and how much is random measurement error that can be made smaller by averaging more measurements?”
Some of our analysis agorithms are unique (for example, the Monte-Carlo error modeler that splits measured error into uncorrelated measurement error and correlated systemic error). It would be a shame to just let these algorithms wither.
The current analysis tools are in no way, shape, or form ready for distribution to anyone beyond the STWG. (Indeed, as I write this, these tools don’t even work for us; a needed software refactoring to address some serious speed issues has broken several features – we need to get the analysis tools back to a stable software build.) The best path forward is probably to eventually incorporate the analysis tool suite into the production starlist processing pipeline. This would permit observers to upload their variable star starlists for calibration and archiving of the variable star measurements, and also permit uploading of starlists from images of standard fields where updated versions of our Calibrator/Analyzer tool would help guide observers through the process of understanding the amount of error that they’re seeing from each error source.
Mark and the rest of the STWG, thank you for working on these issues! I have a SeeStar that I have only used for astrophotography purposes so far, but would be very interested to try it out for exoplanet transit photometry series. Do you have any tips I could apply to reduce error in the measurements, at this point? Or should I just stay tuned to this work in process?
Michelle
Michelle, we didn’t specifically set out to measure performance relative to exoplanet transits. We have been working to measure accuracy, while exoplanet transit detection needs repeatability.
However, I can make a prediction. If one sets a transit depth target of 0.01 magnitude (a Jupiter-sized exoplanet), then you need measurement scatter of no more than about 0.02 mag. (And maybe it doesn’t even need to be that good.) From the data we’ve collected, it may be possible to get scatter down that low by using three techniques:
Use a large comparison star ensemble (50+ stars) (this reduces scatter in the calculation of the image zero point),
If you’re using 10-second exposures, average together sets of 5 exposures (don’t stack them – instead make 5 measurements and average the 5 measurements together) (this reduces scatter due to measurement uncertainty, which is mostly associated with the one-shot-color sensor with its Bayer color filters),
Choose a bright exoplanet host star: magnitude 11 or brighter. (This reduces Poisson “shot noise” in the measurements.)
Yeah, I know that AstroImageJ doesn’t support item number 1 in my list. (I think it can handle at most 8 stars in the comparison star ensemble.) Maybe there’s someone reading this who can suggest an AstroImageJ alternative that can handle more than 8 reference stars.
If you do all three of those things, there’s a good chance you could detect a transit with a depth of 0.01 magnitude. (But bear in mind that this is a prediction based on some modeling using data that was collected for a different end purpose.)
I suppose, the same averaging is using for the fitting, so is there a sense to “loose” real points? I mean, least squares fitting should be more correct than simple averaging.
I know there will be a bolt of lightning coming out of the skies headed straight at me for saying this, but no, color transformations won’t help with (most) exoplanet transit observing.
While color transformations will make your data potentially more accurate, that isn’t what is most needed for transits: instead, it needs to be consistent. Quite frankly, adding more steps to your pipeline to incorporate color corrections means adding more opportunities for mistakes, which will hurt consistency.
Color transformations can be useful for transit observations when you are trying to confirm or refute a candidate exoplanet. One observable attribute that separates an exoplanet transit from an eclipsing binary star is the absence of a measurable color change during an exoplanet transit. That might argue for a benefit to doing color transformations, but even that is debatable.
So, there is a gain and there is a loss when you average points together. What you gain is a reduction is measurement noise, and with a one-shot-color sensor, there is a lot of measurement noise simply caused by the existence of the Bayer filter array. What you lose is time resolution.
But, interestingly enough, the loss of time resolution is something that the STWG actively works hard to achieve with most smart telescope observations. Observations become closer to “optimal” when the observing cadence matches the inherent time constant of the star’s behavior that you’re investigating. The default Seestar exposure time of 10 seconds is way too short for many kinds of variable star observing. And so we’ve incorporated a cadence adjustment feature into our still-under-development processing pipeline for starlists. For those target stars with behavior slower than 10 seconds, the pipeline will average measurements together in order to better match the observing cadence to the behavior time.
In this particular case, I assume that the reference to “least squares fitting” is suggesting that we extract a rate of change to the measurement in addition to the average value. (Remember that averaging is a zero-order least-squares approximation.) And you can certainly argue that what AstroImageJ actually does when fitting measurements to a transit light curve shape is a form of least squares fitting. But, in my experience, the AstroImageJ behavior will be more stable with averaged measurements (when measurement noise is large), although this is a good question to take up with Dennis Conti.
So, yes, you will lose time resolution due to averaging, and that can affect the accuracy of any transit timing information that you extract. But, honestly, if you need transit timing information, a smart telescope is the wrong instrument for the task. And averaging five 10-second measurements to create an averaged measurement that represents one minute of behavior is not a bad match to a transit ingress or egress event that might last as long as 10 or 15 minutes.
Hi, I am using VPHOT for the first time, to analyze SeeStar images (these are baseline images of WASP-52, not looking at the exoplanet transit but at baseline variability of the star). I see from some other posts here that people are stating the filter as TG for SeeStar images; but VPHOT does not give that option in its “upload” function. It does give a choice of SG. That means “solar green”, right? Which filter choice is best for SeeStar images in VPhot?
)