Please reply to this topic for your thoughts and discussion on the use of AI for conducting exoplanet analyses. If your comment is extensive, please consider providing a link to a document that contains the full comment.
Thanks,
Dennis Conti
Lead, AAVSO Exoplanet SIG
I believe that Claude can be used to generate code to either integrate with AIJ or ML models (not agents) to do pipeline analysis. The reason this approach is safe is that the code can be audited, all mathematics involved can be confirmed, and auditing tools can be applied to each step of the pipeline to produce reporting that is statistically viable.
Using discrete ML (huggingface.co), you can select models specifically trained on a target data set that has statistical evidence backing its accuracy. You can also use ML source code to construct your own models using data sets you already have to train a ML block for each step of the pipeline.
This is not the case with a Large Language Model (LLM) like Claude or GPT.
I do believe that the Large Language Models (LLM) can be used to explore ideas. I have often been surprised by the additional details that pop up from time to time in an analysis while testing with Claude and GPT. When I am not clear about how they originated, I ask probing questions about those details which can lead to sources and papers that the LLM ingested to process and craft its responses. I do not rule out the use of the LLM for exploration and illumination.
I firmly stand against the idea that one can create a series of agent prompts and expect that they will exert full control of LLM behavior. This approach requires endless edits to agent scripts in an attempt to place guardrails around processes. I consider these Large Language Models to the be the greatest whack a mole machine inventions of our time. For every restriction you place in a agent script, the LLM will find a way around it.
I also have extensively tested these agent modes of processing with Claude and the only consistent output Iâve have seen is in code generation using the Claude Code model.
At some point this may all change. As a scientist I never say never, but at this point in its evolution I donât trust LLM engines to perform complex multi-step processes driven by plain language agent scripts.
I do think discrete ML models offer a path to the integration of AI in pipeline processing of exoplanet, variable star, and binary star data sets.
2 Centavos deposited
jerry
I think we need to move beyond the assertion that an âoff-the-shelfâ commercial LLM such as ChatGPT cannot be used seriously for astronomical data reduction and analysis.
That assertion is no longer supported by the evidence.
General-purpose GPT models have already been successfully applied to real astronomical data in published research without being specifically trained as astronomy-only models.
In The Astrophysical Journal, Evgeny Smirnov demonstrated that GPT-4 Vision could classify astronomical resonant time-series data using natural-language instructions and without astronomy-specific fine-tuning. The experiment compared the GPT classifications with conventional classifications and reported extremely high performance. DOI: 10.3847/1538-4357/ad3ae1.
That work was subsequently expanded. In a 2026 Scientific Reports benchmark, commercial GPT, Claude, and Gemini models were tested on astronomical dynamical data. On the straightforward RB-PILOT dataset, GPT achieved an F1 score of 100%. On deliberately more difficult and ambiguous cases, performance declinedâexactly what one would expect from a serious scientific benchmark.
That is important because it demonstrates both capability and measurable limitations.
GPT-4o has also been tested directly on astronomical survey images for galaxy morphology, low-surface-brightness galaxy identification, and artifact recognition without specialized astronomical fine-tuning.
Even more relevant is the Mephisto project, where GPT-family reasoning is connected to the established astronomical SED-fitting program CIGALE. The system works with actual multiband photometric measurements, including JWST observations. It proposes models, executes CIGALE, examines quantitative results, modifies assumptions, and repeats the fit.
That is not AI âmaking pretty pictures.â
That is an LLM participating in a quantitative astronomical analysis workflow while established scientific software performs the underlying calculations.
Similar experiments are appearing in radio astronomy, where commercial GPT models have been tested on radio images, source morphology, diffuse emission, artifacts, and source classifications.
Therefore, saying categorically that an off-the-shelf LLM âcannot process astronomical dataâ is no longer scientifically defensible.
GPT-4o dates from 2024.
We are now dealing with systems such as GPT-5.6 and GPT-6 Astra.
OpenAI describes GPT-5.6 Sol as substantially improved in coding, scientific reasoning, knowledge work, and complex multistep tasks. GPT-6 Astra goes further by combining reasoning with computer use and the ability to work directly in specialized software, inspect data, execute analysis, generate plots, and explore scientific results.
GPT-6 Astra also supports approximately a 1.05-million-token context window, making it possible to work simultaneously with extensive documentation, code, calibration information, measurement tables, observing logs, and other scientific material.
Therefore, conclusions drawn from experiments with early ChatGPT or GPT-4 should not automatically be applied to present-generation systems.
This criticism deserves particular attention because the same standard must also be applied to conventional astronomical reduction.
Two astronomers using AstroImageJ on exactly the same FITS images do not necessarily obtain numerically identical light curves unless they use exactly the same reduction decisions and parameters.
Why?
Because AstroImageJ itself allows the observer to choose, among other things:
The AstroImageJ documentation explicitly allows the aperture radius and inner and outer background annuli to be changed by the observer. It also supports variable apertures based upon measured FWHM.
AIJ also permits the comparison-star ensemble and detrending configuration to be changed, and modern versions contain automatic optimization specifically for comparison-star selection, RMS minimization, and detrending-variable selection using BIC.
Even the current AIJ release includes functionality for comparing multiple reductions using different aperture sizes and selecting the best result.
Think about what that means.
If there were one uniquely correct aperture radius that every observer would inevitably choose, there would be no reason to test multiple aperture sizes.
If there were one uniquely correct comparison-star ensemble, there would be no reason to optimize comparison stars.
If detrending choices could not change the result, there would be no reason to optimize detrending parameters.
So demanding that two independent ChatGPT reductions produce precisely identical numbers while accepting analyst-dependent choices in AIJ is applying a different standard to AI than we apply to conventional reduction.
Reproducibility does not mean that every investigator must independently choose the same aperture radius.
Reproducibility means documenting what was done well enough that another investigator can repeat that same procedure and obtain compatible results.
For example:
Aperture radius = 8 pixels
Sky annulus = 15â25 pixels
Comparison stars = C1, C2, C3, C4
Centroiding = enabled
Airmass detrending = enabled
Outlier criterion = defined
Transit model = specified
If those parameters are preserved, the reduction can be reproduced.
That is true whether the parameters were selected by Pablo Lewin, another AAVSO observer, AstroImageJâs optimizer, a Python algorithm, or ChatGPT.
The scientifically meaningful comparison is:
Do both reductions recover the same transit?
Is the transit depth consistent within uncertainties?
Is the mid-transit time consistent within uncertainties?
Are the residual RMS values comparable?
Are the fitted parameters statistically compatible?
Do the same original photons ultimately support the same astrophysical conclusion?
If one AI reduction gives a transit depth of 1.42% and another gives 1.44%, that is not automatically evidence that AI has failed.
We first need to ask how much two independent conventional AIJ reductions differ when the observers independently choose apertures, annuli, comparison stars, detrending parameters, and rejection criteria.
That is the proper control experiment.
Commercial LLMs can hallucinate. Nobody should dispute that.
But the claim that they hallucinate more simply because they were trained on âtrillions of bytesâ of general information is not established science.
A small astronomy-specific model can outperform a general model on a particular narrow problem. That is one of the advantages of domain specialization.
But specialization does not automatically eliminate hallucination, and specialized fine-tuning can introduce its own problems, including degradation of capabilities outside the fine-tuned domain.
More importantly, astronomical reduction does not require trusting an LLMâs internal memory for the measurements.
The original FITS files are the evidence.
Bias values come from the bias frames.
Dark current comes from the dark frames.
Flat-field corrections come from the flats.
Stellar flux comes from measured pixels.
Times come from FITS headers.
Positions can be verified against WCS.
The LLM should be instructed to calculate from supplied data rather than invent missing information.
That distinction is fundamental.
Give present-generation ChatGPT the original FITS observations.
Require it first to inventory every file.
Require it to identify matching bias, dark, and flat frames.
Require matching exposure times, filters, binning, and timestamps.
Require deterministic software to perform pixel arithmetic and aperture photometry.
Require every intermediate measurement to be preserved.
Require the target and comparison stars to be identified explicitly.
Require the aperture and sky-annulus values to be recorded.
Require rejected frames to be listed with reasons.
Require the resulting measurements to be exported.
Then reduce the same observations independently using AstroImageJ.
Compare:
relative flux,
RMS,
comparison-star behavior,
transit depth,
mid-transit time,
duration,
residuals,
and uncertainties.
And repeat the experiment.
That is science.
We should not decide beforehand that it cannot work.
Nor should we decide beforehand that it must work.
We perform the experiment and measure the result.
If ChatGPT fails, document where and why it failed.
If it produces photometry statistically consistent with AstroImageJ, document that too.
If changing ChatGPTâs aperture from 7 to 9 pixels changes the result, compare that sensitivity with changing AstroImageJâs aperture from 7 to 9 pixels.
That last experiment may be particularly revealing.
The correct comparison is not:
human software = perfectly deterministic truth
AI = variable and therefore unreliable.
The correct comparison is:
same observational data + fully documented reduction A versus same observational data + fully documented reduction B.
Astronomical photometry has always contained analyst choices.
AI does not suddenly make those choices scientifically illegitimate.
It simply gives us a new way to make, document, automate, test, and potentially optimize them.
The American Astronomical Societyâs current AI guidance does not prohibit AI in data analysis or analysis-code development. Its emphasis is appropriately on scientific responsibility and verification.
That should be our standard as well.
The question in 2026 should therefore not be:
âCan we trust ChatGPT?â
I donât blindly âtrustâ AstroImageJ either.
I trust measurements that can be inspected, repeated, compared, and validated.
The question should be:
âUsing exactly the same astronomical observations, how does a documented ChatGPT-assisted reduction compare quantitatively with a documented conventional reduction?â
Letâs run that experiment.
â Pablo Lewin
The Maury Lewin Astronomical Observatory
AAVSO LPAC / MPC U76
I have spent many hours working through these two TFOP SG1 observing sequences, and I wanted to post a more complete follow-up to the discussion after my presentation.
Jerry did not recommend that I use ExoMiner++. His point was more fundamental: ChatGPT is not an exoplanet-analysis package, while ExoMiner++ is specifically designed for exoplanet candidate classification and is therefore much more naturally suited to this type of work. I took that criticism seriously and decided to test the underlying scientific question rather than argue about it.
The experiment became: if I impose strict scientific controls on ChatGPT, can it reproduce the portions of the ExoMiner++ methodology that are applicable to my ground-based data, and how do those results compare with the independent audit-first agents I had already built?
I want to be very precise about what I did and did not do. I did NOT run the full trained ExoMiner++ neural-network classifier. I do not have all of the TESS/SPOC inputs that ExoMiner++ expects, and I did not invent them. What I did was apply the parts of the ExoMiner++ methodology that can legitimately be transferred to a single-night TFOP ground-based observation: protect the predicted transit from detrending, estimate the low-frequency trend from out-of-transit information, use Savitzky-Golay detrending with the polynomial order selected quantitatively by BIC, create an ExoMiner+±style 31-bin transit view, and use centroid and difference-image diagnostics.
TESS-specific inputs that were not present in my data were simply not created. That includes momentum-dump information, SPOC DV/TPS scalars, multiple-transit odd/even tests, weak-secondary phase coverage, unfolded multi-transit inputs, and a trained ExoMiner++ probability score.
My original ChatGPT workflow is very different in structure. Instead of asking one model to âfind a transit,â I separated the work into specialized agents with explicit responsibilities.
The Data Triage Agent inventories the observing folder and calibration files. The Pixel Photometry Agent works from the calibrated FITS pixels and produces frame-by-frame target and comparison-star fluxes. The Skeptic / Anti-Hallucination Agent tries to disprove the event with registration checks, multiple apertures, comparison-star leave-one-out tests, control-star curves, airmass, centroids, saturation, gaps and other systematics. The Independent Validator applies a separate analysis route rather than tuning the first route to agree with it. The Report / Provenance Agent separates measured quantities, derived quantities, fitted quantities and reference values.
The rule throughout was simple: do not try to prove a transit. Measure the pixels, try to falsify the result, and report what survives.
Following Jerryâs criticism, I then added the ExoMiner++ methodology as the independent second route.
The important result is that the two methods were NOT forced to agree.
For TIC 274641986.01 / TOI 3545.01, they agree extremely well. The private TFOP prediction was a depth of 15.6 ppt. The audit-first agent workflow measured 15.42 ppt, while the ExoMiner+±methodology reduction measured 15.45 ppt. The fitted midpoints were +2.57 minutes and -3.41 minutes from the predicted midpoint. Two substantially different detrending approaches therefore recovered essentially the same depth and timing from the same measured pixels.
For TIC 190250689.01 / TOI 3802.01, the timing was again consistent, but the depths were not. The audit-first baseline produced a depth of 26.91 ppt and a midpoint +5.73 minutes from prediction. The ExoMiner+±methodology reduction produced 8.17 ppt and a midpoint +3.41 minutes from prediction, compared with the private predicted depth of 8.7 ppt.
I consider that disagreement scientifically important. I did not average the two depths together, tune one pipeline until it agreed with the other, or discard the inconvenient result. Control-star tests showed substantial time-correlated field behavior on that second sequence, so the conclusion is that the timing is considerably more robust than the absolute depth for that particular night.
This is also why I think the experiment says something useful about the role of ChatGPT in scientific work.
ChatGPT by itself is not ExoMiner++ and should not be represented as if it were. ExoMiner++ is a purpose-built exoplanet-classification system with a trained architecture and mission-specific inputs. What ChatGPT can do, however, when it is constrained by explicit scientific rules, modular agents, provenance requirements, independent checks and pixel-level data, is approximate the applicable procedural parts of the ExoMiner++ methodology for a ground-based dataset.
In other words, the useful capability is not that ChatGPT somehow âknowsâ whether a transit exists. The useful capability is that it can be instructed to execute a reproducible scientific procedure: read the real measurements, apply specified mathematics, retain every intermediate result, test alternative explanations, and refuse to manufacture inputs that do not exist.
I also want to be careful with the word âhallucination.â No large language model should be treated as mathematically guaranteed never to hallucinate. My goal was instead to design the workflow so that an unsupported statement could not become a scientific result without being exposed by the audit trail. The controls were:
âą every displayed photometric point had to come from measured FITS-pixel flux;
âą no synthetic observational points were allowed;
âą predicted TFOP depths and times were kept separate from measured and fitted quantities;
âą predicted transit windows could protect the event from baseline fitting but could not create flux values;
âą baseline choices were selected quantitatively, not because they produced a preferred answer;
âą interpolated values in the ExoMiner++ adaptation were used only internally for trend estimation and were never treated as observations;
âą comparison stars, control stars, apertures, centroids, registration and difference images could all falsify or weaken a result;
âą unavailable ExoMiner++/TESS inputs were marked unavailable rather than invented;
âą and the two pipelines were explicitly allowed to disagree.
That last point is important. If the purpose had been to satisfy me or to make ChatGPT look successful, TIC 190250689.01 would not have been reported with one method giving 26.91 ppt and the other 8.17 ppt. The disagreement remained in the report because the pixels and the analysis required it.
So my conclusion after these hours of work is narrower than âChatGPT can replace ExoMiner++.â It cannot. The conclusion is that ChatGPT, when tightly controlled and used as a reproducible scientific-analysis framework rather than as an oracle, can implement and approximate portions of the ExoMiner++ methodology that are applicable to ground-based follow-up data, while simultaneously applying independent ground-based checks that ExoMiner++ itself was not designed to perform at the raw-image level.
The attached figures show the fitted curves for both targets and both analysis routes:
In all four figures, every plotted photometric point is based on the measured flux from the calibrated FITS pixels. The smooth transit curves are fits to those measured points. Predicted ingress, predicted midpoint, fitted midpoint and predicted egress are marked separately. The lower AIJ-style traces are real measured or derived diagnostics such as raw relative flux, the fitted baseline/trend, airmass, X(FITS) and Y(FITS), rescaled only for display.
I would be very interested in criticism and suggestions for additional independent tests, especially from observers who routinely use AIJ, HOPS, ExoClock/TFOP workflows, or other transit-reduction software. The whole point of this experiment is reproducibility and falsifiability, not demonstrating that one particular tool is âright.â
LAISS-inspired analysis of ground-based exoplanet follow-up data
I have been experimenting with a third, independent way of interrogating the same ground-based photometry that I previously processed with my audit-first Agents and with selected ExoMiner++ procedures. This time I borrowed methodology from LAISS - Lightcurve Anomaly Identification and Similarity Search.
For readers who have not encountered LAISS, it was developed for time-domain transient astronomy rather than exoplanet fitting. Aleo et al. published the system in The Astrophysical Journal in 2024. LAISS converts survey light curves into numerical statistical features, combines them with contextual information, identifies unusual transients, and performs fast searches for objects with similar feature-space behavior. The published system operates on ZTF alerts through the ANTARES broker.
I did not run the trained LAISS transient classifier on my exoplanet data and call the result an exoplanet probability. That would be an out-of-domain use of the model. Instead, I transferred only the parts of the methodology that make scientific sense for a single-night ground-based follow-up observation:
For public discussion I am calling the two non-public TFOP SG1 objects Target A and Target B.
#The most important rule: no fake photometry
Every flux point used here came from aperture photometry of an accepted, calibrated FITS frame. No light-curve point was created by ChatGPT, LAISS, a transit model, interpolation, or any other AI system. The predicted transit window was used only to define the out-of-transit reference and to mark the expected event. It was not allowed to alter a measured flux value.
I also deliberately did not report a LAISS P(anom) or Random Forest classification, because the published LAISS classifier was trained on ZTF astrophysical transients and host-galaxy context, not on exoplanet transits. The anomaly numbers in this experiment are simply percentiles relative to the measured out-of-transit feature distribution from these observations.
#What happened?
For Target A, the deepest segment inside the predicted event reached the 98.3th percentile in photometric feature space while its instrumental-context anomaly was only at the 60.3th percentile. Seven of its eight nearest feature-space neighbors were also event-window segments.
For Target B, the deepest event-window segment reached the 98.3th photometric percentile, while the instrumental-context anomaly was only 48.3th percentile. Six of the eight nearest neighbors were event-window segments.
That second case is particularly interesting: the brightness behavior becomes unusual at the expected time without registration, centroid, aperture-stability, and related observing diagnostics becoming comparably unusual.
This does not erase a result I reported previously: Target Bâs absolute transit depth remains sensitive to the detrending method. The Agent and ExoMiner+±methodology reductions do not give the same depth. I have kept that disagreement because it is part of the evidence, not something to be tuned away.
#Why add LAISS at all?
The three approaches ask different questions:
Audit-first Agents: Can the event survive calibration, registration, aperture, comparison-star, control-star, centroid, airmass, and provenance checks?
ExoMiner++ methodology: Does a transit-oriented, mask-first detrending strategy recover consistent event morphology and timing?
LAISS methodology: Is the observed segment statistically unusual in a multivariate feature space, and do its nearest measured analogs show the same type of behavior without equally unusual instrumental context?
Agreement among different methods is useful because they fail in different ways. Disagreement is equally useful because it identifies the limitation.
#Anti-hallucination controls
The workflow was designed so that a language modelâs unsupported statement cannot become an astronomical measurement:
The full illustrated report is attached, together with the diagrams and the two public-safe anomaly plots.
My main conclusion is not that LAISS is an exoplanet pipeline - it is not. The result is that LAISS-style feature extraction, anomaly detection, and similarity search can provide a useful independent diagnostic layer for ground-based exoplanet follow-up, provided the analysis remains completely traceable to the original measured flux.
A longer-term idea would be to build an exoplanet-follow-up similarity bank containing known successful transits, partial transits, null observations, weather failures, guiding problems, comparison-star failures, and false positives. Then a new observing sequence could be compared with a library of real previous nights rather than judged only by one fitted curve.
Reference: Aleo et al. 2024, ApJ 974, 172, DOI 10.3847/1538-4357/ad6869. Public LAISS software: Zenodo DOI 10.5281/zenodo.11541806.
C-14 Transit Photometry: Two Anonymous Targets, Limb Darkening, and Model Dependence
I recently used ChatGPT/Codex to analyze two ground-based transit-follow-up datasets from the Maury Lewin Astronomical Observatory, using my Celestron C-14 EdgeHD and ASI2600MM Pro camera with an R-band filter and five-minute exposures.
Iâm calling the objects Target A and Target B because I am treating their target-specific information as proprietary within the TESS follow-up collaboration. This post omits TIC/TOI numbers, coordinates, observing dates, exact ephemerides, and identifying star-field patterns. The purpose is to discuss the analysis while respecting that confidentiality.
The reduction used dark subtraction, normalized flat-field correction, tracked apertures, local sky subtraction, and differential photometry against seven selected comparison stars. The flats and bias frames had different gain settings, making the flat correction approximateâa limitation that remains relevant to interpreting small brightness changes.
The initial analysis used a trapezoidal model without limb darkening. I requested a revision using quadratic limb darkening, with separate coefficients derived for each star from published stellar-atmosphere tables. Cousins R was adopted as a passband approximation because the exact instrumental transmission curve was unavailable. Missing stellar inputs were explicitly identified as assumptions.
The revised model averages the predicted brightness over each five-minute exposure. The calibrated photometry was retained; the fitting and supporting checks were regenerated.
Target A: light curve
The reference linear-baseline fit gives a central model depth of approximately 2.35% ± 0.63 percentage points. A quadratic baseline reduces this to approximately 1.66% ± 0.87 percentage points.
Target B: light curve
The reference depth is approximately 0.64% ± 0.34 percentage points. A quadratic baseline gives approximately 0.04% ± 0.51 percentage points, consistent with zero.
In both light curves, grey points represent individual exposures with counting/sky error estimates. Blue points are successive three-exposure means, with residual-based standard errors. The orange curve is the limb-darkened, exposure-integrated reference model. The shaded interval is the supplied event window; time is shown relative to its center.
Residuals and baseline sensitivity
These plots show why the baseline choice matters. A flexible baseline can absorb a real event or an instrumental trend; these observations do not decisively separate those explanations.
Additional materials include aperture-size tests, adopted limb-darkening intensity profiles, and anonymized scientific and STEM reports. The scientific reports contain redacted information strips and actual, separately arranged target/comparison-star cutouts. The wider field geometry is withheld to protect target identities.
The reported uncertainties are conditional statistical estimates. They do not include every uncertainty in calibration, contamination, stellar properties, timing, or geometry. Comparison-star checks and moving-block residual resampling provided additional diagnostics.
An independent stellar-density check also exposed tension between the reference circular geometry and the short supplied event durations, particularly for Target B. A fitted transit-shaped depression is therefore not a complete, physically consistent orbital solution.
This experiment illustrates both the usefulness of AI-assisted analysis and the need to examine its assumptions. Adding limb darkening improved the modelâs physical description, but it did not remove calibration limitations or baseline dependence. Target A shows a possible event-shaped depression worth investigating; Target B remains inconclusive.
I would welcome suggestions on handling baseline curvature, choosing limb-darkening assumptions without an exact filter response, and assessing nearby-star contamination at this precision.
Pablo Lewin â AAVSO LPAC
The Maury Lewin Astronomical Observatory
Both reference-fit depths are consistent with the TESS catalog values within their statistical uncertainties. Keeping the targets anonymous, the comparison uses the TESS-project entries on ExoFOP retrieved during the analysis:
| Target | Catalog depth: TESS | Your fitted depth: R band | Comparison |
|---|---|---|---|
| A | 1.918% ± 0.076 | 2.352% ± 0.625 | About 23% deeper, but only 0.7 combined standard errors different |
| B | 0.819% ± 0.072 | 0.639% ± 0.344 | About 22% shallower, but only 0.5 combined standard errors different |
The ± uncertainties are in percentage points. Your values use the limb-darkened model with a linear baseline.
For Target A , your measured depression is broadly compatible with the expected depth. For Target B , the fitted depth is also compatible, but the uncertainty is too large to establish a secure detection.
Two qualifications matter:
Your results therefore show agreement with the expected depths under the reference model , without independently confirming either eventâs planetary interpretation.
What actually happens behind the scenes when ChatGPT helps process exoplanet observations? A detailed account of my October 6 experiment
I recently used ChatGPT/Codex to help calibrate, reduce, and analyze two exoplanet observing sequences acquired on October 6, 2026, at The Maury Lewin Astronomical Observatory: TOI-1516 b and Kepler-695 b , observed with my Celestron C14 EdgeHD and ZWO ASI2600MM Pro.
I wanted to explain what happened behind the scenes, including the actual software used. Saying âAI processed my observationsâ leaves out most of the information needed to evaluate or reproduce the work.
In this experiment, ChatGPT/Codex served as the interface and programming assistant. It retrieved files, wrote and executed Python scripts, selected and configured scientific libraries, inspected results, and produced reports. The calibration, photometry, time conversions, and transit calculations were performed by executable software operating on the FITS images.
The analysis used a custom Python workflow. It did not run AstroImageJ, EXOTIC, PixInsight, ASTAP, or Astrometry.net. Those distinctions matter because the reader should know precisely which methods produced the measurements.
1. Getting the observations into the analysis environment
I supplied the Google Drive folder containing the observations:
G:\My Drive\The Maury Lewin Observatory\C-14\October 6th 2026
The Google Drive connector returned file information in JSON format . This was an inventory containing filenames, identifiers, byte sizes, and information needed to retrieve the files.
The JSON was not the astronomical image data. The scientific input consisted of the actual FITS files , including their pixel arrays and headers.
An initial folder listing returned only 100 items. The retrieval was repeated with a larger listing limit, producing an inventory of 824 items:
The 821 FITS files were downloaded using Python and curl , with concurrent downloads and retries. Each downloaded fileâs byte size was checked against the Drive inventory.
That check established that the local downloads had the expected sizes. It was a transfer check, not a claim that every pixel or header entry had been independently validated.
2. Reading the FITS headers and examining the calibration material
The workflow used Astropyâs FITS interface , astropy.io.fits , to read image arrays and observing metadata.
The headers identified the C14 EdgeHD, ASI2600MM Pro, R filter, 2Ă2 binning, gain 50, offset 25, and a science-frame temperature of approximately â12°C.
The calibration frames were examined before use. This revealed an important problem: the original 20-second flat frames were saturated, with their pixels at 65,535 ADU. These frames could not provide a useful measurement of relative sensitivity across the detector.
I subsequently supplied five 5-second flat frames. Their median signal was approximately 27,300 ADU, and they were not saturated. They matched the science data in filter, binning, gain, offset, and recorded rotator angle.
There were still qualifications: the replacement flats were taken at approximately â10°C, and their recorded focus position differed from that of the science frames. Those differences were documented rather than treated as perfect calibration matches.
This is one example of why a reduction needs inspection of the actual data. A folder containing files named âFLATâ does not establish that those files are usable.
3. Identifying the target and comparison stars
The astrometric identification used three principal components:
DAOStarFinder , to detect stars in the images.The catalog positions were projected into approximate local sky coordinates. Astroalign then matched patterns of stars and calculated a transformation into image-pixel coordinates.
The resulting solution located each target and provided detector positions for potential comparison stars. The matching reported 29 stars for the TOI field and 48 for the Kepler field, with positional residuals of approximately 1.53 and 1.27 pixels, respectively.
There is an important technical qualification here: this was catalog-based star identification and a sky-to-detector coordinate transformation . It did not produce and write a complete conventional WCS solution into the FITS headers.
It therefore should not be described as having run an ASTAP or Astrometry.net plate solve. The method was sufficient to identify the stars used in the photometry, but its purpose and output were narrower than a full astrometric reduction.
4. Calibrating the science images
Calibration was performed using NumPy and custom Python code .
The process was:
The science-frame calculation was:
[ I_{\mathrm{calibrated}} = \frac{I_{\mathrm{raw}}-B-tD}{F} ]
Here, (B) is the master bias, (D) is the estimated dark signal per second, (t) is the exposure duration, and (F) is the normalized master flat.
The 300-second Kepler exposures had matching-duration dark frames. The 15-second TOI exposures and 5-second flats used the dark signal scaled from the 300-second darks.
That scaling is an assumption of this reduction. There were no matching 15-second darks or 5-second flat darks in the material used. The reports disclosed this limitation, along with the flat temperature and focus differences.
The raw FITS images were preserved.
5. Following the stars as the field moved
The workflow could not assume that each star remained at the same pixel position throughout the night.
The TOI sequence showed substantial movement across the detectorâroughly 148 pixels in one coordinate and 185 in the other. The Kepler sequence moved by approximately one pixel.
An initial extraction using fixed coordinates was inadequate for the moving TOI field. The workflow was corrected to register the individual frames before measuring the stars.
Registration used scikit-imageâs phase_cross_correlation on downsampled, logarithmically transformed images. This estimated each frameâs shift relative to a reference frame.
The estimated shift moved the target and comparison-star positions into the current frame. Local intensity-weighted centroids then refined the positions.
This matters because an aperture that does not follow a moving star can produce a light curve dominated by extraction errors. Tracking the field was an essential part of the reduction.
6. Measuring stellar flux and building differential light curves
Photometry used custom Python/NumPy routines with circular apertures.
The workflow tested three aperture radii:
For each star, the local background was estimated in an annulus between 20 and 30 pixels from its center. A three-sigma rejection step removed outlying sky pixels before calculating the background level.
The code summed the aperture pixels and subtracted the estimated sky contribution. It also calculated approximate per-star flux uncertainties using the stellar signal, camera gain, and measured background scatter.
Up to 20 comparison-star candidates were considered. Five comparison stars were selected using their out-of-transit scatter. Each comparison light curve was normalized, and the median of the normalized comparisons formed the reference ensemble.
The target flux was divided by that ensemble to produce the differential light curve.
An additional check star was examined separately. The analysis also tested how the fitted result changed with aperture size and with individual comparison stars removed.
These checks are useful because the result should not depend excessively on one aperture choice or one comparison star.
A qualification is that choosing comparison stars from the same observing sequence is itself an analysis decision. It does not prove that all remaining systematic effects have been removed.
7. Converting the observation times
The time conversion used Astropyâs Time , EarthLocation , and SkyCoord classes .
The FITS MJD-AVG values supplied the mid-exposure UTC timestamps. Those were converted to BJD_TDB , including the barycentric light-travel-time correction for the targetâs direction.
This step is essential when comparing observed transit timing with predictions stated in BJD_TDB. UTC timestamps and barycentric dynamical timestamps cannot simply be substituted for one another.
The calculation used the recorded observatory latitude and longitude. An elevation of 0 meters was used because the actual elevation had not been verified within this analysis; that assumption was documented.
8. Modeling the transit and including limb darkening
Transit models were calculated with batman-package .
The fits used:
The quadratic limb-darkening law was:
[ \frac{I(\mu)}{I(1)} = 1-u_1(1-\mu)-u_2(1-\mu)^2 ]
The nominal coefficients were:
| Target | (u_1) | (u_2) | Source used |
|---|---|---|---|
| TOI-1516 b | 0.21 | 0.34 | R-band ground-based fit reported by Sonbas and collaborators |
| Kepler-695 b | 0.5394 | 0.1709 | Interpolation of Claret & Bloemen R-band ATLAS tables |
The Kepler interpolation used the adopted stellar effective temperature, surface gravity, metallicity, and a microturbulence value of 2 km/s. The table retrieval used Astroquery/VizieR, and the interpolation used SciPyâs LinearNDInterpolator .
The nominal transit fits were optimized with SciPyâs optimize.least_squares . Fitted quantities included the transit midpoint, planet-to-star radius ratio, scaled orbital separation, impact parameter, and baseline parameters.
Including limb darkening matters because the brightness of the stellar disk changes from center to edge. A physically modeled transit therefore has a different shape from a simple rectangular dip, especially during ingress and egress.
9. Estimating uncertainty and testing the result
The analysis used 250 moving-block residual bootstrap trials .
Residuals were resampled in blocks of approximately five minutes, with at least two points per block. The purpose was to retain some of the short-timescale correlation in the residuals rather than treating every point as independent.
The bootstrap also varied the limb-darkening coefficients and refitted the transit. The adopted coefficient perturbations were:
The actual timestamps and gaps remained part of the fitting procedure.
Additional checks included:
This was a bootstrap-based uncertainty analysis. It was not an MCMC run or a full Bayesian posterior calculation.
The resulting uncertainties describe the variation captured by this procedure. They do not guarantee coverage of every possible calibration error, baseline error, or model assumption.
10. Comparing observed and predicted transit properties
The fitted models were used to derive transit midpoint, first and fourth contact times, duration, and modeled depth.
First contact, (T_1), is the beginning of ingress. Fourth contact, (T_4), is the end of egress. The ingress and egress intervals also involve second and third contacts, (T_2) and (T_3); these should not be confused with a single timestamp.
The observing-planner image supplied predicted start, midpoint, end, and depth values. Those predictions were compared with the fitted values on a consistent time basis.
For TOI-1516 b, the final analysis gave approximately:
| Quantity | Observed result | Difference from supplied planner |
|---|---|---|
| Modeled depth | (18.57 \pm 0.71) ppt | Planner depth: 17.5 ppt |
| Midpoint | BJD_TDB 2461320.952777 | (+1.26 \pm 1.55) minutes |
| First contact | BJD_TDB 2461320.891265 | (-4.23 \pm 4.22) minutes |
| Fourth contact | BJD_TDB 2461321.014290 | (+6.75 \pm 3.73) minutes |
One ppt is a 0.1% change in normalized flux, so the fitted TOI depth was approximately 1.86%.
The flat correction affected the result: the TOI fitted depth was approximately 19.98 ppt without the replacement flat correction and 18.57 ppt with it. That difference provides a concrete example of calibration affecting the scientific measurement.
The Kepler-695 b sequence was much less conclusive. Its fitted depth was approximately (26.3 \pm 17.0) ppt, compared with a planner depth of 11.4 ppt. The transit model improved BIC by only about 3 relative to the linear baseline model.
The Kepler timing and depth results were therefore reported as exploratory. The large uncertainties and weak model preference do not support a secure claim of an anomalous transit depth or a transit-timing variation. The fitted grazing geometry also left second and third contact unresolved.
A fit returning numerical values is not, by itself, evidence that those values are scientifically well constrained.
11. Producing the plots, reports, and reproducibility package
Matplotlib generated the light curves, model overlays, diagnostic plots, field images with target and comparison-star labels, and calibration illustrations.
ReportLab generated three PDF reports:
The PDFs were rendered with PyMuPDF and Poppler for visual checks. Pillow was used for image assembly and contact sheets.
The analysis exported several complementary file types:
Pythonâs zipfile module assembled the analysis package. The purpose was to provide material that another person could inspect and use to reproduce the work, rather than asking them to accept a report without the underlying calculations.
The main processing sequence was:
inventory.json â solve_gaia.py â photometry.py â final_fit.py â enrich.py â reports.py
12. The software versions recorded for this run
| Software | Version | Principal role |
|---|---|---|
| NumPy | 2.3.5 | Array calculations, calibration, photometry, resampling |
| SciPy | 1.17.0 | Optimization, interpolation, supporting calculations |
| Astropy | 8.0.1 | FITS access, astronomical coordinates, time conversion |
| Photutils | 3.0.0 | Star detection |
| Astroalign | 2.6.2 | Stellar-pattern matching |
| batman-package | 2.5.3 | Limb-darkened transit models |
| ReportLab | 4.4.9 | PDF generation |
Other components included Astroquery, scikit-image, Matplotlib, Pillow, PyMuPDF, Poppler, curl , and Pythonâs standard-library tools.
NINA was the acquisition software recorded in the FITS headers. The numerical reduction ran on the analysis environmentâs CPU; it did not use my observatory computerâs RTX 3090 GPU.
What I think this experiment demonstrates
My interest is in whether a conversational AI can help an observer construct, execute, document, and examine a scientific reduction using established computational tools.
In this case, the work included discovering saturated flats, replacing them with usable calibration data, correcting the extraction to follow a drifting field, applying barycentric time conversion, fitting limb-darkened transit models, testing sensitivity to reduction choices, and distinguishing a comparatively well-constrained result from a weak one.
Those are concrete operations that can be inspected in code and numerical outputs.
The experiment also shows why detailed disclosure matters. âAI analyzed itâ says very little. A useful account identifies the input files, calibration assumptions, software, photometric method, timing convention, model, uncertainty procedure, and limitations.
For an independent assessment, I would want another reduction of the same framesâpotentially in AstroImageJ or another established pipelineâand a comparison of the extracted light curves, timing, depth, and uncertainties. Agreement would strengthen confidence; disagreement would provide a specific problem to investigate.
I welcome technical comments on the actual choices made here, particularly dark scaling, comparison-star selection, baseline modeling, limb-darkening assumptions, and the treatment of correlated noise. Those are the kinds of questions that can improve this experiment and help determine where this approach is useful.
â Pablo Lewin
The Maury Lewin Astronomical Observatory
i have been reading some of the comments, i do not have experience with exoplanet analysis, but i have some experience with ai, I think some of the problems of AI commented here could be solved using MCP on open-source tools. Basically, the AI would only be able to call a fixed set of validated tools, the same tools you already use, instead of creating its own. As long as it isnât also given the ability to run its own code, that limit is built into the setup rather than written in a prompt, so it canât find a way around it. That way every number comes from tested software, and each step can be logged and repeated. Has anyone tried something like this?
It doesnât solve the problem that the AI could use the tools in the wrong way, but I think it could reduce the chances of that happening. the idea that i have is to split the analysis into stages, something like closed rooms, in each stage the ai could only use certain tools that are allowed for that stage, those tools would be predefined in the server so the ai could only do what is allowed to, maybe it can be even added an option for it to request access to another tool with a justification that has to be approved by a human, its something like an algorithm but it could take decisions that are not only based in a fixed set of ranges, like take a decision based on looking at a light curve, every step and request would be logged, so anyone could review what the ai did.
I agree with your assessment up to a point. I started out testing both Claude and Open AI about a year ago with rudimentary attempts to process image sets. I found varied results and inconsistent output in earlier models.
Along the way I learned MCP and then switched to ICM to build code that connects to individual ML models offered on hugging face. Several of the students I work with in our GBO exoplanet section are working together to build out a pipeline to process data from the scope.
We think that the ICM approach offers tighter control, repeatability, and a clear understanding of the processing since these models are specifically trained to do one job. Where AIJ API offers access to functions not available in a ML model, we are working to incorporate these elements in the workflow.
What concerns me most about the LLM approach is that you just do not know what its doing inside the black box. Even if you provide specific prompting guardrails, it is documented that the AI does not honor the rule if it determines a more efficient method exist, and it does not report how it resolves prompts.
I recommend reviewing the Interpretable Context Methodology (ICM) located at GitHub - RinDig/Interpretable-Context-Methodology: Folder structure as agent architecture. ICM replaces framework-level orchestration with filesystem structure. · GitHub. Especially if your integrating discrete ML models in an orderly processing pipeline.
Thank you Jerry, what an interesting project! Oh yes, then I agree if thereâs no LLM making decisions while the data is processed, MCP isnât really needed. Thank you for sharing ICM, I hadnât heard of it before. Iâm actually building an automatic pipeline to process ZTF stars, classify them and choose a period for them, but I went with somewhat proven methods basically a random forest a set of rules and some algorithms, and it has been working reasonably well for some categories, also its cheap to run, it is not required to have a graphics card or an M series chip.
since you wrote that you write code with claude i also wanted to share this skill with you i have been using to write code with claude GitHub - DietrichGebert/ponytail: Makes your AI agent think like the laziest senior dev in the room. The best code is the code you never wrote. · GitHub it pushes the agent to reuse existing libraries and write less custom code, makes code easier to audit, maybe youll find it useful 
Jerry and Javier, thank you for sharing your approaches. Restricting available tools, separating processing into stages, and logging the operations are useful ideas. I agree that validated software still needs to be used correctly.
As an experienced observer who has worked with exoplanet photometry, I would add that different results are not unique to AI-assisted reductions. Two people processing the same images in AstroImageJ can obtain different depths, midpoints, and uncertainties because they choose different aperture radii, inner and outer sky-annulus radii, comparison stars, centroiding options, rejected observations, detrending variables, or fitting assumptions. AIJ explicitly provides many of these choices.
The computing environment also belongs in the reproducibility record: software versions, dependencies, settings, and numerical implementation. However, I would distinguish those differences from the computer hardware itself. With identical inputs, software, and settings, we should expect substantially consistent results across computers. A large discrepancy deserves investigation; it should not simply be attributed to using a different machine.
My point is that using established software does not eliminate the observerâs methodological decisions. A human can choose an unsuitable annulus or comparison star just as an AI can. We need to document and test those decisions in either case.
I would also be cautious about applying experience with ChatGPT or Claude from a year ago directly to todayâs systems. Those earlier observations are valid evidence about the versions and workflows tested at that time, but they do not establish the performance of the latest versions. Current Codex environments support executable workflows, tool integration, and configured execution controls. That warrants fresh testing, while still requiring scientific validation.
In my recent experiments, ChatGPT/Codex used Astropy, NumPy, SciPy, Photutils, Astroalign, and batman to perform the numerical work. It generated and executed scripts, and the analysis package retained the code, software versions, extracted photometry, calibration products, model parameters, and diagnostic figures. There was custom code involved, and that code deserves inspection too.
I agree that we cannot inspect every internal operation of an LLM. But we can inspect the scientific processing it actually executed: which files it read, which calculations the scripts performed, which coefficients and settings were used, and how the results changed when apertures or comparison stars changed. The modelâs internal workings and the numerical reduction are different questions.
For example, my latest reduction supported a TrES-5 transit but did not securely detect the much shallower TOI-2445 transit. The latterâs observed midpoint and contacts were reported as unresolved. That is an essential part of the experiment: a workflow must be able to report that the available data do not justify a measurement.
I see MCP, staged pipelines, and specialized models as potentially valuable approaches. My experiment explores another approach: using a current general-purpose assistant to build and execute an inspectable scientific workflow.
The most useful comparison would be to process the same FITS images through these different approaches, document the settings, and compare the light curves, timing, depth, residuals, and uncertainty estimates. That would give us concrete evidence about where each method succeeds and where it needs improvement.
Best,
Pablo
Jerry, I think your concern is valid, but I would distinguish between the LLMâs internal reasoning and the scientific calculations it directs. I cannot inspect every internal decision the model makes. However, I can inspect the code, inputs, settings, intermediate products, and numerical results used to produce the analysis.
In my October 6 and 7 work, ChatGPT wrote and ran a Python workflow using identifiable scientific libraries: Astropy for FITS handling and barycentric time conversion, Photutils for star detection, Astroalign for matching image stars to Gaia positions, and BATMAN for transit models incorporating limb darkening. SciPy performed the numerical fitting. The analysis package includes the scripts, photometry tables, model parameters, diagnostic figures, and software versions. That gives another observer a concrete processing record to examine and rerun.
There were also processing decisions that needed to be documented. The October 6 saturated flats were rejected. The October 7 analysis tested different aperture radii, comparison-star selections, and processing without flat correction. Limb-darkening coefficients and their sources were recorded. These choices can affect results whether a human operates AIJ or an LLM helps construct the workflow.
One particularly useful example is the October 7 TOI-2445 observation. An exploratory fit produced a transit-like solution, but it was unstable, and the statistical comparison did not support a detection. The final report therefore classified the observation as inconclusive and did not present that fitâs midpoint, ingress, egress, or depth as measured transit properties. The optimizer producing a number was insufficient grounds to accept it.
I also agree that a prompt saying âfollow these rulesâ is insufficient protection. In the workflow I used, the model could write and execute code; it was not confined to a fixed set of validated tools. Consequently, I would not claim that departures were technically impossible or that every decision was captured in a tamper-proof log. Restricting permissions and adding explicit checks would strengthen the process.
What I would challenge is the implication that an opaque internal reasoning process necessarily makes the resulting scientific analysis opaque. We do not need access to the modelâs private reasoning to audit whether it used the correct timestamps, calibration frames, apertures, limb-darkening coefficients, or fitting assumptions. We need those operations and assumptions made explicit, with reproducible outputs and independent checks. An explanation written by the AI is useful, but the saved code and processing products are the evidence against which that explanation must be checked.
My work is an experiment in making that possible, not proof that an LLM should be trusted without supervision. The appropriate standard is whether another observer can inspect the processing, identify mistakes, and reproduce the conclusions. That is the standard I want applied to my results.