Pvcode + Cvode improvements #2889

dschwoerer · 2024-03-19T15:03:07Z

Exposes more options to the user.
Allows to dump if the solver fails. This can be useful for figuring out why the solver is slow. Currently only for pvode, but should also be possible to implement this for cvode (for cvode we can use a documented API)

Use the new track feature (better name required) to dump the different components of the ddt() as well as the residuum for the evolved fields.

This keeps track of all the changes done to the field and stores them to a OptionsObject.

include/bout/field.hxx

ZedThree · 2024-03-19T15:41:12Z

My worry here is the tracking storing a lot of data if it's storing the full Field3D on every single change. Should it also be a compile-time option so that it completely disappears for production runs?

github-actions

clang-tidy made some suggestions

include/bout/field3d.hxx

src/field/field3d.cxx

src/solver/impls/pvode/pvode.cxx

dschwoerer · 2024-03-19T16:10:10Z

My worry here is the tracking storing a lot of data if it's storing the full Field3D on every single change. Should it also be a compile-time option so that it completely disappears for production runs?

It could be, but I don't think it is needed. I would personally keep it enabled for production runs, as it has a runtime switch, that is disabled by default. And I would also keep it like this, at least by default.

If a simulation crashes, it can be very useful to immediately get why it failed (this is currently the only use of the name tracking). Having to restart + queue again, just to get to the point where it crashes can be very time consuming, and the cost should be really negligable, if disabled.

I did notice a slow down in my simulations with this enabled, but that was when it was enabled all the time and the names did not get reset, thus the names have been growing exponentially.

dschwoerer · 2024-03-19T16:28:23Z

On 3/19/24 16:34, Peter Hill wrote: ***@***.**** commented on this pull request. ------------------------------------------------------------------------ In include/bout/field.hxx <#2889 (comment)>: > @@ -683,4 +683,12 @@ inline T floor(const T& var, BoutReal f, const std::string& rgn = "RGN_ALL") { #undef FIELD_FUNC +template <typename T, typename = bout::utils::EnableIfField<T>, class... Types> +inline T setName(T&& f, const std::string& name, Types... args) { I think most other setters like this are member functions

But then they return either Field, or need to be overwritten.

github-actions

clang-tidy made some suggestions

src/solver/impls/pvode/pvode.cxx

github-actions · 2024-03-20T09:09:08Z

src/solver/impls/pvode/pvode.cxx

  // Check return flag
  if (flag != SUCCESS) {
    output_error.write("ERROR CVODE step failed, flag = {:d}\n", flag);
+    CVodeMemRec* cv_mem = (CVodeMem)cvode_mem;


warning: do not use C-style cast to convert between unrelated types [cppcoreguidelines-pro-type-cstyle-cast]

CVodeMemRec* cv_mem = (CVodeMem)cvode_mem; ^

src/solver/impls/pvode/pvode.cxx

dschwoerer · 2024-03-20T10:49:00Z

The cuda build fails with:

/__w/BOUT-dev/BOUT-dev/src/field/field3d.cxx: In member function 'Options* Field3D::track(const T&, std::string)':
/__w/BOUT-dev/BOUT-dev/src/field/field3d.cxx:851:63: error: expected ';' before '}' token
  851 |     (*tracking)[outname].setAttributes({
      |                                                               ^
      |                                                               ;
/__w/BOUT-dev/BOUT-dev/src/field/field3d.cxx:851:64: error: expected primary-expression before ',' token
  851 |     (*tracking)[outname].setAttributes({
      |                                                                ^
/__w/BOUT-dev/BOUT-dev/src/field/field3d.cxx:851:66: error: expected primary-expression before '{' token
  851 |     (*tracking)[outname].setAttributes({
      |                                                                  ^
/__w/BOUT-dev/BOUT-dev/src/field/field3d.cxx:851:95: error: invalid use of void expression
  851 |     (*tracking)[outname].setAttributes({
      |                                                                                               ^
[ 16%] Building CUDA object CMakeFiles/bout++.dir/src/field/field_data.cxx.o

The relevant code is:

    (*tracking)[outname].setAttributes({
      {"operation", operation},
#if BOUT_USE_TRACK
      {"rhs.name", change.name},
#endif
    });

Is gcc9.4 not able to use setAttributes?

ZedThree · 2024-03-20T12:32:03Z

It could be, but I don't think it is needed. I would personally keep it enabled for production runs, as it has a runtime switch, that is disabled by default. And I would also keep it like this, at least by default.

We do try to completely remove debugging features for production runs, see TRACE for example. Being able to remove the function call entirely at lower CHECK levels would make me much happier about this.

If a simulation crashes, it can be very useful to immediately get why it failed (this is currently the only use of the name tracking). Having to restart + queue again, just to get to the point where it crashes can be very time consuming, and the cost should be really negligable, if disabled.

This is true, but I would really like to understand the costs. It looks like it's storing potentially hundreds or thousands of copies of full fields, so although it looks like a really interesting and useful feature, it would be good to have a good idea of the costs.

I did notice a slow down in my simulations with this enabled, but that was when it was enabled all the time and the names did not get reset, thus the names have been growing exponentially.

Is that with the current form?

But then they return either Field, or need to be overwritten.

It should almost certainly return Field& and then this wouldn't be an issue

dschwoerer · 2024-03-20T12:53:03Z

It could be, but I don't think it is needed. I would personally keep it enabled for production runs, as it has a runtime switch, that is disabled by default. And I would also keep it like this, at least by default.

We do try to completely remove debugging features for production runs, see TRACE for example. Being able to remove the function call entirely at lower CHECK levels would make me much happier about this.

If at all, I would be in favor of adding a new configure flag. I use this in runs with CHECK=0.

If a simulation crashes, it can be very useful to immediately get why it failed (this is currently the only use of the name tracking). Having to restart + queue again, just to get to the point where it crashes can be very time consuming, and the cost should be really negligable, if disabled.

This is true, but I would really like to understand the costs. It looks like it's storing potentially hundreds or thousands of copies of full fields, so although it looks like a really interesting and useful feature, it would be good to have a good idea of the costs.

It is only storing fields, if the solver has failed. Unless that happens, no additional storage is used.
There are some calls, but they are all no-ops, if you think that might be a problem, I can certainly guard them with #if statements, but I think that should be orthogonal to any of the flags we have.

In the case the solver crashes, I think the cost is really negligible. This is only done once, if the solver decided it cannot continue. For hermes-2, the dump file contains 53 Field3Ds, while the normal dump file contains 45 Field3Ds.
And part of that is just the metric, so I guess the overhead is less then 100% of the storage requirement of the run.

But if BOUT++ runs out of memory here, it would not be much worse then if not - the simulation finishes anyway.

I did notice a slow down in my simulations with this enabled, but that was when it was enabled all the time and the names did not get reset, thus the names have been growing exponentially.

Is that with the current form?

No, currently it is not noticeable. I activate it for all my runs.

But then they return either Field, or need to be overwritten.

It should almost certainly return Field& and then this wouldn't be an issue

That is exactly what I tried:

diff --git a/include/bout/field.hxx b/include/bout/field.hxx
index 04035f5b7..488aecaab 100644
--- a/include/bout/field.hxx
+++ b/include/bout/field.hxx
@@ -84,6 +84,14 @@ public:
     return *this;
   }
 
+  template <class... Types>
+  inline Field& setName(const std::string& name, Types... args) {
+#if BOUT_USE_TRACK
+    this->name = fmt::format(name, args...);
+#endif
+    return *this;
+  }
+
   std::string name;
 
 #if CHECK > 0
@@ -683,19 +691,4 @@ inline T floor(const T& var, BoutReal f, const std::string& rgn = "RGN_ALL") {
 
 #undef FIELD_FUNC
 
-template <typename T, typename = bout::utils::EnableIfField<T>, class... Types>
-inline void setName(T& f, const std::string& name, Types... args) {
-#if BOUT_USE_TRACK
-  f.name = fmt::format(name, args...);
-#endif
-}
-
-template <typename T, typename = bout::utils::EnableIfField<T>, class... Types>
-inline T setName(T&& f, const std::string& name, Types... args) {
-#if BOUT_USE_TRACK
-  f.name = fmt::format(name, args...);
-#endif
-  return f;
-}
-
 #endif /* FIELD_H */
diff --git a/src/mesh/coordinates.cxx b/src/mesh/coordinates.cxx
index 32774d622..643e148b1 100644
--- a/src/mesh/coordinates.cxx
+++ b/src/mesh/coordinates.cxx
@@ -1542,7 +1542,7 @@ Field3D Coordinates::Grad_par(const Field3D& var, CELL_LOC outloc,
   TRACE("Coordinates::Grad_par( Field3D )");
   ASSERT1(location == outloc || outloc == CELL_DEFAULT);
 
-  return setName(::DDY(var, outloc, method) * invSg(), "Grad_par({:s})", var.name);
+  return (::DDY(var, outloc, method) * invSg()).setName("Grad_par({:s})", var.name);
 }
 
 /////////////////////////////////////////////////////////
@@ -1601,7 +1601,7 @@ Field3D Coordinates::Div_par(const Field3D& f, CELL_LOC outloc,
     f_B.yup(i) = f.yup(i) / Bxy_floc.yup(i);
     f_B.ydown(i) = f.ydown(i) / Bxy_floc.ydown(i);
   }
-  return setName(Bxy * Grad_par(f_B, outloc, method), "Div_par({:s})", f.name);
+  return (Bxy * Grad_par(f_B, outloc, method)).setName("Div_par({:s})", f.name);
 }
 
 /////////////////////////////////////////////////////////
diff --git a/src/solver/impls/pvode/pvode.cxx b/src/solver/impls/pvode/pvode.cxx
index db28f64d8..c36985ae0 100644
--- a/src/solver/impls/pvode/pvode.cxx
+++ b/src/solver/impls/pvode/pvode.cxx
@@ -376,7 +376,7 @@ BoutReal PvodeSolver::run(BoutReal tout) {
 
       for (auto& f : f3d) {
         f.F_var->enableTracking(fmt::format("ddt_{:s}", f.name), debug);
-        setName(*f.var, f.name);
+        f.var->setName(f.name);
       }
       run_rhs(simtime);

gcc14 complains with:

/u/dave/soft/BOUT-dev/merge/src/mesh/coordinates.cxx: In member function ‘Field3D Coordinates::Grad_par(const Field3D&, CELL_LOC, const std::string&)’:                                                                                                                       /u/dave/soft/BOUT-dev/merge/src/mesh/coordinates.cxx:1545:56: error: could not convert ‘operator*(const Field3D&, const Field2D&)((* &((Coordinates*)this)->Coordinates::invSg())).Field3D::Field.Field::setName<std::__cxx11::basic_string<char, std::char_traits<char>, std:
:allocator<char> > >(std::__cxx11::basic_string<char>(((const char*)"Grad_par({:s})"), std::allocator<char>()), std::__cxx11::basic_string<char>(var.Field3D::Field.Field::name))’ from ‘Field’ to ‘Field3D’
 1545 |   return (::DDY(var, outloc, method) * invSg()).setName("Grad_par({:s})", var.name);      
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                        |
      |                                                        Field                       
/u/dave/soft/BOUT-dev/merge/src/mesh/coordinates.cxx: In member function ‘Field3D Coordinates::Div_par(const Field3D&, CELL_LOC, const std::string&)’:
/u/dave/soft/BOUT-dev/merge/src/mesh/coordinates.cxx:1604:55: error: could not convert ‘operator*(const Field2D&, const Field3D&)(Coordinates::Grad_par(const Field3D&, CELL_LOC, const std::string&)(f_B, outloc, (* & method))).Field3D::Field.Field::setName<std::__cxx11::
basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char>(((const char*)"Div_par({:s})"), std::allocator<char>()), std::__cxx11::basic_string<char>(f.Field3D::Field.Field::name))’ from ‘Field’ to ‘Field3D’
 1604 |   return (Bxy * Grad_par(f_B, outloc, method)).setName("Div_par({:s})", f.name);                                                                                                                                                                                            |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                       |
      |                                                       Field

gcc 9.4 is unable to correctly parse the construction for the function argument, if name.change is used directly. First making a copy seems to work around that issue.

ZedThree · 2024-04-08T09:56:12Z

@dschwoerer After discussing this with @bendudson, we're quite concerned about the potential overhead of the temporary field dumping mechanism, both in terms of memory and performance, for what seems like a small benefit when debugging. We'd really like to see some performance analysis and scaling to see it doesn't have an affect when disabled. I'm also worried that we'd need to see setName everywhere in the code for this to be really useful.

Could something similar not be achieved with a custom monitor instead?

Exposing more CVODE options to the user seems very straightforward and useful, so maybe that could be split out?

dschwoerer · 2024-04-08T10:40:52Z

@dschwoerer After discussing this with @bendudson, we're quite concerned about the potential overhead of the temporary field dumping mechanism, both in terms of memory and performance, for what seems like a small benefit when debugging.

But the memory overhead is only if the solver fails. Do you still think at that point that is an issue?
I know of @totork using BOUT++ in a inner-loop of an optimization script, where just failing vs crashing would matter. Thus having an option to disable would make sense.

We'd really like to see some performance analysis and scaling to see it doesn't have an affect when disabled.

I can certainly run blob2d for this branch and next. But I am certain there are no significant differences, thus I haven't done it. Would you like to see something else? Would just the BOUT++ internal timings sufficient? Would you like mpi runs?

I'm also worried that we'd need to see setName everywhere in the code for this to be really useful.

Sure, having them makes things more easily readable. But that is optional, and I can certainly add some if you think this is worth merging an wanted before merging. It would make BOUT_USE_TRACK also more useful. Right now

Could something similar not be achieved with a custom monitor instead?

I would not know how. If you can explain how to, I am happy to change the design.

Exposing more CVODE options to the user seems very straightforward and useful, so maybe that could be split out?

I can split them out 👍

github-actions

clang-tidy made some suggestions

src/solver/impls/euler/euler.cxx

src/solver/impls/pvode/pvode.cxx

dschwoerer · 2026-01-29T09:15:17Z

With cb0d3f1 setName is not that important anymore. The stacktrace is stored, so it is fairly easy to reliably figure out where things are comming from :-)

We could remove setName again, if that is desired ...

github-actions

clang-tidy made some suggestions

src/solver/impls/euler/euler.cxx

I think I addressed all requested changes

github-actions

clang-tidy made some suggestions

src/mesh/coordinates.cxx

ZedThree

There's some minor bits that would be nice to tidy up.

Please could you also:

add something about this to the debugging section in the manual?
change the PR title to something about tracking field operations for debugging?

src/solver/impls/euler/euler.cxx

ZedThree · 2026-02-02T15:22:16Z

src/solver/impls/euler/euler.cxx

+
+    if (mesh != nullptr) {
+      mesh->outputVars(debug);
+      debug["BOUT_VERSION"].force(bout::version::as_double);


Could do:

Suggested change

debug["BOUT_VERSION"].force(bout::version::as_double);

bout::experimental::addBuildFlagsToOptions(debug);

but the extra bits probably not necessary?

I do not think they are needed. I have moved it to the function, so we can change it later all in one place 👍

src/solver/impls/pvode/pvode.cxx

ZedThree · 2026-02-02T15:27:36Z

src/solver/impls/pvode/pvode.cxx

+      const std::string outname = fmt::format(
+          "{}/BOUT.debug.{}.nc",
+          Options::root()["datadir"].withDefault<std::string>("data"), BoutComm::rank());
+
+      bout::OptionsIO::create(outname)->write(debug);


Same comment as for the euler solver. In fact, if it's possible we might want to add this in more places, maybe worth a helper function to create these files?

I think it should be possible. The CVode solver even has a documented API to get the residua :-)

Allows to use other backends then netcdf

Co-authored-by: Peter Hill <peter.hill@york.ac.uk>

dschwoerer · 2026-02-03T13:50:44Z

Should be ready for re-review @ZedThree

github-actions

clang-tidy made some suggestions

github-actions · 2026-02-03T13:55:08Z

include/bout/options_io.hxx

+  /// Write some data to a file with a given name prefix
+  /// This will be done in parallel. If Mesh is given, also mesh data will be
+  /// added, which is needed for xBOUT or boutdata to read the files.
+  static void write(const std::string& prefix, Options data, Mesh* mesh = nullptr);


warning: no header providing "Mesh" is directly included [misc-include-cleaner]

include/bout/options_io.hxx:41:

- #ifndef OPTIONS_IO_H + #include "bout/field2d.hxx" + #ifndef OPTIONS_IO_H

github-actions · 2026-02-03T13:55:08Z

src/solver/impls/euler/euler.cxx

 #include <bout/openmpwrap.hxx>
 #include <bout/output.hxx>
+#include <bout/version.hxx>
+


warning: included header version.hxx is not used directly [misc-include-cleaner]

Suggested change

github-actions · 2026-02-03T13:55:08Z

src/solver/impls/euler/euler.cxx

+    const std::string outnumber =
+        dump_at_time < -3 ? fmt::format(".{}", debug_counter++) : "";
+    const std::string outname = fmt::format("BOUT.debug{}", outnumber);
+    bout::OptionsIO::write(outname, debug, mesh);


warning: 'Options' is deprecated: Please use a reference or .copy() instead [clang-diagnostic-deprecated-declarations]

bout::OptionsIO::write(outname, debug, mesh); ^

Additional context

include/bout/options.hxx:254: 'Options' has been explicitly marked deprecated here

[[deprecated("Please use a reference or .copy() instead")]] Options( ^

github-actions · 2026-02-03T13:55:08Z

src/solver/impls/euler/euler.cxx

+    const std::string outnumber =
+        dump_at_time < -3 ? fmt::format(".{}", debug_counter++) : "";
+    const std::string outname = fmt::format("BOUT.debug{}", outnumber);
+    bout::OptionsIO::write(outname, debug, mesh);


warning: no header providing "bout::OptionsIO" is directly included [misc-include-cleaner]

src/solver/impls/euler/euler.cxx:2:

- #include <bout/boutcomm.hxx> + #include "bout/options_io.hxx" + #include <bout/boutcomm.hxx>

github-actions · 2026-02-03T13:55:08Z

src/solver/impls/pvode/pvode.cxx

 #include <bout/sys/timer.hxx>
+#include <bout/unused.hxx>
+#include <bout/version.hxx>



warning: included header version.hxx is not used directly [misc-include-cleaner]

Suggested change

src/solver/impls/pvode/pvode.cxx

github-actions · 2026-02-03T13:55:08Z

src/solver/impls/pvode/pvode.cxx

+      for (auto& f : f3d) {
+        saveParallel(debug, f.name, *f.var);
+      }
+      bout::OptionsIO::write("BOUT.debug", debug, mesh);


warning: no header providing "bout::OptionsIO" is directly included [misc-include-cleaner]

src/solver/impls/pvode/pvode.cxx:26:

+ #include "bout/options_io.hxx"

dschwoerer added 9 commits March 19, 2024 15:51

Dump field before and after rhs evaluation for debugging

fee27c9

Add setName function

96da2e9

Set div_par and grad_par names

e56981c

Dump debug file if PVODE fails

6b2c132

Use the new track feature (better name required) to dump the different components of the ddt() as well as the residuum for the evolved fields.

Add tracking to Field3D

df2d661

This keeps track of all the changes done to the field and stores them to a OptionsObject.

Add tracking to Field3D

263f9fe

This keeps track of all the changes done to the field and stores them to a OptionsObject.

cvode: Add option to use Adams Moulton solver instead of BDF

bac4ca9

Expose more pvode option to user

708bdcb

Fix bad cherry-pick

d88b454

ZedThree reviewed Mar 19, 2024

View reviewed changes

include/bout/field.hxx Outdated Show resolved Hide resolved

dschwoerer added 2 commits March 19, 2024 16:35

Update to new API

023bc41

Fix documentation

affc995

Apply clang-format changes

71f5b6a

github-actions bot reviewed Mar 19, 2024

View reviewed changes

dschwoerer and others added 4 commits March 20, 2024 10:02

Apply recomendations from code-review

31fd461

Use more meaningful names

17e46cf

Apply clang-format changes

9c0ae16

Apply suggestions from code review

4a17b49

github-actions bot reviewed Mar 20, 2024

View reviewed changes

Workaround for gcc 9.4

2f7c3c0

gcc 9.4 is unable to correctly parse the construction for the function argument, if name.change is used directly. First making a copy seems to work around that issue.

dschwoerer added 2 commits April 9, 2024 15:15

Add option to debug on failure

d611758

Add option to euler solver to dump debug info

db96b7e

Apply clang-format changes

f2ed814

github-actions bot reviewed Jan 29, 2026

View reviewed changes

src/solver/impls/euler/euler.cxx Show resolved Hide resolved

src/solver/impls/euler/euler.cxx Show resolved Hide resolved

src/solver/impls/pvode/pvode.cxx Outdated Show resolved Hide resolved

src/solver/impls/pvode/pvode.cxx Outdated Show resolved Hide resolved

Apply clang-format suggestions

e3f8923

github-actions bot reviewed Jan 29, 2026

View reviewed changes

src/solver/impls/euler/euler.cxx Show resolved Hide resolved

dschwoerer and others added 2 commits January 29, 2026 10:33

Apply clang-format suggestions

873c94f

Apply clang-format changes

7e5c311

dschwoerer added 3 commits February 2, 2026 11:49

do not TRACK names

e5f2d37

Remove setName again

751abc4

Update generated source code

3cc061b

dschwoerer force-pushed the pvcode-cvode-improvements branch from c8c1180 to 3cc061b Compare February 2, 2026 10:51

dschwoerer and others added 3 commits February 2, 2026 10:52

[bot] Apply format changes

5230138

[bot] Add last format changes commit to ignore file

03283f4

Merge branch 'next' into pvcode-cvode-improvements

fc2e28f

github-actions bot reviewed Feb 2, 2026

View reviewed changes

src/mesh/coordinates.cxx Outdated Show resolved Hide resolved

dschwoerer commented Feb 2, 2026

View reviewed changes

src/mesh/coordinates.cxx Outdated Show resolved Hide resolved

Revert accidential typo

26651cc

ZedThree requested changes Feb 2, 2026

View reviewed changes

dschwoerer and others added 7 commits February 3, 2026 14:04

Fix comment

f319c61

Add simple method to write data in parallel

1519bf2

Use new interface to write data

b61f815

Allows to use other backends then netcdf

Add docs on using solver-dumping

eab6b03

Improve docs

508dcda

Add more docs

c9732b7

simplify to_load creation

9b96dad

Co-authored-by: Peter Hill <peter.hill@york.ac.uk>

github-actions bot reviewed Feb 3, 2026

View reviewed changes

dschwoerer added 2 commits February 3, 2026 15:13

Fixup to_load

94f9ea6

Take reference to avoid copy

6ec3917

	debug["BOUT_VERSION"].force(bout::version::as_double);
	bout::experimental::addBuildFlagsToOptions(debug);

Pvcode + Cvode improvements #2889

Are you sure you want to change the base?

Pvcode + Cvode improvements #2889

Uh oh!

Conversation

dschwoerer commented Mar 19, 2024

Uh oh!

Uh oh!

ZedThree commented Mar 19, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dschwoerer commented Mar 19, 2024

Uh oh!

dschwoerer commented Mar 19, 2024 via email

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot Mar 20, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dschwoerer commented Mar 20, 2024

Uh oh!

ZedThree commented Mar 20, 2024

Uh oh!

dschwoerer commented Mar 20, 2024

Uh oh!

ZedThree commented Apr 8, 2024

Uh oh!

dschwoerer commented Apr 8, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dschwoerer commented Jan 29, 2026

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ZedThree left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ZedThree Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

dschwoerer Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ZedThree Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!