pytests out of date in release 3.1.2

[omsai]

Looks like minor changes to the pytests are necessary like changing import PyUQTk.array as uqtkarray to from PyUQTk import uqtkarray etc:

$ spack load uqtk@3.1.2 +pyuqtk
$ git clone --depth=1 https://github.com/sandialabs/UQTk.git -b v3.1.2
$ cd UQTk/PyUQTk/pytests/
$ for file in *.py; do echo "=== $file ==="; python3 $file 1>/dev/null; done
=== PyArrayTest2.py ===
=== PyArrayTest.py ===
=== PyBADPTest.py ===
Traceback (most recent call last):
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyBADPTest.py", line 137, in <module>
    pc_model0, c_k0, totquat0 = forward_propagation(mu, sigma, \
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyBADPTest.py", line 85, in forward_propagation
    qdpts, totquat= pce_tools.UQTkGetQuadPoints(pc_model)
NameError: name 'pce_tools' is not defined
=== PyBCSTest1D.py ===
=== PyBCSTest2D.py ===
=== PyBCSTest.py ===
Traceback (most recent call last):
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyBCSTest.py", line 105, in <module>
    regmodel = bcs.bcsreg(ndim=2,pcorder=pcorder,pctype="LU")
NameError: name 'bcs' is not defined
=== PyHMCMCTest.py ===
Traceback (most recent call last):
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyHMCMCTest.py", line 118, in <module>
    qz =  exp(-U(array(zip(qx.flatten(),qy.flatten()))))
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyHMCMCTest.py", line 57, in U
    return b*(q[:,1] - q[:,0]**2)**2 + (a - q[:,0])**2
IndexError: index 1 is out of bounds for axis 1 with size 1
=== PyMCMC2dTest.py ===
Traceback (most recent call last):
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyMCMC2dTest.py", line 52, in <module>
    class pyLikelihood(uqtkmcmc.LikelihoodBase):
NameError: name 'uqtkmcmc' is not defined
=== PyMCMCTest.py ===
Traceback (most recent call last):
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyMCMCTest.py", line 64, in <module>
    class pyLikelihood(uqtkmcmc.LikelihoodBase):
NameError: name 'uqtkmcmc' is not defined
=== PyModTest.py ===
Traceback (most recent call last):
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyModTest.py", line 52, in <module>
    import _tools
ImportError: /home/pnanda/spack/opt/spack/linux-ubuntu18.04-westmere/gcc-11.3.0/uqtk-3.1.2-kfxhs7iftxrsy5isacage3clpadbtteo/PyUQTk/_tools.cpython-39-x86_64-linux-gnu.so: undefined symbol: _gfortran_transfer_character_write
=== PyPCE1dTest.py ===
Traceback (most recent call last):
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyPCE1dTest.py", line 49, in <module>
    x = uqtkarray.dblArray2D()
NameError: name 'uqtkarray' is not defined
=== PyPCE2dTest.py ===
Traceback (most recent call last):
  File "/home/pnanda/src/UQTk/PyUQTk/pytests/PyPCE2dTest.py", line 49, in <module>
    x = uqtkarray.dblArray2D()
NameError: name 'uqtkarray' is not defined
=== PyQuadTest.py ===
$ grep import PyPCE1dTest.py 
from __future__ import print_function # To make print() in Python 2 behave like in Python 3
import sys
	from numpy import *
	from matplotlib.pyplot import *
	import PyUQTk.array as uqtkarray
	import PyUQTk.quad as uqtkquad
	import PyUQTk.pce as uqtkpce
$ python3
Python 3.9.13 (main, Jun 15 2022, 19:27:16) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import PyUQTk.array
Scipy and/or matplotlib may need to be installed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'PyUQTk.array'
>>> import PyUQTk.quad
>>> import PyUQTk.pce
>>> from numpy import *
>>> from matplotlib.pyplot import *
$ python3
Python 3.9.13 (main, Jun 15 2022, 19:27:16) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyUQTk import uqtkarray
Scipy and/or matplotlib may need to be installed
>>> 

[bjdebus]

Hi @omsai , thanks for reaching out. I am actually about to push UQTk v 3.1.3 to github, so your request is very timely.

Can you tell me which platform you are running on so I can test it there.

Also, to clarify, I am not familiar with spack, but it looks like you are running the PyUQTk tests before compiling UQTk? A lot of the modules you are trying to load will not be available until you compile UQTk, and then run ctest from the build folder.

Either way, I will double check our Python tests.

Thanks again!
Bert

[omsai]

Platform: an old Intel Westmere machine running Ubuntu 18.04.

Indeed, I already compiled and loaded UQTk. The pytests directory is not included in the release tarball. I think it would be nice if it was, because these are standalone tests that don't need to be run at build time and are therefore useful in an installation just like the programs in $install_prefix/bin/tests/.

I probably falsely assumed that ctest would not run pytests, but now that you mention it I'll double check.

[bjdebus]

Ok, let me know what you find when you run ctest in the build folder. The PyTests are actually not standalone tests. They rely on the compiled UQTk binaries.

thanks!

[omsai]

Right, I understand that you have to load the UQTk binaries, and I am doing that. I'm having some issues running ctest directly at the minute via spack, but I expect at least some of the errors above in the top comment are reproducible. If they're not, I'll have a go at investigating them in 3.1.3.

spack is a project out of LLNL since about 2015 and is a package manager for HPC systems. This is what the recipe for uqtk looks like: https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/uqtk/package.py

Below are the folks who've contributed to the spack package so far: maybe they're all from the community:

$ spack blame uqtk
LAST_COMMIT   LINES  %      AUTHOR             EMAIL
just now      60     58.8   Not Committed Yet  <not.committed.yet>
9 months ago  1      1.0    Tom Scogland       <tscogland@llnl.gov>
a year ago    21     20.6   Paul Adamson       <5460406+padamson@users.noreply.github.com>
2 years ago   10     9.8    Tomoki, Karatsu    <49965247+t-karatsu@users.noreply.github.com>
2 years ago   10     9.8    sknigh             <sknigh@sandia.gov>
                                               
just now      102    100.0

I've been making changes to get it running with pymc because the spack package still depends on the 2 versions old pymc3:

diff --git a/var/spack/repos/builtin/packages/uqtk/package.py b/var/spack/repos/builtin/packages/uqtk/package.py
index ca23b6dc73..a3a8e15203 100644
--- a/var/spack/repos/builtin/packages/uqtk/package.py
+++ b/var/spack/repos/builtin/packages/uqtk/package.py
@@ -15,6 +15,7 @@ class Uqtk(CMakePackage):
     git = "https://github.com/sandialabs/UQTk.git"
 
     version("master", branch="master")
+    version("3.1.2", sha256="57ce0cea709777cbefb46f3bd86a0996a0ed5f50fc54cc297599df6e4bb9ab83")
     version("3.1.0", sha256="56ecd3d13bdd908d568e9560dc52cc0f66d7bdcdbe64ab2dd0147a7cf1734f97")
     version("3.0.4", sha256="0a72856438134bb571fd328d1d30ce3d0d7aead32eda9b7fb6e436a27d546d2e")
 
@@ -23,7 +24,7 @@ class Uqtk(CMakePackage):
     )
 
     depends_on("expat")
-    depends_on("sundials", when="@3.1.0:")
+    depends_on("sundials@:5", when="@3.1.0:")
     depends_on("blas", when="@3.1.0:")
     depends_on("lapack", when="@3.1.0:")
 
@@ -31,15 +32,68 @@ class Uqtk(CMakePackage):
     depends_on("py-numpy", when="+pyuqtk")
     depends_on("py-scipy", when="+pyuqtk")
     depends_on("py-matplotlib", when="+pyuqtk")
-    depends_on("py-pymc3", when="+pyuqtk")
+    depends_on("py-pymc", when="+pyuqtk")
     depends_on("swig", when="+pyuqtk")
 
-    # Modify the process of directly specifying blas/lapack
-    # as the library name.
-    patch("remove_unique_libname.patch", when="@3.1.0:")
+    # The two patches for 3.1.0 fail with 3.1.2, therefore convert the patches
+    # to more versatile and reliable sed-like filter_file substitutions.
+    def patch(self):
+        cmakelists = find(".", "CMakeLists.txt")
 
-    # Do not link the gfortran library when using the Fujitsu compiler.
-    patch("not_link_gfortran.patch", when="@3.1.0:%fj")
+        # Modify the process of directly specifying blas/lapack as the library
+        # name.
+        if "@3.1.0:" in self.spec:
+            # Replace duplicate entries.
+            filter_file(
+                (
+                    r"(.*[tT][aA][rR][gG][eE][tT]_[lL][iI][nN][kK]_"
+                    r"[lL][iI][bB][rR][aA][rR][iI][eE][sS].+)"
+                    r"lapack (\${LAPACK_LIBRARIES}.+)"
+                ),
+                r"\1 \2",
+                *cmakelists,
+            )
+            filter_file(
+                (
+                    r"(.*[tT][aA][rR][gG][eE][tT]_[lL][iI][nN][kK]_"
+                    r"[lL][iI][bB][rR][aA][rR][iI][eE][sS].+)"
+                    r"blas (\${BLAS_LIBRARIES}.+)"
+                ),
+                r"\1 \2",
+                *cmakelists,
+            )
+
+            # Replace with the variable.
+            filter_file(
+                (
+                    r"(.*[tT][aA][rR][gG][eE][tT]_[lL][iI][nN][kK]_"
+                    r"[lL][iI][bB][rR][aA][rR][iI][eE][sS].+)"
+                    r"lapack(.+)"
+                ),
+                r"\1${LAPACK_LIBRARIES}\2",
+                *cmakelists,
+            )
+            filter_file(
+                (
+                    r"(.*[tT][aA][rR][gG][eE][tT]_[lL][iI][nN][kK]_"
+                    r"[lL][iI][bB][rR][aA][rR][iI][eE][sS].+)"
+                    r"blas(.+)"
+                ),
+                r"\1${BLAS_LIBRARIES}\2",
+                *cmakelists,
+            )
+
+        # Do not link the gfortran library when using the Fujitsu compiler.
+        if "@3.1.0:%fj" in self.spec:
+            filter_file(
+                (
+                    r"(.*[tT][aA][rR][gG][eE][tT]_[lL][iI][nN][kK]_"
+                    r"[lL][iI][bB][rR][aA][rR][iI][eE][sS].+)"
+                    r" gfortran(.+stdc[+][+].+)"
+                ),
+                r"\1\2",
+                *cmakelists,
+            )
 
     @when("@3.1.0:")
     def cmake_args(self):
@@ -50,6 +104,7 @@ def cmake_args(self):
         blas_libs = spec["blas"].libs.joined(";")
 
         args = [
+            self.define("CMAKE_CXX_STANDARD", 14),
             self.define("CMAKE_SUNDIALS_DIR", spec["sundials"].prefix),
             self.define("LAPACK_LIBRARIES", lapack_libs),
             self.define("BLAS_LIBRARIES", blas_libs),

(I probably need to remove the c++14 standard - that was an artifact of using SUNDIALS version 6 instead of version 5).

[bjdebus]

Hi @omsai ,

thanks for clarifying. I looked over the spack package for UQTk and it looks like it is using a version of UQTk that is quite old. We have actually moved away from swig to pybind. Also, version 3.1.3 will pull in Sundials version 6, so you will need the C++14 standard.

I am very interested in helping you set up spack for UQTk. However, I would propose you hold off a day or so till I push UQTk 3.1.3 onto github. I expect to do so this evening or tomorrow.

Kind Regards,
Bert

[bjdebus]

Hi @omsai

I just pushed UQTk version 3.1.3 to github.

I look forward to hearing about your experiences with this new release. I would be happy to work with you to build a spack for UQTk.

thanks!
Bert

[bjdebus]

Hi @omsai , do you have an update on how the PyTests are behaving with UQTk 3.1.3? If not, can I go ahead and close this issue?

Regards,
Bert

[omsai]

@bjdebus Thank you! I haven't had a look yet and will try your new version and retest tomorrow. Was buried in writing these past couple of days.

Actually I was wondering if you may be familiar with UQTk's pymc dependency? The spack package declares a dependency on version 3 of it but my suspicion is that the version was used because spack's pymc dependency is out of date and because I was updating pymc from version 3 to the current 5 I also got interested in UQTk it's currently that's the only spack package that depends on pymc3. I see that pymc is being used in PyUQTk/inference/postproc.py#L43L-53L as well as some mentions in some of the example python notebooks. Usually I can rely on requirements.txt, setup.py or pyproject.toml files to determine what dependency versions are, but because cmake is being used throughout for the build system, the python dependency tree is not as clear to me but I may have missed it.

I would be happy to work with you to build a spack for UQTk.

That's fantastic! I'll add you along with myself to the list of maintainers so that you get notified on any activity with the py-uqtk spack package. When I'm done updating pymc to version 5 I'll create an issue / PR with the updates to the spack UQTk package with you mentioned.

[bjdebus]

Sounds good. To my knowledge, PyMC is only used for some convergence tests in the MCMC postprocessing tools (inference/postproc.py). Also, these tests are not critical. If PyMC is not detected, then those tests are simple disabled.
It's been a few years since I touched or used that code, so I am not clear on whether these work with PyMC 3 or 5. I will take a look at that over the next couple of days.

[omsai]

With 3.1.3 I now only get 6 test failures instead of 8, though some of the errors are a little different:

$ for file in *.py; do echo "=== $file ==="; python3 $file 1>/dev/null; done
=== PyArrayTest.py ===
=== PyArrayTest2.py ===
=== PyBADPTest.py ===
=== PyBCSTest.py ===
Traceback (most recent call last):
  File "/Users/pnanda/src/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/uqtk-3.1.3-c7av5vwp3xy4o2ve6ca62piba27ur7hv/PyUQTk/pytests/PyBCSTest.py", line 108, in <module>
    regmodel = bcs.bcsreg(ndim=2,pcorder=pcorder,pctype="LU")
AttributeError: module 'bcs' has no attribute 'bcsreg'
=== PyBCSTest1D.py ===
=== PyBCSTest2D.py ===
=== PyGalerkinTest.py ===
=== PyHMCMCTest.py ===
Traceback (most recent call last):
  File "/Users/pnanda/src/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/uqtk-3.1.3-c7av5vwp3xy4o2ve6ca62piba27ur7hv/PyUQTk/pytests/PyHMCMCTest.py", line 139, in <module>
    qz =  exp(-U(array(zip(qx.flatten(),qy.flatten()))))
  File "/Users/pnanda/src/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/uqtk-3.1.3-c7av5vwp3xy4o2ve6ca62piba27ur7hv/PyUQTk/pytests/PyHMCMCTest.py", line 78, in U
    return b*(q[:,1] - q[:,0]**2)**2 + (a - q[:,0])**2
IndexError: index 1 is out of bounds for axis 1 with size 1
=== PyMCMC2dTest.py ===
Traceback (most recent call last):
  File "/Users/pnanda/src/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/uqtk-3.1.3-c7av5vwp3xy4o2ve6ca62piba27ur7hv/PyUQTk/pytests/PyMCMC2dTest.py", line 62, in <module>
    class pyLikelihood(uqtkmcmc.LikelihoodBase):
NameError: name 'uqtkmcmc' is not defined
=== PyMCMCTest.py ===
  File "/Users/pnanda/src/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/uqtk-3.1.3-c7av5vwp3xy4o2ve6ca62piba27ur7hv/PyUQTk/pytests/PyMCMCTest.py", line 42
    try:
    ^
IndentationError: expected an indented block after 'except' statement on line 41
=== PyModTest.py ===
=== PyPCE1dTest.py ===
Traceback (most recent call last):
  File "/Users/pnanda/src/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/uqtk-3.1.3-c7av5vwp3xy4o2ve6ca62piba27ur7hv/PyUQTk/pytests/PyPCE1dTest.py", line 57, in <module>
    x = uqtkarray.dblArray2D()
NameError: name 'uqtkarray' is not defined. Did you mean: 'bytearray'?
=== PyPCE2dTest.py ===
Traceback (most recent call last):
  File "/Users/pnanda/src/spack/opt/spack/darwin-ventura-skylake/gcc-12.2.0/uqtk-3.1.3-c7av5vwp3xy4o2ve6ca62piba27ur7hv/PyUQTk/pytests/PyPCE2dTest.py", line 58, in <module>
    x = uqtkarray.dblArray2D()
NameError: name 'uqtkarray' is not defined. Did you mean: 'bytearray'?
=== PyQuadTest.py ===
=== PyRegressionTest.py ===

I need to cleanup several other spack files and upload a patch set for you to fully reproduce the issue. This is how I ran the tests my yet as uncommitted uqtk:

$ pwd
~/src
$ git clone https://github.com/sandialabs/UQTk.git -b v3.1.3
...
$ spack load uqtk@3.1.3
$ spack cd -i uqtk@3.1.3
$ cd PyUQTk/
$ # pytests directory is not installed, so copy it over from the git branch:
$ cp -av ~/src/UQTk/PyUQTk/pytests .
$ cp -av ~/src/UQTk/PyUQTk/pytests .
/Users/pnanda/src/UQTk/PyUQTk/pytests -> ./pytests
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyModTest.py -> ./pytests/PyModTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyPCE2dTest.py -> ./pytests/PyPCE2dTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyMCMCTest.py -> ./pytests/PyMCMCTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyQuadTest.py -> ./pytests/PyQuadTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyHMCMCTest.py -> ./pytests/PyHMCMCTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/CMakeLists.txt -> ./pytests/CMakeLists.txt
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyArrayTest2.py -> ./pytests/PyArrayTest2.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyPCE1dTest.py -> ./pytests/PyPCE1dTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyBCSTest.py -> ./pytests/PyBCSTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyRegressionTest.py -> ./pytests/PyRegressionTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyBCSTest1D.py -> ./pytests/PyBCSTest1D.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyBCSTest2D.py -> ./pytests/PyBCSTest2D.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyGalerkinTest.py -> ./pytests/PyGalerkinTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyMCMC2dTest.py -> ./pytests/PyMCMC2dTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyArrayTest.py -> ./pytests/PyArrayTest.py
/Users/pnanda/src/UQTk/PyUQTk/pytests/PyBADPTest.py -> ./pytests/PyBADPTest.py
$ cd pytests/
$ for file in *.py; do echo "=== $file ==="; python3 $file 1>/dev/null; done
(Then one gets the test output above)

[bjdebus]

Thanks for this update. Once you have updated the spack files, let me know and I will try to replicate these issues. I will need to read up a bit on spack to understand how the building and testing happens.

Thanks,
Bert