Getting PyTables to Play Nice with Windows

I originally started this as a wiki page on my fork of ViTables. After trying to understand why ViTables would crash on some Windows boxes but not others, I found out that it had nothing to do with ViTables. So, I’ll just put it here. These are primarily the notes I wrote while digging into the problem.

tl;dr: Your Python installation on Windows is stupid and needs to be reinstalled.

Running Notes

So, apparently something screwy is going on with Python on Windows. Everything seems to work just dandy fine on other systems. Digging around, it appears that the problem is isolated exclusively in the Anaconda distribution of Python. More to the point, the PyTables packaging within the Anaconda distribution. A quick check is to run

$ python3 -c 'import tables; tables.test()'

and see if it works. Cygwin is just plain happy to run the tests using the native Python in Cygwin. Only one error related to the type of error raised by one method. WinPython runs to the end and doesn’t even complain once.

Using Anaconda, it runs for about 2.5 lines, but then it pops up a window saying “python.exe has stopped working.” Opening up a debug instance (after fighting with the community version of Visual Studio 2015) shows a trace to netdll.dll through hdf5extension.pyd. Searching the site-package tree reveals the offending file is in the tables package. file tells me the .pyd file is a DLL for Windows, but nm and objdump don’t give me any useful information. I assume it should be linking to the HDF5 libraries because PyTables is a simple extension of the HDF5 file. greping around, I find

$ find Lib -name 'hdf5*' | grep -v '__pycache__'
...
Lib/site-packages/tables/hdfExtension.py
Lib/site-packages/tables/hdfextension.pyd

$ find libs -name 'hdf5*' | grep -v '__pycache__'

$ find Library -name 'hdf5*' | grep -v '__pycache__'
Library/bin/hdf5.dll
Library/bin/hdf5_cpp.dll
Library/bin/hdf5_hl.dll
...
Library/lib/hdf5.lib
Library/lib/hdf5_cpp.lib
...

So, the libraries are there, but the runtime cannot find them. Taking a look at the WinPython distribution we find

$ find Lib -name 'hdf5*' | grep -v '__pycache__'
...
Lib/site-packages/tables/hdf5.dll
Lib/site-packages/tables/hdfExtension.py
Lib/site-packages/tables/hdfextension.pyd
Lib/site-packages/tables/hdf5_hl.dll

$ find libs -name 'hdf5*' | grep -v '__pycache__'

and no Library directory. The key point is the DLLs are next to the files. This is making no sense to me.

Rooting around the internet some more, I come across the search order for DLLs. Checking my PATH variable, I see that indeed /c/path/to/anaconda/Library/bin is not on my path. Thus, the current directory approach of WinPython works out of the box. Trying to add this to the path didn’t help on the outset

And if I go to a computer that has Anaconda installed that I know still works, the DLLS are installed in the Lib/site-packages/tables directory! (I hate Windows sometimes). I tried reinstalling Anaconda using the 2.3.0 version available at the time of writing. That didn’t help.

A (Hacked) Solution

It appears like we have to do things the hard way if we want to use Anaconda. First, uninstall PyTables from Anaconda

$ conda uninstall pytables

Now, we need to us pip

$ pip install tables

Odds are, this will spew out a long and confusing string of error messages. If you see something referring to “MSC V.1600 64 bit (AMD64)” and “Microsoft Visual C++ 10.0 is required,” it means you will need to hunt down the proper version of the version of Visual Studio. In this case, you want Visual Studio 2010. Sadly, that does not appear to be a viable option. I can get the professional version installed from my IT department, but I was not able to find a legitimate link from Microsoft. I’m sure I could dig up a link from a non-Microsoft source, but that would not be a good idea in my opinion.

A (Real) Solution

A real solution is to just install a version of Python that stashes the DLLs somewhere in the modules can find. Most likely, this will be one that installs the libraries into the site-package/tables directory just like WinPython does.

Written on August 10, 2015