When Things Go Wrong
- Contents
-
- Finding out What Went Wrong
- Buildtime Warnings
- When an Analysis step runs, it produces a warnings file (named warnproject.txt) in the spec file's directory. Generally, most of these warnings are harmless. For example, os.py (which is cross-platform) works by figuring out what platform it is on, then importing (and rebinding names from) the appropriate platform-specific module. So analyzing os.py will produce a set of warnings like:
-
- W: no module named dos (conditional import by os)
- W: no module named ce (conditional import by os)
- W: no module named os2 (conditional import by os)
-
- Note that the analysis has detected that the import is within a conditional block (an if statement). The analysis also detects if an import within a function or class, (delayed) or at the top level. A top-level, non-conditional import failure is really a hard error. There's at least a reasonable chance that conditional and / or delayed import will be handled gracefully at runtime.
- Ignorable warnings may also be produced when a class or function is declared in a package (an __init__.py module), and the import specifies package.name. In this case, the analysis can't tell if name is supposed to refer to a submodule of package.
- Warnings are also produced when an __import__, exec or eval statement is encountered. The __import__ warnings should almost certainly be investigated. Both exec and eval can be used to implement import hacks, but usually their use is more benign.
- Any problem detected here can be handled by hooking the analysis of the module. See Listing Hidden Imports below for how to do it.
- Getting Debug Messages
- Setting debug=1 on an EXE will cause the executable to put out progress messages (for console apps, these go to stdout; for Windows apps, these show as MessageBoxes). This can be useful if you are doing complex packaging, or your app doesn't seem to be starting, or just to learn how the runtime works.
- Getting Python's Verbose Imports
- You can also pass a -v (verbose imports) flag to the embedded Python. This can be extremely useful. I usually try it even on apparently working apps, just to make sure that I'm always getting my copies of the modules and no import has leaked out to the installed Python.
- You set this (like the other runtime options) by feeding a phone TOC entry to the EXE. The easiest way to do this is to change the EXE from:
-
- EXE(..., anal.scripts, ....)
- to
- EXE(..., anal.scripts + [('v', '', 'OPTION')], ...)
-
- These messages will always go to stdout, so you won't see them on Windows if console=0.
-
- Helping Installer Find Modules
- Extending the Path
- When the analysis phase cannot find needed modules, it may be that the code is manipulating sys.path. The easiest thing to do in this case is tell Analysis about the new directory through the second arg to the constructor.
-
- anal = Analysis(['somedir/myscript.py'],
- ['path/to/thisdir', 'path/to/thatdir'])
-
- ['somedir', 'path/to/thisdir', 'path/to/thatdir'] + sys.path
-
- You can do the same when running Makespec
- Makespec.py --paths=path/to/thisdir;path/to/thatdir ...
-
- (on *nix, use : as the path separator).
- Listing Hidden Imports
- Hidden imports are fairly common. These can occur when the code is using __import__ (or, perhaps exec or eval), in which case you will see a warning in the warnproject.txt file. They can also occur when an extension module uses the Python/C API to do an import, in which case Analysis can't detect anything. You can verify that hidden import is the problem by using Python's verbose imports flag. If the import messages say "module not found", but the warnproject.txt file has no "no module named..." message for the same module, then the problem is a hidden import.
- Hidden imports are handled by hooking the module (the one doing the hidden imports) at Analysis time. Do this by creating a file named hook-module.py (where module is the fully-qualified Python name, eg, hook-xml.dom.py), and placing it in the hooks package under Installer's root directory, (alternatively, you can save it elsewhere, and then use the hookspath arg to Analysis so your private hooks directory will be searched). Normally, it will have only one line:
-
- hiddenimports = ['module1', 'module2']
- module1 and module2. (Full details on the analysis-time hook mechanism is here).
- If you successfully hook a publicly distributed module in this way, please send me the hook so I can make it available to others.
- Extending a Package's __path__
- Python allows a package to extend the search path used to find modules and sub-packages through the __path__ mechanism. Normally, a package's __path__ has only one entry - the directory in which the __init__.py was found. But __init__.py is free to extend its __path__ to include other directories. For example, the win32com.shell.shell module actually resolves to win32com/win32comext/shell/shell.pyd. This is because win32com/__init__.py appends ../win32comext to its __path__.
- Because the __init__.py is not actually run during an analysis, we use the same hook mechanism we use for hiddenimports. A static list of names won't do, however, because the new entry on __path__ may well require computation. So hook-module.py should define a method hook(mod). The mod argument is an instance of mf.Module which has (more or less) the same attributes as a real module object. The hook function should return a mf.Module instance - perhaps a brand new one, but more likely the same one used as an arg, but mutated. See mf for details, and hook/hook-win32com.py for an example.
- Note that manipulations of __path__ hooked in this way apply to the analysis, and only the analysis. That is, at runtime win32com.shell is resolved the same way as win32com.anythingelse, and win32com.__path__ knows nothing of ../win32comext.
- Once in awhile, that's not enough.
- Changing Runtime Behavior
- More bizarre situations can be accomodated with runtime hooks. These are small scripts that manipulate the environment before your main script runs, effectively providing additional top-level code to your script.
- At the tail end of an analysis, the module list is examined for matches in rthooks.dat, which is the string representation of a Python dictionary. The key is the module name, and the value is a list of hook-script pathnames.
- So putting an entry:
-
- 'somemodule': ['path/to/somescript.py'],
- rthooks.dat is almost the same thing as
-
- anal = Analysis(['path/to/somescript.py', 'main.py'], ...
- path/to/somescript.py will not be analyzed, (that's not a feature - I just haven't found a sane way fit the recursion into my persistence scheme).
- Hooks done in this way, while they need to be careful of what they import, are free to do almost anything. One provided hook sets things up so that win32com can generate modules at runtime (to disk), and the generated modules can be found in the win32com package.
- Adapting to being "frozen"
- In most sophisticated apps, it becomes necessary to figure out (at runtime) whether you're running "live" or "frozen". For example, you might have a configuration file that (running "live") you locate based on a module's __file__ attribute. That won't work once the code is packaged up. You'll probably want to look for it based on sys.executable instead.
- The run executables set sys.frozen=1 (and, for in-process COM servers, the embedding DLL sets sys.frozen='dll').
- For really advanced users, you can access the iu.ImportManager as sys.importManager. See iu for how you might make use of this fact.
- Accessing Data Files
- In a --onedir distribution, this is easy: pass a list of your data files (in TOC format) to the COLLECT, and they will show up in the distribution
- directory tree. The name in the (name, path, 'DATA') tuple can be a relative
- path name. Then, at runtime, you can use code like this to find the file:
- os.path.join(os.path.dirname(sys.executable), relativename))
-
- In a --onefile, it's a bit trickier. You can cheat, and add the files to the EXE as BINARY. They will then be extracted at runtime into the work directory by the C code (which does not create directories, so the name must be a plain name), and cleaned up on exit. The work directory is best found by os.environ['_MEIPASS2']. Be awawre, though, that if you use --strip or --upx, strange things may happen to your data - BINARY is really for shared libs / dlls.
- If you add them as 'DATA' to the EXE, then it's up to you to extract them.
- Use code like this:
- import sys, carchive
- this = carchive.CArchive(sys.executable)
- data = this.extract('mystuff')[1]
- support/unpackTK.py for an advanced example (the TCL and TK lib files are in a PKG which is opened in place, and
- then extracted to the filesystem).
- Reporting Bugs
- Report bugs (or feature requests) here. Please make sure you set Product to Installer. (If you choose not to become a registered user, please include some contact information in the report itself.) Subscribe to the Installer Mailing List to discuss Installer related issues. And don't forget to show your appreciation here.
- Miscellaneous
- Pmw
- Pmw comes with a script named bundlepmw in the bin directory. If you follow the instructions in that script, you'll end up with a module named Pmw.py. Ensure that Builder finds that module and not the development package.
- Win9xpopen
- If you're using popen on Windows and want the code to work on Win9x, you'll need to distribute
- win9xpopen.exe with your app. On older Pythons with Win32all, this would apply to Win32pipe and win32popenWin9x.exe. (On yet older Pythons, no form of popen
- worked on Win9x).
-
- |
- 
- figure 3 - Self Extracting Executable