Version bump: 0.11.4

Fixed classifiers in setup.py

.gitignore

@@ -22,4 +22,10 @@ coverage.xml
 nosetests.xml
 junit-py25.xml
 junit-py26.xml
-junit-py27.xml
+junit-py27.xml
+
+# tox noise
+.tox
+
+# pyenv noise
+.python-version

.travis.yml

@@ -0,0 +1,10 @@
+language: python
+python:
+  - 2.6
+  - 2.7
+  - 3.2
+  - 3.3
+  - 3.4
+install:
+  - python setup.py install
+script: python test_tablib.py

AUTHORS

@@ -4,9 +4,14 @@ various contributors:
 Development Lead
 ````````````````
 
-- Kenneth Reitz <_@kennethreitz.com>
+- Kenneth Reitz <me@kennethreitz.org>
 
 
+Core Contributors
+`````````````````
+
+- Iuri de Silvio <iurisilvio@gmail.com>
+
 Patches and Suggestions
 ```````````````````````
 
@@ -15,4 +20,17 @@ Patches and Suggestions
 - Luca Beltrame
 - Benjamin Wohlwend
 - Erik Youngren
-- Mark Rogers
+- Mark Rogers
+- Mark Walling
+- Mike Waldner
+- Joel Friedly
+- Jakub Janoszek
+- Marc Abramowitz
+- Alex Gaynor
+- James Douglass
+- Tommy Anthony
+- Rabin Nankhwa
+- Marco Dallagiacoma
+- Mathias Loesch
+- Tushar Makkar
+- Andrii Soldatenko

HISTORY.rst

@@ -1,12 +1,88 @@
 History
 -------
 
-0.9.9 (?)
-+++++++++
+0.11.4 (2017-01-23)
++++++++++++++++++++
+
+- Use built-in `json` package if available
+- Support Python 3.5+ in classifiers
+
+** Bugfixes **
+
+- Fixed textual representation for Dataset with no headers
+- Handle decimal types
+
+0.11.3 (2016-02-16)
++++++++++++++++++++
+
+- Release fix.
+
+0.11.2 (2016-02-16)
++++++++++++++++++++
+
+**Bugfixes**
+
+- Fix export only formats.
+- Fix for xlsx output.
+
+0.11.1 (2016-02-07)
++++++++++++++++++++
+
+**Bugfixes**
+
+- Fixed packaging error on Python 3.
+
+
+0.11.0 (2016-02-07)
++++++++++++++++++++
+
+**New Formats!**
+
+- Added LaTeX table export format (``Dataset.latex``).
+- Support for dBase (DBF) files (``Dataset.dbf``).
+
+**Improvements**
+
+- New import/export interface (``Dataset.export()``, ``Dataset.load()``).
+- CSV custom delimiter support (``Dataset.export('csv', delimiter='$')``).
+- Adding ability to remove duplicates to all rows in a dataset (``Dataset.remove_duplicates()``).
+- Added a mechanism to avoid ``datetime.datetime`` issues when serializing data.
+- New ``detect_format()`` function (mostly for internal use).
+- Update the vendored unicodecsv to fix ``None`` handling.
+- Only freeze the headers row, not the headers columns (xls).
+
+**Breaking Changes**
+
+- ``detect()`` function removed.
+
+**Bugfixes**
+
+- Fix XLSX import.
+- Bugfix for ``Dataset.transpose().transpose()``.
+
+
+0.10.0 (2014-05-27)
++++++++++++++++++++
+
+* Unicode Column Headers
+* ALL the bugfixes!
+
+0.9.11 (2011-06-30)
++++++++++++++++++++
+
+* Bugfixes
+
+0.9.10 (2011-06-22)
++++++++++++++++++++
+
+* Bugfixes
+
+0.9.9 (2011-06-21)
+++++++++++++++++++
 
 * Dataset API Changes
 * ``stack_rows`` => ``stack``, ``stack_columns`` => ``stack_cols``
-* column operations have their own methods now (``apend_col``, ``insert_col``)
+* column operations have their own methods now (``append_col``, ``insert_col``)
 * List-style ``pop()``
 * Redis-style ``rpush``, ``lpush``, ``rpop``, ``lpop``, ``rpush_col``, and ``lpush_col``
 

LICENSE

@@ -1,4 +1,4 @@
-Copyright 2011 Kenneth Reitz
+Copyright 2016 Kenneth Reitz
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

MANIFEST.in

@@ -1 +1 @@
-include HISTORY.rst README.rst LICENSE AUTHORS 
+include HISTORY.rst README.rst LICENSE AUTHORS NOTICE test_tablib.py

Makefile

@@ -0,0 +1,6 @@
+test:
+	python test_tablib.py
+publish:
+	python setup.py register
+	python setup.py sdist upload
+	python setup.py bdist_wheel --universal upload

NOTICE

@@ -1,5 +1,5 @@
-Tablib includes some vendorized python libraries: ordereddict, pyyaml,
-simplejson, unicodecsv, and xlwt.
+Tablib includes some vendorized python libraries: ordereddict, odfpy, pyyaml,
+simplejson, unicodecsv, xlrd, xlrd3, xlwt, and xlwt3.
 
 Markup License
 ==============
@@ -7,6 +7,213 @@ Markup License
 Markup is in the public domain.
 
 
+Odfpy License
+=============
+
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
 
 OrderedDict License
 ===================
@@ -92,8 +299,8 @@ or implied, of Jeremy Dunck.
 
 
 
-XLWT License
-============
+XLWT (and XLWT3) License
+========================
 
 Portions copyright © 2007, Stephen John Machin, Lingfo Pty Ltd
 All rights reserved.
@@ -172,4 +379,81 @@ Libknecht St., 4
 +7(0712)56-09-83
 
 <roman@kiseliov.ru>
-Subject: pyExcelerator
+Subject: pyExcelerator
+
+
+
+
+XLRD (and XLRD3) License
+========================
+
+Portions copyright © 2005-2009, Stephen John Machin, Lingfo Pty Ltd
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice,
+this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+this list of conditions and the following disclaimer in the documentation
+and/or other materials provided with the distribution.
+
+3. None of the names of Stephen John Machin, Lingfo Pty Ltd and any
+contributors may be used to endorse or promote products derived from this
+software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS
+BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+THE POSSIBILITY OF SUCH DAMAGE.
+
+
+Copyright (c) 2001 David Giffin.
+All rights reserved.
+
+Based on the the Java version: Andrew Khan Copyright (c) 2000.
+
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in
+   the documentation and/or other materials provided with the
+   distribution.
+
+3. All advertising materials mentioning features or use of this
+   software must display the following acknowledgment:
+   "This product includes software developed by
+    David Giffin <david@giffin.org>."
+
+4. Redistributions of any form whatsoever must retain the following
+   acknowledgment:
+   "This product includes software developed by
+    David Giffin <david@giffin.org>."
+
+THIS SOFTWARE IS PROVIDED BY DAVID GIFFIN ``AS IS'' AND ANY
+EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL DAVID GIFFIN OR
+ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+OF THE POSSIBILITY OF SUCH DAMAGE.

README.rst

@@ -1,6 +1,9 @@
 Tablib: format-agnostic tabular dataset library
 ===============================================
 
+.. image:: https://travis-ci.org/kennethreitz/tablib.svg?branch=master
+    :target: https://travis-ci.org/kennethreitz/tablib
+
 ::
 
 	_____         ______  ___________ ______
@@ -20,7 +23,9 @@ Output formats supported:
 - YAML (Sets + Books)
 - HTML (Sets)
 - TSV (Sets)
+- OSD (Sets)
 - CSV (Sets)
+- DBF (Sets)
 
 Note that tablib *purposefully* excludes XML support. It always will. (Note: This is a joke. Pull requests are welcome.)
 
@@ -28,7 +33,7 @@ Overview
 --------
 
 `tablib.Dataset()`
-	A Dataset is a table of tabular data. It may or may not have a header row. They can be build and manipulated as raw Python datatypes (Lists of tuples|dictionaries). Datasets can be imported from JSON, YAML, and CSV; they can be exported to XLSX, XLS, ODS, JSON, YAML, CSV, TSV, and HTML.
+	A Dataset is a table of tabular data. It may or may not have a header row. They can be build and manipulated as raw Python datatypes (Lists of tuples|dictionaries). Datasets can be imported from JSON, YAML, DBF, and CSV; they can be exported to XLSX, XLS, ODS, JSON, YAML, DBF, CSV, TSV, and HTML.
 
 `tablib.Databook()`
 	A Databook is a set of Datasets. The most common form of a Databook is an Excel file with multiple spreadsheets. Databooks can be imported from JSON and YAML; they can be exported to XLSX, XLS, ODS, JSON, and YAML.
@@ -55,7 +60,7 @@ Intelligently add new rows: ::
 
 Intelligently add new columns: ::
 
-    >>> data.append(col=(90, 67, 83), header='age')
+    >>> data.append_col((90, 67, 83), header='age')
 
 Slice rows:  ::
 
@@ -117,7 +122,15 @@ EXCEL!
 ++++++
 ::
 
-	>>> open('people.xls', 'wb').write(data.xls)
+	>>> with open('people.xls', 'wb') as f:
+	...     f.write(data.xls)
+
+DBF!
+++++
+::
+
+    >>> with open('people.dbf', 'wb') as f:
+    ...     f.write(data.dbf)
 
 It's that easy.
 
@@ -129,9 +142,8 @@ To install tablib, simply: ::
 
 	$ pip install tablib
 
-Or, if you absolutely must: ::
+Make sure to check out `Tablib on PyPi <https://pypi.python.org/pypi/tablib/>`_!
 
-	$ easy_install tablib
 
 Contribute
 ----------

TODO.rst

@@ -1,7 +0,0 @@
-* Hooks System
-  - pre/post-append
-  - pre/post-import
-  - pre/post-export
-* Add Tablib.ext namespace
-* Width detection for XLS output
-* Documentation Improvements

docs/_templates/sidebarintro.html

@@ -1,4 +1,4 @@
-<h3><a href="http://tablib.org">About Tablib</a></h3>
+<h3><a href="http://docs.python-tablib.org">About Tablib</a></h3>
 <p>
   Tablib is an MIT Licensed format-agnostic tabular dataset library, written in Python. It allows you to import, export, and manipulate tabular data sets. Advanced features include, segregation, dynamic columns, tags & filtering, and seamless format import & export.
 </p>
@@ -13,7 +13,7 @@
 
 <h3>Useful Links</h3>
 <ul>
-  <li><a href="http://tablib.org/">The Tablib Website</a></li>
+  <li><a href="http://docs.python-tablib.org/">The Tablib Website</a></li>
   <li><a href="http://pypi.python.org/pypi/tablib">Tablib @ PyPI</a></li>
   <li><a href="http://github.com/kennethreitz/tablib">Tablib @ GitHub</a></li>
   <li><a href="http://github.com/kennethreitz/tablib/issues">Issue Tracker</a></li>

docs/_templates/sidebarlogo.html

@@ -1,4 +1,4 @@
-<h3><a href="http://tablib.org">About Tablib</a></h3>
+<h3><a href="http://docs.python-tablib.org/">About Tablib</a></h3>
 <p>
   Tablib is an MIT Licensed format-agnostic tabular dataset library, written in Python. It allows you to import, export, and manipulate tabular data sets. Advanced features include, segregation, dynamic columns, tags & filtering, and seamless format import & export.
 </p>

docs/_themes/kr/layout.html

@@ -13,8 +13,14 @@
       © Copyright {{ copyright }}.
     </div>
     <a href="https://github.com/kennethreitz/tablib">
-        <img style="position: absolute; top: 0; right: 0; border: 0;" src="http://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub" />
+        <img style="position: absolute; top: 0; right: 0; border: 0;" src="//s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub" />
     </a>
+
+    <script type="text/javascript" src="//www.hellobar.com/hellobar.js"></script>
+    <script type="text/javascript">
+        new HelloBar(36402,48802);
+    </script>
+
     <script type="text/javascript">
 
       var _gaq = _gaq || [];

docs/_themes/kr/static/flasky.css_t

@@ -462,3 +462,9 @@ a:hover tt {
     background-color: #ccc;
     -webkit-border-radius: 3px;
 }
+
+/* misc. */
+
+.revsys-inline {
+    display: none!important;
+}

docs/_themes/kr_small/layout.html

@@ -14,8 +14,8 @@
 {% block relbar1 %}{% endblock %}
 {% block relbar2 %}
   {% if theme_github_fork %}
-    <a href="http://github.com/{{ theme_github_fork }}"><img style="position: fixed; top: 0; right: 0; border: 0;"
-    src="http://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub" /></a>
+    <a href="https://github.com/{{ theme_github_fork }}"><img style="position: fixed; top: 0; right: 0; border: 0;"
+    src="//s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub" /></a>
   {% endif %}
 {% endblock %}
 {% block sidebar1 %}{% endblock %}

docs/conf.py

@@ -41,7 +41,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'Tablib'
-copyright = u'2011. A <a href="http://kennethreitz.com/pages/open-projects.html">Kenneth Reitz</a> Project'
+copyright = u'2016. A <a href="http://kennethreitz.org/">Kenneth Reitz</a> Project'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/development.rst

@@ -8,11 +8,6 @@ Tablib is under active development, and contributors are welcome.
 If you have a feature request, suggestion, or bug report, please open a new
 issue on GitHub_. To submit patches, please send a pull request on GitHub_.
 
-If you'd like to contribute, there's plenty to do. Here's a short todo list.
-
-    .. include:: ../TODO.rst
-
-
 .. _GitHub: http://github.com/kennethreitz/tablib/
 
 
@@ -46,7 +41,7 @@ Source Control
 Tablib source is controlled with Git_, the lean, mean, distributed source
 control machine.
 
-The repository is publicly accessable.
+The repository is publicly accessible.
 
     ``git clone git://github.com/kennethreitz/tablib.git``
 
@@ -66,12 +61,10 @@ Feature / Hotfix / Release branches follow a `Successful Git Branching Model`_ .
     The "next release" branch. Likely unstable.
 ``master``
     Current production release (|version|) on PyPi.
-``gh-pages``
-    Current release of http://tablib.org.
 
 Each release is tagged.
 
-When submitting patches, please place your feature/change in its own branch prior to opening a pull reqeust on GitHub_.
+When submitting patches, please place your feature/change in its own branch prior to opening a pull request on GitHub_.
 
 
 .. _Git: http://git-scm.org
@@ -87,9 +80,7 @@ Adding New Formats
 
 Tablib welcomes new format additions! Format suggestions include:
 
-* Tab Separated Values
 * MySQL Dump
-* HTML Table
 
 
 Coding by Convention
@@ -132,7 +123,7 @@ Tablib features a micro-framework for adding format support. The easiest way to
 
 2.
 
-    Add your new format module to the :class:`tablib.formats.avalable` tuple.
+    Add your new format module to the :class:`tablib.formats.available` tuple.
 
 3.
     Add a mock property to the :class:`Dataset <tablib.Dataset>` class with verbose `reStructured Text`_ docstring. This alleviates IDE confusion, and allows for pretty auto-generated Sphinx_ documentation.
@@ -174,20 +165,15 @@ This will generate a **nosetests.xml** file, which can then be analyzed.
 Continuous Integration
 ----------------------
 
-Every commit made to the **develop** branch is automatically tested and inspected upon receipt with `Jenkins CI`_. If you have access to the main repository and broke the build, you will receive an email accordingly.
+Every commit made to the **develop** branch is automatically tested and inspected upon receipt with `Travis CI`_. If you have access to the main repository and broke the build, you will receive an email accordingly.
 
 Anyone may view the build status and history at any time.
 
-    http://ci.kennethreitz.com/
-
-
-If you are trustworthy and plan to contribute to tablib on a regular basis, please contact `Kenneth Reitz`_ to get an account on the Jenkins Server.
-
+    https://travis-ci.org/kennethreitz/tablib
 
 Additional reports will also be included here in the future, including :pep:`8` checks and stress reports for extremely large datasets.
 
-.. _`Jenkins CI`: http://jenkins-ci.org/
-.. _`Kenneth Reitz`: http://kennethreitz.com/contact-me/
+.. _`Jenkins CI`: https://travis-ci.org/
 
 
 .. _docs:
@@ -210,37 +196,12 @@ Then, to build an HTML version of the docs, simply run the following from the **
 
 Your ``docs/_build/html`` directory will then contain an HTML representation of the documentation, ready for publication on most web servers.
 
-You can also generate the documentation in **ebpub**, **latex**, **json**, *&c* similarly.
-
-.. admonition:: GitHub Pages
-
-	To push the documentation up to `GitHub Pages`_, you will first need to run `sphinx-to-github`_ against your ``docs/_build/html`` directory.
-
-	GitHub Pages are powered by an HTML generation system called Jeckyl_, which is configured to ignore files and folders that begin with "``_``" (*ie.* **_static**).
-
-
-
-
-
-
-	 and `sphinx-to-github`_. ::
-
-	Installing sphinx-to-github is simple. ::
-
-		$ pip install sphinx-to-github
-
-	Running it against the docs is even simpler. ::
-
-		$ sphinx-to-github _build/html
-
-	Move the resulting files to the **gh-pages** branch of your repository, and push it up to GitHub.
+You can also generate the documentation in **epub**, **latex**, **json**, *&c* similarly.
 
 .. _`reStructured Text`: http://docutils.sourceforge.net/rst.html
 .. _Sphinx: http://sphinx.pocoo.org
 .. _`GitHub Pages`: http://pages.github.com
-.. _Jeckyl: http://github.com/mojombo/jekyll
-.. _`sphinx-to-github`: http://github.com/michaeljones/sphinx-to-github
 
 ----------
 
-Make sure to check out the :ref:`API Documentation <api>`.
+Make sure to check out the :ref:`API Documentation <api>`.

docs/index.rst

@@ -29,10 +29,10 @@ Tablib is an :ref:`MIT Licensed <mit>` format-agnostic tabular dataset library,
    >>> data = tablib.Dataset(headers=['First Name', 'Last Name', 'Age'])
    >>> map(data.append, [('Kenneth', 'Reitz', 22), ('Bessie', 'Monke', 21)])
 
-   >>> data.json
+   >>> print data.json
    [{"Last Name": "Reitz", "First Name": "Kenneth", "Age": 22}, {"Last Name": "Monke", "First Name": "Bessie", "Age": 21}]
 
-   >>> data.yaml
+   >>> print data.yaml
    - {Age: 22, First Name: Kenneth, Last Name: Reitz}
    - {Age: 21, First Name: Bessie, Last Name: Monke}
 
@@ -55,7 +55,7 @@ and `The Sunlight Foundation <http://sunlightfoundation.com/>`_ use Tablib inter
    Tablib by @kennethreitz saved my life. I had to consolidate like 5 huge poorly maintained lists of domains and data. It was a breeze!
 
 **Dave Coutts**
-   It's turning into one of my most used modules of 2010. You really hit a sweat spot for managing tabular data with a minimal amount of code and effort.
+   It's turning into one of my most used modules of 2010. You really hit a sweet spot for managing tabular data with a minimal amount of code and effort.
 
 **Joshua Ourisman**
    Tablib has made it so much easier to deal with the inevitable 'I want an Excel file!' requests from clients...
@@ -99,4 +99,4 @@ method, this part of the documentation is for you.
 .. toctree::
    :maxdepth: 2
 
-   api
+   api

docs/install.rst

@@ -2,7 +2,7 @@
 Installation
 ============
 
-This part of the documentation covers the installation of Tablib. The first step to using any software package is getting it properly installed. Please read this section carefully, or you may miss out on some nice  :ref:`speed enhancements <peed-extentions>`.
+This part of the documentation covers the installation of Tablib. The first step to using any software package is getting it properly installed. Please read this section carefully, or you may miss out on some nice  :ref:`speed enhancements <peed-extensions>`.
 
 
 .. _installing:
@@ -14,27 +14,10 @@ Installing Tablib
 Distribute & Pip
 ----------------
 
-Installing Tablib is simple with `pip <http://www.pip-installer.org/>`_::
+Of course, the recommended way to install Tablib is with `pip <http://www.pip-installer.org/>`_::
 
     $ pip install tablib
 
-or, with `easy_install <http://pypi.python.org/pypi/setuptools>`_::
-
-    $ easy_install tablib
-
-But, you really `shouldn't do that <http://www.pip-installer.org/en/latest/index.html#pip-compared-to-easy-install>`_.
-
-
-
-Cheeseshop Mirror
------------------
-
-If the Cheeseshop is down, you can also install Requests from Kenneth Reitz's personal `Cheeseshop mirror <pip.kreitz.co/>`_::
-
-    $ pip install -i http://pip.kreitz.co/simple tablib
-
-
-
 
 -------------------
 Download the Source
@@ -57,15 +40,15 @@ To download the full source history from Git, see :ref:`Source Control <scm>`.
 .. _zipball: http://github.com/kennethreitz/tablib/zipball/master
 
 
-.. _speed-extentions:
-Speed Extentions
+.. _speed-extensions:
+Speed Extensions
 ----------------
 
 .. versionadded:: 0.8.5
 
 Tablib is partially dependent on the **pyyaml**, **simplejson**, and **xlwt** modules. To reduce installation issues, fully integrated versions of all required libraries are included in Tablib.
 
-However, if performance is important to you (and it should be), you can install  **pyyaml** with C extentions from PyPi. ::
+However, if performance is important to you (and it should be), you can install  **pyyaml** with C extensions from PyPi. ::
 
     $ pip install PyYAML
 
@@ -89,4 +72,4 @@ When a new version is available, upgrading is simple::
     $ pip install tablib --upgrade
 
 
-Now, go get a :ref:`Quick Start <quickstart>`.
+Now, go get a :ref:`Quick Start <quickstart>`.

docs/intro.rst

@@ -10,7 +10,7 @@ Advanced features include, segregation, dynamic columns, tags / filtering, and
 seamless format import/export.
 
 
-Philosphy
+Philosophy
 ---------
 
 Tablib was developed with a few :pep:`20` idioms in mind.
@@ -49,7 +49,7 @@ Tablib is released under terms of `The MIT License`_.
 Tablib License
 --------------
 
-Copyright 2011 Kenneth Reitz
+Copyright 2016 Kenneth Reitz
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -90,4 +90,4 @@ Support for other Pythons will be rolled out soon.
 
 
 
-Now, go :ref:`Install Tablib <install>`.
+Now, go :ref:`Install Tablib <install>`.

docs/tutorial.rst

@@ -39,6 +39,7 @@ You can now start filling this :class:`Dataset <tablib.Dataset>` object with dat
 
 
 
+
 -----------
 Adding Rows
 -----------
@@ -68,7 +69,7 @@ Adding Headers
 --------------
 
 
-It's time enhance our :class:`Dataset` by giving our columns some titles. To do so, set :class:`Dataset.headers`. ::
+It's time to enhance our :class:`Dataset` by giving our columns some titles. To do so, set :class:`Dataset.headers`. ::
 
     data.headers = ['First Name', 'Last Name']
 
@@ -97,6 +98,15 @@ Let's view the data now. ::
 It's that easy.
 
 
+--------------
+Importing Data
+--------------
+Creating a :class:`tablib.Dataset` object by importing a pre-existing file is simple. ::
+
+   imported_data = Dataset().load(open('data.csv').read())
+
+This detects what sort of data is being passed in, and uses an appropriate formatter to do the import. So you can import from a variety of different file types.
+
 --------------
 Exporting Data
 --------------
@@ -146,6 +156,13 @@ To do so, we access the :class:`Dataset` as if it were a standard Python diction
     >>> data['First Name']
     ['Kenneth', 'Bessie']
 
+You can also access the column using its index. ::
+
+    >>> data.headers
+    ['Last Name', 'First Name', 'Age']
+    >>> data.get_col(1)
+    ['Kenneth', 'Bessie']
+
 Let's find the average age. ::
 
     >>> ages = data['Age']
@@ -244,7 +261,7 @@ Filtering Datasets with Tags
 
 
 When constructing a :class:`Dataset` object, you can add tags to rows by specifying the ``tags`` parameter.
-This allows you to filter your :class:`Dataset` later. This can be useful so separate rows of data based on
+This allows you to filter your :class:`Dataset` later. This can be useful to separate rows of data based on
 arbitrary criteria (*e.g.* origin) that you don't want to include in your :class:`Dataset`.
 
 Let's tag some students. ::
@@ -256,7 +273,7 @@ Let's tag some students. ::
     students.rpush(['Kenneth', 'Reitz'], tags=['male', 'technical'])
     students.rpush(['Bessie', 'Monke'], tags=['female', 'creative'])
 
-Now that we have extra meta-data on our rows, we can use easily filter our :class:`Dataset`. Let's just see Male students. ::
+Now that we have extra meta-data on our rows, we can easily filter our :class:`Dataset`. Let's just see Male students. ::
 
 
     >>> students.filter(['male']).yaml

setup.py

@@ -7,9 +7,9 @@ import sys
 import tablib
 
 try:
-    from setuptools import setup, find_packages
+    from setuptools import setup
 except ImportError:
-    from distutils.core import setup, find_packages
+    from distutils.core import setup
 
 
 if sys.argv[-1] == 'publish':
@@ -33,8 +33,34 @@ if sys.argv[-1] == 'test':
         print('py.test required.')
         sys.exit(1)
 
-    os.system('pytest test_tablib.py')
-    sys.exit()
+    errors = os.system('py.test test_tablib.py')
+    sys.exit(bool(errors))
+
+packages = [
+    'tablib', 'tablib.formats',
+    'tablib.packages',
+    'tablib.packages.omnijson',
+    'tablib.packages.unicodecsv',
+    'tablib.packages.xlwt',
+    'tablib.packages.xlrd',
+    'tablib.packages.odf',
+    'tablib.packages.openpyxl',
+    'tablib.packages.openpyxl.shared',
+    'tablib.packages.openpyxl.reader',
+    'tablib.packages.openpyxl.writer',
+    'tablib.packages.yaml',
+    'tablib.packages.dbfpy',
+    'tablib.packages.xlwt3',
+    'tablib.packages.xlrd3',
+    'tablib.packages.odf3',
+    'tablib.packages.openpyxl3',
+    'tablib.packages.openpyxl3.shared',
+    'tablib.packages.openpyxl3.reader',
+    'tablib.packages.openpyxl3.writer',
+    'tablib.packages.yaml3',
+    'tablib.packages.dbfpy3'
+]
+
 
 setup(
     name='tablib',
@@ -43,11 +69,11 @@ setup(
     long_description=(open('README.rst').read() + '\n\n' +
         open('HISTORY.rst').read()),
     author='Kenneth Reitz',
-    author_email='me@kennethreitz.com',
-    url='http://tablib.org',
-    packages=find_packages(exclude=('docs',)),
+    author_email='me@kennethreitz.org',
+    url='http://python-tablib.org',
+    packages=packages,
     license='MIT',
-    classifiers=(
+    classifiers=[
         'Development Status :: 5 - Production/Stable',
         'Intended Audience :: Developers',
         'Natural Language :: English',
@@ -59,5 +85,10 @@ setup(
         'Programming Language :: Python :: 3.0',
         'Programming Language :: Python :: 3.1',
         'Programming Language :: Python :: 3.2',
-    ),
+        'Programming Language :: Python :: 3.3',
+        'Programming Language :: Python :: 3.4',
+        'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
+    ],
+    tests_require=['pytest'],
 )

tablib/__init__.py

@@ -1,8 +1,7 @@
-""" Tablib.
-"""
+""" Tablib. """
 
 from tablib.core import (
-    Databook, Dataset, detect, import_set,
+    Databook, Dataset, detect_format, import_set, import_book,
     InvalidDatasetType, InvalidDimensions, UnsupportedFormat,
     __version__
 )

tablib/compat.py

@@ -23,27 +23,36 @@ except ImportError:
 if is_py3:
     from io import BytesIO
     import tablib.packages.xlwt3 as xlwt
+    import tablib.packages.xlrd3 as xlrd
+    from tablib.packages.xlrd3.biffh import XLRDError
     from tablib.packages import markup3 as markup
     from tablib.packages import openpyxl3 as openpyxl
     from tablib.packages.odf3 import opendocument, style, text, table
+    import tablib.packages.dbfpy3 as dbfpy
 
     import csv
     from io import StringIO
     # py3 mappings
 
+    ifilter = filter
     unicode = str
     bytes = bytes
     basestring = str
+    xrange = range
 
 else:
     from cStringIO import StringIO as BytesIO
     from cStringIO import StringIO
     import tablib.packages.xlwt as xlwt
+    import tablib.packages.xlrd as xlrd
+    from tablib.packages.xlrd.biffh import XLRDError
     from tablib.packages import markup
     from itertools import ifilter
     from tablib.packages import openpyxl
     from tablib.packages.odf import opendocument, style, text, table
 
     from tablib.packages import unicodecsv as csv
+    import tablib.packages.dbfpy as dbfpy
 
-    unicode = unicode
+    unicode = unicode
+    xrange = xrange

tablib/core.py

@@ -5,7 +5,7 @@
 
     This module implements the central Tablib objects.
 
-    :copyright: (c) 2011 by Kenneth Reitz.
+    :copyright: (c) 2016 by Kenneth Reitz.
     :license: MIT, see LICENSE for more details.
 """
 
@@ -14,22 +14,22 @@ from operator import itemgetter
 
 from tablib import formats
 
-from tablib.compat import OrderedDict
+from tablib.compat import OrderedDict, unicode
 
 
 __title__ = 'tablib'
-__version__ = '0.9.9'
-__build__ = 0x000909
+__version__ = '0.11.4'
+__build__ = 0x001104
 __author__ = 'Kenneth Reitz'
 __license__ = 'MIT'
-__copyright__ = 'Copyright 2011 Kenneth Reitz'
+__copyright__ = 'Copyright 2016 Kenneth Reitz'
 __docformat__ = 'restructuredtext'
 
 
 class Row(object):
     """Internal Row object. Mainly used for filtering."""
 
-    __slots__ = ['tuple', '_row', 'tags']
+    __slots__ = ['_row', 'tags']
 
     def __init__(self, row=list(), tags=list()):
         self._row = list(row)
@@ -45,7 +45,7 @@ class Row(object):
         return repr(self._row)
 
     def __getslice__(self, i, j):
-        return self._row[i,j]
+        return self._row[i:j]
 
     def __getitem__(self, i):
         return self._row[i]
@@ -57,7 +57,14 @@ class Row(object):
         del self._row[i]
 
     def __getstate__(self):
-        return {'slot': [getattr(self, slot) for slot in self.__slots__]}
+
+        slots = dict()
+
+        for slot in self.__slots__:
+            attribute = getattr(self, slot)
+            slots[slot] = attribute
+
+        return slots
 
     def __setstate__(self, state):
         for (k, v) in list(state.items()): setattr(self, k, v)
@@ -105,7 +112,7 @@ class Dataset(object):
     functionality.
 
     Usually you create a :class:`Dataset` instance in your main module, and append
-    rows and columns as you collect data. ::
+    rows as you collect data. ::
 
         data = tablib.Dataset()
         data.headers = ('name', 'age')
@@ -113,17 +120,30 @@ class Dataset(object):
         for (name, age) in some_collector():
             data.append((name, age))
 
-    You can also set rows and headers upon instantiation. This is useful if dealing
-    with dozens or hundres of :class:`Dataset` objects. ::
+
+    Setting columns is similar. The column data length must equal the
+    current height of the data and headers must be set ::
+
+        data = tablib.Dataset()
+        data.headers = ('first_name', 'last_name')
+
+        data.append(('John', 'Adams'))
+        data.append(('George', 'Washington'))
+
+        data.append_col((90, 67), header='age')
+
+
+    You can also set rows and headers upon instantiation. This is useful if
+    dealing with dozens or hundreds of :class:`Dataset` objects. ::
 
         headers = ('first_name', 'last_name')
         data = [('John', 'Adams'), ('George', 'Washington')]
 
         data = tablib.Dataset(*data, headers=headers)
 
-
     :param \*args: (optional) list of rows to populate Dataset
     :param headers: (optional) list strings for Dataset header row
+    :param title: (optional) string to use as title of the Dataset
 
 
     .. admonition:: Format Attributes Definition
@@ -134,6 +154,8 @@ class Dataset(object):
 
     """
 
+    _formats = {}
+
     def __init__(self, *args, **kwargs):
         self._data = list(Row(arg) for arg in args)
         self.__headers = None
@@ -144,15 +166,9 @@ class Dataset(object):
         # (column, callback) tuples
         self._formatters = []
 
-        try:
-            self.headers = kwargs['headers']
-        except KeyError:
-            self.headers = None
+        self.headers = kwargs.get('headers')
 
-        try:
-            self.title = kwargs['title']
-        except KeyError:
-            self.title = None
+        self.title = kwargs.get('title')
 
         self._register_formats()
 
@@ -162,7 +178,7 @@ class Dataset(object):
 
 
     def __getitem__(self, key):
-        if isinstance(key, str):
+        if isinstance(key, str) or isinstance(key, unicode):
             if key in self.headers:
                 pos = self.headers.index(key) # get 'key' index from each data
                 return [row[pos] for row in self._data]
@@ -175,14 +191,13 @@ class Dataset(object):
             else:
                 return [result.tuple for result in _results]
 
-
     def __setitem__(self, key, value):
         self._validate(value)
         self._data[key] = Row(value)
 
 
     def __delitem__(self, key):
-        if isinstance(key, str):
+        if isinstance(key, str) or isinstance(key, unicode):
 
             if key in self.headers:
 
@@ -205,6 +220,29 @@ class Dataset(object):
         except AttributeError:
             return '<dataset object>'
 
+    def __unicode__(self):
+        result = []
+
+        # Add unicode representation of headers.
+        if self.__headers:
+            result.append([unicode(h) for h in self.__headers])
+
+        # Add unicode representation of rows.
+        result.extend(list(map(unicode, row)) for row in self._data)
+
+        lens = [list(map(len, row)) for row in result]
+        field_lens = list(map(max, zip(*lens)))
+
+        # delimiter between header and data
+        if self.__headers:
+            result.insert(1, ['-' * length for length in field_lens])
+
+        format_string = '|'.join('{%s:%s}' % item for item in enumerate(field_lens))
+
+        return '\n'.join(format_string.format(*row) for row in result)
+
+    def __str__(self):
+        return self.__unicode__()
 
     # ---------
     # Internals
@@ -217,11 +255,16 @@ class Dataset(object):
             try:
                 try:
                     setattr(cls, fmt.title, property(fmt.export_set, fmt.import_set))
+                    setattr(cls, 'get_%s' % fmt.title, fmt.export_set)
+                    setattr(cls, 'set_%s' % fmt.title, fmt.import_set)
+                    cls._formats[fmt.title] = (fmt.export_set, fmt.import_set)
                 except AttributeError:
                     setattr(cls, fmt.title, property(fmt.export_set))
+                    setattr(cls, 'get_%s' % fmt.title, fmt.export_set)
+                    cls._formats[fmt.title] = (fmt.export_set, None)
 
             except AttributeError:
-                pass
+                cls._formats[fmt.title] = (None, None)
 
 
     def _validate(self, row=None, col=None, safety=False):
@@ -312,7 +355,7 @@ class Dataset(object):
         A dataset object can also be imported by setting the `Dataset.dict` attribute: ::
 
             data = tablib.Dataset()
-            data.json = '[{"last_name": "Adams","age": 90,"first_name": "John"}]'
+            data.dict = [{'age': 90, 'first_name': 'Kenneth', 'last_name': 'Reitz'}]
 
         """
         return self._package()
@@ -392,11 +435,41 @@ class Dataset(object):
                 return 0
 
 
+    def load(self, in_stream, format=None, **kwargs):
+        """
+        Import `in_stream` to the :class:`Dataset` object using the `format`.
+
+        :param \*\*kwargs: (optional) custom configuration to the format `import_set`.
+        """
+
+        if not format:
+            format = detect_format(in_stream)
+
+        export_set, import_set = self._formats.get(format, (None, None))
+        if not import_set:
+            raise UnsupportedFormat('Format {0} cannot be imported.'.format(format))
+
+        import_set(self, in_stream, **kwargs)
+        return self
+
+
+
+    def export(self, format, **kwargs):
+        """
+        Export :class:`Dataset` object to `format`.
+
+        :param \*\*kwargs: (optional) custom configuration to the format `export_set`.
+        """
+        export_set, import_set = self._formats.get(format, (None, None))
+        if not export_set:
+            raise UnsupportedFormat('Format {0} cannot be exported.'.format(format))
+
+        return export_set(self, **kwargs)
+
     # -------
     # Formats
     # -------
 
-
     @property
     def xls():
         """A Legacy Excel Spreadsheet representation of the :class:`Dataset` object, with :ref:`separators`. Cannot be set.
@@ -410,7 +483,7 @@ class Dataset(object):
              :class:`Dataset.xls` contains binary data, so make sure to write in binary mode::
 
                 with open('output.xls', 'wb') as f:
-                    f.write(data.xls)'
+                    f.write(data.xls)
         """
         pass
 
@@ -423,7 +496,7 @@ class Dataset(object):
              :class:`Dataset.xlsx` contains binary data, so make sure to write in binary mode::
 
                 with open('output.xlsx', 'wb') as f:
-                    f.write(data.xlsx)'
+                    f.write(data.xlsx)
         """
         pass
 
@@ -433,10 +506,10 @@ class Dataset(object):
 
          .. admonition:: Binary Warning
 
-             :class:`Dataset.xlsx` contains binary data, so make sure to write in binary mode::
+             :class:`Dataset.ods` contains binary data, so make sure to write in binary mode::
 
                 with open('output.ods', 'wb') as f:
-                    f.write(data.ods)'
+                    f.write(data.ods)
         """
         pass
 
@@ -452,6 +525,17 @@ class Dataset(object):
             data.csv = 'age, first_name, last_name\\n90, John, Adams'
 
         Import assumes (for now) that headers exist.
+
+        .. admonition:: Binary Warning
+
+             :class:`Dataset.csv` uses \\r\\n line endings by default, so make
+             sure to write in binary mode::
+
+                 with open('output.csv', 'wb') as f:
+                     f.write(data.csv)
+
+             If you do not do this, and you export the file on Windows, your
+             CSV file will open in Excel with a blank line between each row.
         """
         pass
 
@@ -477,7 +561,7 @@ class Dataset(object):
         set, a YAML list of objects will be returned. If no headers have
         been set, a YAML list of lists (rows) will be returned instead.
 
-        A dataset object can also be imported by setting the :class:`Dataset.json` attribute: ::
+        A dataset object can also be imported by setting the :class:`Dataset.yaml` attribute: ::
 
             data = tablib.Dataset()
             data.yaml = '- {age: 90, first_name: John, last_name: Adams}'
@@ -496,10 +580,11 @@ class Dataset(object):
         A dataset object can also be imported by setting the :class:`Dataset.json` attribute: ::
 
             data = tablib.Dataset()
-            data.json = '[{age: 90, first_name: "John", liast_name: "Adams"}]'
+            data.json = '[{"age": 90, "first_name": "John", "last_name": "Adams"}]'
 
         Import assumes (for now) that headers exist.
         """
+        pass
 
     @property
     def html():
@@ -510,6 +595,40 @@ class Dataset(object):
         """
         pass
 
+    @property
+    def dbf():
+        """A dBASE representation of the :class:`Dataset` object.
+
+        A dataset object can also be imported by setting the
+        :class:`Dataset.dbf` attribute. ::
+
+            # To import data from an existing DBF file:
+            data = tablib.Dataset()
+            data.dbf = open('existing_table.dbf').read()
+
+            # to import data from an ASCII-encoded bytestring:
+            data = tablib.Dataset()
+            data.dbf = '<bytestring of tabular data>'
+
+        .. admonition:: Binary Warning
+
+            :class:`Dataset.dbf` contains binary data, so make sure to write in binary mode::
+
+                with open('output.dbf', 'wb') as f:
+                    f.write(data.dbf)
+        """
+        pass
+
+
+    @property
+    def latex():
+        """A LaTeX booktabs representation of the :class:`Dataset` object. If a
+        title has been set, it will be exported as the table caption.
+
+        .. note:: This method can be used for export only.
+        """
+        pass
+
 
     # ----
     # Rows
@@ -518,30 +637,11 @@ class Dataset(object):
     def insert(self, index, row, tags=list()):
         """Inserts a row to the :class:`Dataset` at the given index.
 
-        Rows and columns inserted must be the correct size (height or width).
+        Rows inserted must be the correct size (height or width).
 
         The default behaviour is to insert the given row to the :class:`Dataset`
-        object at the given index. If the ``col`` parameter is given, however,
-        a new column will be insert to the :class:`Dataset` object instead.
-
-        You can also insert a column of a single callable object, which will
-        add a new column with the return values of the callable each as an
-        item in the column. ::
-
-            data.append(col=random.randint)
-
-        See :ref:`dyncols` for an in-depth example.
-
-        .. versionchanged:: 0.9.0
-           If inserting a column, and :class:`Dataset.headers` is set, the
-           header attribute must be set, and will be considered the header for
-           that row.
-
-        .. versionadded:: 0.9.0
-           If inserting a row, you can add :ref:`tags <tags>` to the row you are inserting.
-           This gives you the ability to :class:`filter <Dataset.filter>` your
-           :class:`Dataset` later.
-        """
+        object at the given index.
+       """
 
         self._validate(row)
         self._data.insert(index, Row(row, tags=tags))
@@ -570,6 +670,14 @@ class Dataset(object):
 
         self.rpush(row, tags)
 
+    def extend(self, rows, tags=list()):
+        """Adds a list of rows to the :class:`Dataset` using
+        :class:`Dataset.append`
+        """
+
+        for row in rows:
+            self.append(row, tags)
+
 
     def lpop(self):
         """Removes and returns the first row of the :class:`Dataset`."""
@@ -615,13 +723,25 @@ class Dataset(object):
         that row.
 
         See :ref:`dyncols` for an in-depth example.
+
+        .. versionchanged:: 0.9.0
+           If inserting a column, and :class:`Dataset.headers` is set, the
+           header attribute must be set, and will be considered the header for
+           that row.
+
+        .. versionadded:: 0.9.0
+           If inserting a row, you can add :ref:`tags <tags>` to the row you are inserting.
+           This gives you the ability to :class:`filter <Dataset.filter>` your
+           :class:`Dataset` later.
+
         """
 
-        col = list(col)
+        if col is None:
+            col = []
 
         # Callable Columns...
-        if len(col) == 1 and hasattr(col[0], '__call__'):
-            col = list(map(col[0], self._data))
+        if hasattr(col, '__call__'):
+            col = list(map(col, self._data))
 
         col = self._clean_col(col)
         self._validate(col=col)
@@ -630,8 +750,14 @@ class Dataset(object):
             # pop the first item off, add to headers
             if not header:
                 raise HeadersNeeded()
+
+            # corner case - if header is set without data
+            elif header and self.height == 0 and len(col):
+                raise InvalidDimensions
+
             self.headers.insert(index, header)
 
+
         if self.height and self.width:
 
             for i, row in enumerate(self._data):
@@ -686,6 +812,12 @@ class Dataset(object):
         self.rpush_col(col, header)
 
 
+    def get_col(self, index):
+        """Returns the column from the :class:`Dataset` at the given index."""
+
+        return [row[index] for row in self._data]
+
+
     # ----
     # Misc
     # ----
@@ -732,13 +864,13 @@ class Dataset(object):
         sorted.
         """
 
-        if isinstance(col, str):
+        if isinstance(col, str) or isinstance(col, unicode):
 
             if not self.headers:
                 raise HeadersNeeded
 
             _sorted = sorted(self.dict, key=itemgetter(col), reverse=reverse)
-            _dset = Dataset(headers=self.headers)
+            _dset = Dataset(headers=self.headers, title=self.title)
 
             for item in _sorted:
                 row = [item[key] for key in self.headers]
@@ -749,7 +881,7 @@ class Dataset(object):
                 col = self.headers[col]
 
             _sorted = sorted(self.dict, key=itemgetter(col), reverse=reverse)
-            _dset = Dataset(headers=self.headers)
+            _dset = Dataset(headers=self.headers, title=self.title)
 
             for item in _sorted:
                 if self.headers:
@@ -777,17 +909,17 @@ class Dataset(object):
         new_headers = [self.headers[0]] + self[self.headers[0]]
 
         _dset.headers = new_headers
-        for column in self.headers:
+        for index, column in enumerate(self.headers):
 
             if column == self.headers[0]:
                 # It's in the headers, so skip it
                 continue
 
             # Adding the column name as now they're a regular column
-            row_data = [column] + self[column]
+            # Use `get_col(index)` in case there are repeated values
+            row_data = [column] + self.get_col(index)
             row_data = Row(row_data)
             _dset.append(row=row_data)
-
         return _dset
 
 
@@ -848,17 +980,66 @@ class Dataset(object):
         return _dset
 
 
+    def remove_duplicates(self):
+        """Removes all duplicate rows from the :class:`Dataset` object
+        while maintaining the original order."""
+        seen = set()
+        self._data[:] = [row for row in self._data if not (tuple(row) in seen or seen.add(tuple(row)))]
+
+
     def wipe(self):
         """Removes all content and headers from the :class:`Dataset` object."""
         self._data = list()
         self.__headers = None
 
 
+    def subset(self, rows=None, cols=None):
+        """Returns a new instance of the :class:`Dataset`,
+        including only specified rows and columns.
+        """
+
+        # Don't return if no data
+        if not self:
+            return
+
+        if rows is None:
+            rows = list(range(self.height))
+
+        if cols is None:
+            cols = list(self.headers)
+
+        #filter out impossible rows and columns
+        rows = [row for row in rows if row in range(self.height)]
+        cols = [header for header in cols if header in self.headers]
+
+        _dset = Dataset()
+
+        #filtering rows and columns
+        _dset.headers = list(cols)
+
+        _dset._data = []
+        for row_no, row in enumerate(self._data):
+            data_row = []
+            for key in _dset.headers:
+                if key in self.headers:
+                    pos = self.headers.index(key)
+                    data_row.append(row[pos])
+                else:
+                    raise KeyError
+
+            if row_no in rows:
+                _dset.append(row=Row(data_row))
+
+        return _dset
+
+
 
 class Databook(object):
     """A book of :class:`Dataset` objects.
     """
 
+    _formats = {}
+
     def __init__(self, sets=None):
 
         if sets is None:
@@ -874,7 +1055,6 @@ class Databook(object):
         except AttributeError:
             return '<databook object>'
 
-
     def wipe(self):
         """Removes all :class:`Dataset` objects from the :class:`Databook`."""
         self._datasets = []
@@ -887,16 +1067,20 @@ class Databook(object):
             try:
                 try:
                     setattr(cls, fmt.title, property(fmt.export_book, fmt.import_book))
+                    cls._formats[fmt.title] = (fmt.export_book, fmt.import_book)
                 except AttributeError:
                     setattr(cls, fmt.title, property(fmt.export_book))
+                    cls._formats[fmt.title] = (fmt.export_book, None)
 
             except AttributeError:
-                pass
+                cls._formats[fmt.title] = (None, None)
 
+    def sheets(self):
+        return self._datasets
 
     def add_sheet(self, dataset):
         """Adds given :class:`Dataset` to the :class:`Databook`."""
-        if type(dataset) is Dataset:
+        if isinstance(dataset, Dataset):
             self._datasets.append(dataset)
         else:
             raise InvalidDatasetType
@@ -924,29 +1108,55 @@ class Databook(object):
         """The number of the :class:`Dataset` objects within :class:`Databook`."""
         return len(self._datasets)
 
+    def load(self, format, in_stream, **kwargs):
+        """
+        Import `in_stream` to the :class:`Databook` object using the `format`.
 
-def detect(stream):
-    """Return (format, stream) of given stream."""
+        :param \*\*kwargs: (optional) custom configuration to the format `import_book`.
+        """
+
+        if not format:
+            format = detect_format(in_stream)
+
+        export_book, import_book = self._formats.get(format, (None, None))
+        if not import_book:
+            raise UnsupportedFormat('Format {0} cannot be loaded.'.format(format))
+
+        import_book(self, in_stream, **kwargs)
+        return self
+
+    def export(self, format, **kwargs):
+        """
+        Export :class:`Databook` object to `format`.
+
+        :param \*\*kwargs: (optional) custom configuration to the format `export_book`.
+        """
+        export_book, import_book = self._formats.get(format, (None, None))
+        if not export_book:
+            raise UnsupportedFormat('Format {0} cannot be exported.'.format(format))
+
+        return export_book(self, **kwargs)
+
+
+def detect_format(stream):
+    """Return format name of given stream."""
     for fmt in formats.available:
         try:
             if fmt.detect(stream):
-                return (fmt, stream)
+                return fmt.title
         except AttributeError:
             pass
-    return (None, stream)
 
-
-def import_set(stream):
+def import_set(stream, format=None, **kwargs):
     """Return dataset of given stream."""
-    (format, stream) = detect(stream)
 
-    try:
-        data = Dataset()
-        format.import_set(data, stream)
-        return data
+    return Dataset().load(stream, format, **kwargs)
 
-    except AttributeError:
-        return None
+
+def import_book(stream, format=None, **kwargs):
+    """Return dataset of given stream."""
+
+    return Databook().load(stream, format, **kwargs)
 
 
 class InvalidDatasetType(Exception):

tablib/formats/__init__.py

@@ -11,5 +11,7 @@ from . import _tsv as tsv
 from . import _html as html
 from . import _xlsx as xlsx
 from . import _ods as ods
+from . import _dbf as dbf
+from . import _latex as latex
 
-available = (json, xls, yaml, csv, tsv, html, xlsx, ods)
+available = (json, xls, yaml, csv, dbf, tsv, html, latex, xlsx, ods)

tablib/formats/_csv.py

@@ -1,27 +1,28 @@
 # -*- coding: utf-8 -*-
 
-""" Tablib - CSV Support.
+""" Tablib - *SV Support.
 """
 
 from tablib.compat import is_py3, csv, StringIO
 
 
 title = 'csv'
-extentions = ('csv',)
+extensions = ('csv',)
 
 
 DEFAULT_ENCODING = 'utf-8'
+DEFAULT_DELIMITER = ','
 
 
-
-def export_set(dataset):
+def export_set(dataset, **kwargs):
     """Returns CSV representation of Dataset."""
     stream = StringIO()
 
-    if is_py3:
-        _csv = csv.writer(stream)
-    else:
-        _csv = csv.writer(stream, encoding=DEFAULT_ENCODING)
+    kwargs.setdefault('delimiter', DEFAULT_DELIMITER)
+    if not is_py3:
+        kwargs.setdefault('encoding', DEFAULT_ENCODING)
+
+    _csv = csv.writer(stream, **kwargs)
 
     for row in dataset._package(dicts=False):
         _csv.writerow(row)
@@ -29,15 +30,16 @@ def export_set(dataset):
     return stream.getvalue()
 
 
-def import_set(dset, in_stream, headers=True):
+def import_set(dset, in_stream, headers=True, **kwargs):
     """Returns dataset from CSV stream."""
 
     dset.wipe()
 
-    if is_py3:
-        rows = csv.reader(in_stream.splitlines())
-    else:
-        rows = csv.reader(in_stream.splitlines(), encoding=DEFAULT_ENCODING)
+    kwargs.setdefault('delimiter', DEFAULT_DELIMITER)
+    if not is_py3:
+        kwargs.setdefault('encoding', DEFAULT_ENCODING)
+
+    rows = csv.reader(StringIO(in_stream), **kwargs)
     for i, row in enumerate(rows):
 
         if (i == 0) and (headers):
@@ -46,10 +48,10 @@ def import_set(dset, in_stream, headers=True):
             dset.append(row)
 
 
-def detect(stream):
+def detect(stream, delimiter=DEFAULT_DELIMITER):
     """Returns True if given stream is valid CSV."""
     try:
-        csv.Sniffer().sniff(stream)
+        csv.Sniffer().sniff(stream, delimiters=delimiter)
         return True
-    except csv.Error:
-        return False
+    except (csv.Error, TypeError):
+        return False

tablib/formats/_dbf.py

@@ -0,0 +1,94 @@
+# -*- coding: utf-8 -*-
+
+""" Tablib - DBF Support.
+"""
+import tempfile
+import struct
+import os
+
+from tablib.compat import StringIO
+from tablib.compat import dbfpy
+from tablib.compat import is_py3
+
+if is_py3:
+    from tablib.packages.dbfpy3 import dbf
+    from tablib.packages.dbfpy3 import dbfnew
+    from tablib.packages.dbfpy3 import record as dbfrecord
+    import io
+else:
+    from tablib.packages.dbfpy import dbf
+    from tablib.packages.dbfpy import dbfnew
+    from tablib.packages.dbfpy import record as dbfrecord
+
+
+title = 'dbf'
+extensions = ('csv',)
+
+DEFAULT_ENCODING = 'utf-8'
+
+def export_set(dataset):
+    """Returns DBF representation of a Dataset"""
+    new_dbf = dbfnew.dbf_new()
+    temp_file, temp_uri = tempfile.mkstemp()
+
+    # create the appropriate fields based on the contents of the first row
+    first_row = dataset[0]
+    for fieldname, field_value in zip(dataset.headers, first_row):
+        if type(field_value) in [int, float]:
+            new_dbf.add_field(fieldname, 'N', 10, 8)
+        else:
+            new_dbf.add_field(fieldname, 'C', 80)
+
+    new_dbf.write(temp_uri)
+
+    dbf_file = dbf.Dbf(temp_uri, readOnly=0)
+    for row in dataset:
+        record = dbfrecord.DbfRecord(dbf_file)
+        for fieldname, field_value in zip(dataset.headers, row):
+            record[fieldname] = field_value
+        record.store()
+
+    dbf_file.close()
+    dbf_stream = open(temp_uri, 'rb')
+    if is_py3:
+        stream = io.BytesIO(dbf_stream.read())
+    else:
+        stream = StringIO(dbf_stream.read())
+    dbf_stream.close()
+    os.close(temp_file)
+    os.remove(temp_uri)
+    return stream.getvalue()
+
+def import_set(dset, in_stream, headers=True):
+    """Returns a dataset from a DBF stream."""
+
+    dset.wipe()
+    if is_py3:
+        _dbf = dbf.Dbf(io.BytesIO(in_stream))
+    else:
+        _dbf = dbf.Dbf(StringIO(in_stream))
+    dset.headers = _dbf.fieldNames
+    for record in range(_dbf.recordCount):
+        row = [_dbf[record][f] for f in _dbf.fieldNames]
+        dset.append(row)
+
+def detect(stream):
+    """Returns True if the given stream is valid DBF"""
+    #_dbf = dbf.Table(StringIO(stream))
+    try:
+        if is_py3:
+            if type(stream) is not bytes:
+                stream = bytes(stream, 'utf-8')
+            _dbf = dbf.Dbf(io.BytesIO(stream), readOnly=True)
+        else:
+            _dbf = dbf.Dbf(StringIO(stream), readOnly=True)
+        return True
+    except (ValueError, struct.error):
+        # When we try to open up a file that's not a DBF, dbfpy raises a
+        # ValueError.
+        # When unpacking a string argument with less than 8 chars, struct.error is
+        # raised.
+        return False
+
+
+

tablib/formats/_html.py

@@ -5,56 +5,66 @@
 
 import sys
 
-
 if sys.version_info[0] > 2:
-    from io import StringIO
+    from io import BytesIO as StringIO
     from tablib.packages import markup3 as markup
 else:
     from cStringIO import StringIO
     from tablib.packages import markup
 
 import tablib
+from tablib.compat import unicode
+import codecs
 
 BOOK_ENDINGS = 'h3'
 
 title = 'html'
-extentions = ('html', )
+extensions = ('html', )
 
 
 def export_set(dataset):
-	"""HTML representation of a Dataset."""
+    """HTML representation of a Dataset."""
 
-	stream = StringIO()
+    stream = StringIO()
 
-	page = markup.page()
-	page.table.open()
+    page = markup.page()
+    page.table.open()
 
-	if dataset.headers is not None:
-		page.thead.open()
-		headers = markup.oneliner.th(dataset.headers)
-		page.tr(headers)
-		page.thead.close()
+    if dataset.headers is not None:
+        new_header = [item if item is not None else '' for item in dataset.headers]
 
-	for row in dataset:
-		html_row = markup.oneliner.td(row)
-		page.tr(html_row)
+        page.thead.open()
+        headers = markup.oneliner.th(new_header)
+        page.tr(headers)
+        page.thead.close()
 
-	page.table.close()
+    for row in dataset:
+        new_row = [item if item is not None else '' for item in row]
 
-	stream.writelines(str(page))
+        html_row = markup.oneliner.td(new_row)
+        page.tr(html_row)
 
-	return stream.getvalue()
+    page.table.close()
+
+    # Allow unicode characters in output
+    wrapper = codecs.getwriter("utf8")(stream)
+    wrapper.writelines(unicode(page))
+
+    return stream.getvalue().decode('utf-8')
 
 
 def export_book(databook):
-	"""HTML representation of a Databook."""
+    """HTML representation of a Databook."""
 
-	stream = StringIO()
+    stream = StringIO()
 
-	for i, dset in enumerate(databook._datasets):
-		title = (dset.title if dset.title else 'Set %s' % (i))
-		stream.write('<%s>%s</%s>\n' % (BOOK_ENDINGS, title, BOOK_ENDINGS))
-		stream.write(dset.html)
-		stream.write('\n')
+    # Allow unicode characters in output
+    wrapper = codecs.getwriter("utf8")(stream)
 
-	return stream.getvalue()
+    for i, dset in enumerate(databook._datasets):
+        title = (dset.title if dset.title else 'Set %s' % (i))
+        wrapper.write('<%s>%s</%s>\n' % (BOOK_ENDINGS, title, BOOK_ENDINGS))
+        wrapper.write(dset.html)
+        wrapper.write('\n')
+
+    return stream.getvalue().decode('utf-8')

tablib/formats/_json.py

@@ -2,25 +2,38 @@
 
 """ Tablib - JSON Support
 """
+import decimal
 
 import tablib
 
-import sys
-from tablib.packages import omnijson as json
+try:
+    import json
+except ImportError:
+    from tablib.packages import omnijson as json
 
 
 title = 'json'
-extentions = ('json', 'jsn')
+extensions = ('json', 'jsn')
+
+
+def date_handler(obj):
+    if isinstance(obj, decimal.Decimal):
+        return str(obj)
+    elif hasattr(obj, 'isoformat'):
+        return obj.isoformat()
+    else:
+        return obj
+    # return obj.isoformat() if hasattr(obj, 'isoformat') else obj
 
 
 def export_set(dataset):
     """Returns JSON representation of Dataset."""
-    return json.dumps(dataset.dict)
+    return json.dumps(dataset.dict, default=date_handler)
 
 
 def export_book(databook):
     """Returns JSON representation of Databook."""
-    return json.dumps(databook._package())
+    return json.dumps(databook._package(), default=date_handler)
 
 
 def import_set(dset, in_stream):

tablib/formats/_latex.py

@@ -0,0 +1,134 @@
+# -*- coding: utf-8 -*-
+
+"""Tablib - LaTeX table export support.
+
+   Generates a LaTeX booktabs-style table from the dataset.
+"""
+import re
+
+from tablib.compat import unicode
+
+title = 'latex'
+extensions = ('tex',)
+
+TABLE_TEMPLATE = """\
+%% Note: add \\usepackage{booktabs} to your preamble
+%%
+\\begin{table}[!htbp]
+  \\centering
+  %(CAPTION)s
+  \\begin{tabular}{%(COLSPEC)s}
+    \\toprule
+%(HEADER)s
+    %(MIDRULE)s
+%(BODY)s
+    \\bottomrule
+  \\end{tabular}
+\\end{table}
+"""
+
+TEX_RESERVED_SYMBOLS_MAP = dict([
+    ('\\', '\\textbackslash{}'),
+    ('{', '\\{'),
+    ('}', '\\}'),
+    ('$', '\\$'),
+    ('&', '\\&'),
+    ('#', '\\#'),
+    ('^', '\\textasciicircum{}'),
+    ('_', '\\_'),
+    ('~', '\\textasciitilde{}'),
+    ('%', '\\%'),
+])
+
+TEX_RESERVED_SYMBOLS_RE = re.compile(
+    '(%s)' % '|'.join(map(re.escape, TEX_RESERVED_SYMBOLS_MAP.keys())))
+
+
+def export_set(dataset):
+    """Returns LaTeX representation of dataset
+
+    :param dataset: dataset to serialize
+    :type dataset: tablib.core.Dataset
+    """
+
+    caption = '\\caption{%s}' % dataset.title if dataset.title else '%'
+    colspec = _colspec(dataset.width)
+    header = _serialize_row(dataset.headers) if dataset.headers else ''
+    midrule = _midrule(dataset.width)
+    body = '\n'.join([_serialize_row(row) for row in dataset])
+    return TABLE_TEMPLATE % dict(CAPTION=caption, COLSPEC=colspec,
+                                 HEADER=header, MIDRULE=midrule, BODY=body)
+
+
+def _colspec(dataset_width):
+    """Generates the column specification for the LaTeX `tabular` environment
+    based on the dataset width.
+
+    The first column is justified to the left, all further columns are aligned
+    to the right.
+
+    .. note:: This is only a heuristic and most probably has to be fine-tuned
+    post export. Column alignment should depend on the data type, e.g., textual
+    content should usually be aligned to the left while numeric content almost
+    always should be aligned to the right.
+
+    :param dataset_width: width of the dataset
+    """
+
+    spec = 'l'
+    for _ in range(1, dataset_width):
+        spec += 'r'
+    return spec
+
+
+def _midrule(dataset_width):
+    """Generates the table `midrule`, which may be composed of several
+    `cmidrules`.
+
+    :param dataset_width: width of the dataset to serialize
+    """
+
+    if not dataset_width or dataset_width == 1:
+        return '\\midrule'
+    return ' '.join([_cmidrule(colindex, dataset_width) for colindex in
+                     range(1, dataset_width + 1)])
+
+
+def _cmidrule(colindex, dataset_width):
+    """Generates the `cmidrule` for a single column with appropriate trimming
+    based on the column position.
+
+    :param colindex: Column index
+    :param dataset_width: width of the dataset
+    """
+
+    rule = '\\cmidrule(%s){%d-%d}'
+    if colindex == 1:
+        # Rule of first column is trimmed on the right
+        return rule % ('r', colindex, colindex)
+    if colindex == dataset_width:
+        # Rule of last column is trimmed on the left
+        return rule % ('l', colindex, colindex)
+    # Inner columns are trimmed on the left and right
+    return rule % ('lr', colindex, colindex)
+
+
+def _serialize_row(row):
+    """Returns string representation of a single row.
+
+    :param row: single dataset row
+    """
+
+    new_row = [_escape_tex_reserved_symbols(unicode(item)) if item else '' for
+               item in row]
+    return 6 * ' ' + ' & '.join(new_row) + ' \\\\'
+
+
+def _escape_tex_reserved_symbols(input):
+    """Escapes all TeX reserved symbols ('_', '~', etc.) in a string.
+
+    :param input: String to escape
+    """
+    def replace(match):
+        return TEX_RESERVED_SYMBOLS_MAP[match.group()]
+    return TEX_RESERVED_SYMBOLS_RE.sub(replace, input)

tablib/formats/_ods.py

@@ -14,7 +14,7 @@ else:
 from tablib.compat import opendocument, style, table, text, unicode
 
 title = 'ods'
-extentions = ('ods',)
+extensions = ('ods',)
 
 bold = style.Style(name="bold", family="paragraph")
 bold.addElement(style.TextProperties(fontweight="bold", fontweightasian="bold", fontweightcomplex="bold"))

tablib/formats/_tsv.py

@@ -3,57 +3,28 @@
 """ Tablib - TSV (Tab Separated Values) Support.
 """
 
-from tablib.compat import is_py3, csv, StringIO
-
-
+from tablib.formats._csv import (
+    export_set as export_set_wrapper,
+    import_set as import_set_wrapper,
+    detect as detect_wrapper,
+)
 
 title = 'tsv'
-extentions = ('tsv',)
+extensions = ('tsv',)
 
 DEFAULT_ENCODING = 'utf-8'
+DELIMITER = '\t'
 
 def export_set(dataset):
-    """Returns a TSV representation of Dataset."""
-
-    stream = StringIO()
-
-    if is_py3:
-        _tsv = csv.writer(stream, delimiter='\t')
-    else:
-        _tsv = csv.writer(stream, encoding=DEFAULT_ENCODING, delimiter='\t')
-
-    for row in dataset._package(dicts=False):
-        _tsv.writerow(row)
-
-    return stream.getvalue()
+    """Returns TSV representation of Dataset."""
+    return export_set_wrapper(dataset, delimiter=DELIMITER)
 
 
 def import_set(dset, in_stream, headers=True):
     """Returns dataset from TSV stream."""
-
-    dset.wipe()
-
-    if is_py3:
-        rows = csv.reader(in_stream.split('\r\n'), delimiter='\t')
-    else:
-        rows = csv.reader(in_stream.split('\r\n'), delimiter='\t',
-                          encoding=DEFAULT_ENCODING)
-
-    for i, row in enumerate(rows):
-        # Skip empty rows
-        if not row:
-            continue
-
-        if (i == 0) and (headers):
-            dset.headers = row
-        else:
-            dset.append(row)
+    return import_set_wrapper(dset, in_stream, headers=headers, delimiter=DELIMITER)
 
 
 def detect(stream):
     """Returns True if given stream is valid TSV."""
-    try:
-        csv.Sniffer().sniff(stream, delimiters='\t')
-        return True
-    except csv.Error:
-        return False
+    return detect_wrapper(stream, delimiter=DELIMITER)

tablib/formats/_xls.py

@@ -5,17 +5,36 @@
 
 import sys
 
-from tablib.compat import BytesIO, xlwt
-
+from tablib.compat import BytesIO, xlwt, xlrd, XLRDError, xrange
+import tablib
 
 title = 'xls'
-extentions = ('xls',)
+extensions = ('xls',)
 
 # special styles
 wrap = xlwt.easyxf("alignment: wrap on")
 bold = xlwt.easyxf("font: bold on")
 
 
+def detect(stream):
+    """Returns True if given stream is a readable excel file."""
+    try:
+        xlrd.open_workbook(file_contents=stream)
+        return True
+    except (TypeError, XLRDError):
+        pass 
+    try:
+        xlrd.open_workbook(file_contents=stream.read())
+        return True
+    except (AttributeError, XLRDError):
+        pass
+    try:
+        xlrd.open_workbook(filename=stream)
+        return True
+    except:
+        return False
+
+
 def export_set(dataset):
     """Returns XLS representation of Dataset."""
 
@@ -45,6 +64,42 @@ def export_book(databook):
     return stream.getvalue()
 
 
+def import_set(dset, in_stream, headers=True):
+    """Returns databook from XLS stream."""
+
+    dset.wipe()
+
+    xls_book = xlrd.open_workbook(file_contents=in_stream)
+    sheet = xls_book.sheet_by_index(0)
+
+    dset.title = sheet.name
+
+    for i in xrange(sheet.nrows):
+        if (i == 0) and (headers):
+            dset.headers = sheet.row_values(0)
+        else:
+            dset.append(sheet.row_values(i))
+
+def import_book(dbook, in_stream, headers=True):
+    """Returns databook from XLS stream."""
+
+    dbook.wipe()
+
+    xls_book = xlrd.open_workbook(file_contents=in_stream)
+
+    for sheet in xls_book.sheets():
+        data = tablib.Dataset()
+        data.title = sheet.name
+
+        for i in xrange(sheet.nrows):
+            if (i == 0) and (headers):
+                data.headers = sheet.row_values(0)
+            else:
+                data.append(sheet.row_values(i))
+
+        dbook.add_sheet(data)
+
+
 def dset_sheet(dataset, ws):
     """Completes given worksheet from given Dataset."""
     _package = dataset._package(dicts=False)
@@ -78,5 +133,3 @@ def dset_sheet(dataset, ws):
                         ws.write(i, j, col)
                 except TypeError:
                     ws.write(i, j, col)
-
-

tablib/formats/_xlsx.py

@@ -12,6 +12,7 @@ else:
     from cStringIO import StringIO as BytesIO
 
 from tablib.compat import openpyxl
+import tablib
 
 Workbook = openpyxl.workbook.Workbook
 ExcelWriter = openpyxl.writer.excel.ExcelWriter
@@ -21,40 +22,88 @@ from tablib.compat import unicode
 
 
 title = 'xlsx'
-extentions = ('xlsx',)
+extensions = ('xlsx',)
 
-def export_set(dataset):
+
+def detect(stream):
+    """Returns True if given stream is a readable excel file."""
+    try:
+        openpyxl.reader.excel.load_workbook(stream)
+        return True
+    except openpyxl.shared.exc.InvalidFileException:
+        pass
+
+def export_set(dataset, freeze_panes=True):
     """Returns XLSX representation of Dataset."""
 
     wb = Workbook()
     ws = wb.worksheets[0]
     ws.title = dataset.title if dataset.title else 'Tablib Dataset'
 
-    dset_sheet(dataset, ws)
+    dset_sheet(dataset, ws, freeze_panes=freeze_panes)
 
     stream = BytesIO()
     wb.save(stream)
     return stream.getvalue()
 
 
-def export_book(databook):
+def export_book(databook, freeze_panes=True):
     """Returns XLSX representation of DataBook."""
 
     wb = Workbook()
-    ew = ExcelWriter(workbook = wb)
+    wb.worksheets = []
     for i, dset in enumerate(databook._datasets):
         ws = wb.create_sheet()
         ws.title = dset.title if dset.title else 'Sheet%s' % (i)
 
-        dset_sheet(dset, ws)
+        dset_sheet(dset, ws, freeze_panes=freeze_panes)
 
 
     stream = BytesIO()
-    ew.save(stream)
+    wb.save(stream)
     return stream.getvalue()
 
 
-def dset_sheet(dataset, ws):
+def import_set(dset, in_stream, headers=True):
+    """Returns databook from XLS stream."""
+
+    dset.wipe()
+
+    xls_book = openpyxl.reader.excel.load_workbook(BytesIO(in_stream))
+    sheet = xls_book.get_active_sheet()
+
+    dset.title = sheet.title
+
+    for i, row in enumerate(sheet.rows):
+        row_vals = [c.value for c in row]
+        if (i == 0) and (headers):
+            dset.headers = row_vals
+        else:
+            dset.append(row_vals)
+
+
+def import_book(dbook, in_stream, headers=True):
+    """Returns databook from XLS stream."""
+
+    dbook.wipe()
+
+    xls_book = openpyxl.reader.excel.load_workbook(BytesIO(in_stream))
+
+    for sheet in xls_book.worksheets:
+        data = tablib.Dataset()
+        data.title = sheet.title
+
+        for i, row in enumerate(sheet.rows):
+            row_vals = [c.value for c in row]
+            if (i == 0) and (headers):
+                data.headers = row_vals
+            else:
+                data.append(row_vals)
+
+        dbook.add_sheet(data)
+
+
+def dset_sheet(dataset, ws, freeze_panes=True):
     """Completes given worksheet from given Dataset."""
     _package = dataset._package(dicts=False)
 
@@ -74,9 +123,11 @@ def dset_sheet(dataset, ws):
                 ws.cell('%s%s'%(col_idx, row_number)).value = unicode(col)
                 style = ws.get_style('%s%s' % (col_idx, row_number))
                 style.font.bold = True
-                ws.freeze_panes = '%s%s' % (col_idx, row_number)
-
-
+                if freeze_panes:
+                    # As already done in #53, but after Merge lost:
+                    #  Export Freeze only after first Line
+                    ws.freeze_panes = 'A2'
+                    
             # bold separators
             elif len(row) < dataset.width:
                 ws.cell('%s%s'%(col_idx, row_number)).value = unicode(

tablib/formats/_yaml.py

@@ -13,32 +13,28 @@ except ImportError:
     else:
         import tablib.packages.yaml as yaml
 
-
 import tablib
 
-
-
 title = 'yaml'
-extentions = ('yaml', 'yml')
-
+extensions = ('yaml', 'yml')
 
 
 def export_set(dataset):
     """Returns YAML representation of Dataset."""
 
-    return yaml.dump(dataset._package(ordered=False))
+    return yaml.safe_dump(dataset._package(ordered=False))
 
 
 def export_book(databook):
     """Returns YAML representation of Databook."""
-    return yaml.dump(databook._package())
+    return yaml.safe_dump(databook._package(ordered=False))
 
 
 def import_set(dset, in_stream):
     """Returns dataset from YAML stream."""
 
     dset.wipe()
-    dset.dict = yaml.load(in_stream)
+    dset.dict = yaml.safe_load(in_stream)
 
 
 def import_book(dbook, in_stream):
@@ -52,13 +48,15 @@ def import_book(dbook, in_stream):
         data.dict = sheet['data']
         dbook.add_sheet(data)
 
+
 def detect(stream):
     """Returns True if given stream is valid YAML."""
     try:
-        _yaml = yaml.load(stream)
+        _yaml = yaml.safe_load(stream)
         if isinstance(_yaml, (list, tuple, dict)):
             return True
         else:
             return False
-    except yaml.parser.ParserError:
-        return False
+    except (yaml.parser.ParserError, yaml.reader.ReaderError,
+            yaml.scanner.ScannerError):
+        return False

tablib/packages/dbfpy/dbf.py

@@ -0,0 +1,297 @@
+#! /usr/bin/env python
+"""DBF accessing helpers.
+
+FIXME: more documentation needed
+
+Examples:
+
+    Create new table, setup structure, add records:
+
+        dbf = Dbf(filename, new=True)
+        dbf.addField(
+            ("NAME", "C", 15),
+            ("SURNAME", "C", 25),
+            ("INITIALS", "C", 10),
+            ("BIRTHDATE", "D"),
+        )
+        for (n, s, i, b) in (
+            ("John", "Miller", "YC", (1980, 10, 11)),
+            ("Andy", "Larkin", "", (1980, 4, 11)),
+        ):
+            rec = dbf.newRecord()
+            rec["NAME"] = n
+            rec["SURNAME"] = s
+            rec["INITIALS"] = i
+            rec["BIRTHDATE"] = b
+            rec.store()
+        dbf.close()
+
+    Open existed dbf, read some data:
+
+        dbf = Dbf(filename, True)
+        for rec in dbf:
+            for fldName in dbf.fieldNames:
+                print '%s:\t %s (%s)' % (fldName, rec[fldName],
+                    type(rec[fldName]))
+            print
+        dbf.close()
+
+"""
+"""History (most recent first):
+11-feb-2007 [als]   export INVALID_VALUE;
+                    Dbf: added .ignoreErrors, .INVALID_VALUE
+04-jul-2006 [als]   added export declaration
+20-dec-2005 [yc]    removed fromStream and newDbf methods:
+                    use argument of __init__ call must be used instead;
+                    added class fields pointing to the header and
+                    record classes.
+17-dec-2005 [yc]    split to several modules; reimplemented
+13-dec-2005 [yc]    adapted to the changes of the `strutil` module.
+13-sep-2002 [als]   support FoxPro Timestamp datatype
+15-nov-1999 [jjk]   documentation updates, add demo
+24-aug-1998 [jjk]   add some encodeValue methods (not tested), other tweaks
+08-jun-1998 [jjk]   fix problems, add more features
+20-feb-1998 [jjk]   fix problems, add more features
+19-feb-1998 [jjk]   add create/write capabilities
+18-feb-1998 [jjk]   from dbfload.py
+"""
+
+__version__ = "$Revision: 1.7 $"[11:-2]
+__date__ = "$Date: 2007/02/11 09:23:13 $"[7:-2]
+__author__ = "Jeff Kunce <kuncej@mail.conservation.state.mo.us>"
+
+__all__ = ["Dbf"]
+
+from . import header
+from . import record
+from utils import INVALID_VALUE
+
+
+class Dbf(object):
+    """DBF accessor.
+
+    FIXME:
+        docs and examples needed (dont' forget to tell
+        about problems adding new fields on the fly)
+
+    Implementation notes:
+        ``_new`` field is used to indicate whether this is
+        a new data table. `addField` could be used only for
+        the new tables! If at least one record was appended
+        to the table it's structure couldn't be changed.
+
+    """
+
+    __slots__ = ("name", "header", "stream",
+                 "_changed", "_new", "_ignore_errors")
+
+    HeaderClass = header.DbfHeader
+    RecordClass = record.DbfRecord
+    INVALID_VALUE = INVALID_VALUE
+
+    # initialization and creation helpers
+
+    def __init__(self, f, readOnly=False, new=False, ignoreErrors=False):
+        """Initialize instance.
+
+        Arguments:
+            f:
+                Filename or file-like object.
+            new:
+                True if new data table must be created. Assume
+                data table exists if this argument is False.
+            readOnly:
+                if ``f`` argument is a string file will
+                be opend in read-only mode; in other cases
+                this argument is ignored. This argument is ignored
+                even if ``new`` argument is True.
+            headerObj:
+                `header.DbfHeader` instance or None. If this argument
+                is None, new empty header will be used with the
+                all fields set by default.
+            ignoreErrors:
+                if set, failing field value conversion will return
+                ``INVALID_VALUE`` instead of raising conversion error.
+
+        """
+        if isinstance(f, basestring):
+            # a filename
+            self.name = f
+            if new:
+                # new table (table file must be
+                # created or opened and truncated)
+                self.stream = file(f, "w+b")
+            else:
+                # tabe file must exist
+                self.stream = file(f, ("r+b", "rb")[bool(readOnly)])
+        else:
+            # a stream
+            self.name = getattr(f, "name", "")
+            self.stream = f
+        if new:
+            # if this is a new table, header will be empty
+            self.header = self.HeaderClass()
+        else:
+            # or instantiated using stream
+            self.header = self.HeaderClass.fromStream(self.stream)
+        self.ignoreErrors = ignoreErrors
+        self._new = bool(new)
+        self._changed = False
+
+    # properties
+
+    closed = property(lambda self: self.stream.closed)
+    recordCount = property(lambda self: self.header.recordCount)
+    fieldNames = property(
+        lambda self: [_fld.name for _fld in self.header.fields])
+    fieldDefs = property(lambda self: self.header.fields)
+    changed = property(lambda self: self._changed or self.header.changed)
+
+    def ignoreErrors(self, value):
+        """Update `ignoreErrors` flag on the header object and self"""
+        self.header.ignoreErrors = self._ignore_errors = bool(value)
+
+    ignoreErrors = property(
+        lambda self: self._ignore_errors,
+        ignoreErrors,
+        doc="""Error processing mode for DBF field value conversion
+
+        if set, failing field value conversion will return
+        ``INVALID_VALUE`` instead of raising conversion error.
+
+        """)
+
+    # protected methods
+
+    def _fixIndex(self, index):
+        """Return fixed index.
+
+        This method fails if index isn't a numeric object
+        (long or int). Or index isn't in a valid range
+        (less or equal to the number of records in the db).
+
+        If ``index`` is a negative number, it will be
+        treated as a negative indexes for list objects.
+
+        Return:
+            Return value is numeric object maning valid index.
+
+        """
+        if not isinstance(index, (int, long)):
+            raise TypeError("Index must be a numeric object")
+        if index < 0:
+            # index from the right side
+            # fix it to the left-side index
+            index += len(self) + 1
+        if index >= len(self):
+            raise IndexError("Record index out of range")
+        return index
+
+    # iterface methods
+
+    def close(self):
+        self.flush()
+        self.stream.close()
+
+    def flush(self):
+        """Flush data to the associated stream."""
+        if self.changed:
+            self.header.setCurrentDate()
+            self.header.write(self.stream)
+            self.stream.flush()
+            self._changed = False
+
+    def indexOfFieldName(self, name):
+        """Index of field named ``name``."""
+        # FIXME: move this to header class
+        return self.header.fields.index(name)
+
+    def newRecord(self):
+        """Return new record, which belong to this table."""
+        return self.RecordClass(self)
+
+    def append(self, record):
+        """Append ``record`` to the database."""
+        record.index = self.header.recordCount
+        record._write()
+        self.header.recordCount += 1
+        self._changed = True
+        self._new = False
+
+    def addField(self, *defs):
+        """Add field definitions.
+
+        For more information see `header.DbfHeader.addField`.
+
+        """
+        if self._new:
+            self.header.addField(*defs)
+        else:
+            raise TypeError("At least one record was added, "
+                            "structure can't be changed")
+
+    # 'magic' methods (representation and sequence interface)
+
+    def __repr__(self):
+        return "Dbf stream '%s'\n" % self.stream + repr(self.header)
+
+    def __len__(self):
+        """Return number of records."""
+        return self.recordCount
+
+    def __getitem__(self, index):
+        """Return `DbfRecord` instance."""
+        return self.RecordClass.fromStream(self, self._fixIndex(index))
+
+    def __setitem__(self, index, record):
+        """Write `DbfRecord` instance to the stream."""
+        record.index = self._fixIndex(index)
+        record._write()
+        self._changed = True
+        self._new = False
+
+        # def __del__(self):
+        #    """Flush stream upon deletion of the object."""
+        #    self.flush()
+
+
+def demo_read(filename):
+    _dbf = Dbf(filename, True)
+    for _rec in _dbf:
+        print
+        print(repr(_rec))
+    _dbf.close()
+
+
+def demo_create(filename):
+    _dbf = Dbf(filename, new=True)
+    _dbf.addField(
+        ("NAME", "C", 15),
+        ("SURNAME", "C", 25),
+        ("INITIALS", "C", 10),
+        ("BIRTHDATE", "D"),
+    )
+    for (_n, _s, _i, _b) in (
+            ("John", "Miller", "YC", (1981, 1, 2)),
+            ("Andy", "Larkin", "AL", (1982, 3, 4)),
+            ("Bill", "Clinth", "", (1983, 5, 6)),
+            ("Bobb", "McNail", "", (1984, 7, 8)),
+    ):
+        _rec = _dbf.newRecord()
+        _rec["NAME"] = _n
+        _rec["SURNAME"] = _s
+        _rec["INITIALS"] = _i
+        _rec["BIRTHDATE"] = _b
+        _rec.store()
+    print(repr(_dbf))
+    _dbf.close()
+
+
+if __name__ == '__main__':
+    import sys
+
+    _name = len(sys.argv) > 1 and sys.argv[1] or "county.dbf"
+    demo_create(_name)
+    demo_read(_name)
+
+    # vim: set et sw=4 sts=4 :

tablib/packages/dbfpy/dbfnew.py

@@ -0,0 +1,189 @@
+#!/usr/bin/python
+""".DBF creation helpers.
+
+Note: this is a legacy interface.  New code should use Dbf class
+    for table creation (see examples in dbf.py)
+
+TODO:
+  - handle Memo fields.
+  - check length of the fields accoring to the
+    `http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
+
+"""
+"""History (most recent first)
+04-jul-2006 [als]   added export declaration;
+                    updated for dbfpy 2.0
+15-dec-2005 [yc]    define dbf_new.__slots__
+14-dec-2005 [yc]    added vim modeline; retab'd; added doc-strings;
+                    dbf_new now is a new class (inherited from object)
+??-jun-2000 [--]    added by Hans Fiby
+"""
+
+__version__ = "$Revision: 1.4 $"[11:-2]
+__date__ = "$Date: 2006/07/04 08:18:18 $"[7:-2]
+
+__all__ = ["dbf_new"]
+
+from dbf import *
+from fields import *
+from header import *
+from record import *
+
+
+class _FieldDefinition(object):
+    """Field definition.
+
+    This is a simple structure, which contains ``name``, ``type``,
+    ``len``, ``dec`` and ``cls`` fields.
+
+    Objects also implement get/setitem magic functions, so fields
+    could be accessed via sequence iterface, where 'name' has
+    index 0, 'type' index 1, 'len' index 2, 'dec' index 3 and
+    'cls' could be located at index 4.
+
+    """
+
+    __slots__ = "name", "type", "len", "dec", "cls"
+
+    # WARNING: be attentive - dictionaries are mutable!
+    FLD_TYPES = {
+        # type: (cls, len)
+        "C": (DbfCharacterFieldDef, None),
+        "N": (DbfNumericFieldDef, None),
+        "L": (DbfLogicalFieldDef, 1),
+        # FIXME: support memos
+        # "M": (DbfMemoFieldDef),
+        "D": (DbfDateFieldDef, 8),
+        # FIXME: I'm not sure length should be 14 characters!
+        # but temporary I use it, cuz date is 8 characters
+        # and time 6 (hhmmss)
+        "T": (DbfDateTimeFieldDef, 14),
+    }
+
+    def __init__(self, name, type, len=None, dec=0):
+        _cls, _len = self.FLD_TYPES[type]
+        if _len is None:
+            if len is None:
+                raise ValueError("Field length must be defined")
+            _len = len
+        self.name = name
+        self.type = type
+        self.len = _len
+        self.dec = dec
+        self.cls = _cls
+
+    def getDbfField(self):
+        "Return `DbfFieldDef` instance from the current definition."
+        return self.cls(self.name, self.len, self.dec)
+
+    def appendToHeader(self, dbfh):
+        """Create a `DbfFieldDef` instance and append it to the dbf header.
+
+        Arguments:
+            dbfh: `DbfHeader` instance.
+
+        """
+        _dbff = self.getDbfField()
+        dbfh.addField(_dbff)
+
+
+class dbf_new(object):
+    """New .DBF creation helper.
+
+    Example Usage:
+
+        dbfn = dbf_new()
+        dbfn.add_field("name",'C',80)
+        dbfn.add_field("price",'N',10,2)
+        dbfn.add_field("date",'D',8)
+        dbfn.write("tst.dbf")
+
+    Note:
+        This module cannot handle Memo-fields,
+        they are special.
+
+    """
+
+    __slots__ = ("fields",)
+
+    FieldDefinitionClass = _FieldDefinition
+
+    def __init__(self):
+        self.fields = []
+
+    def add_field(self, name, typ, len, dec=0):
+        """Add field definition.
+
+        Arguments:
+            name:
+                field name (str object). field name must not
+                contain ASCII NULs and it's length shouldn't
+                exceed 10 characters.
+            typ:
+                type of the field. this must be a single character
+                from the "CNLMDT" set meaning character, numeric,
+                logical, memo, date and date/time respectively.
+            len:
+                length of the field. this argument is used only for
+                the character and numeric fields. all other fields
+                have fixed length.
+                FIXME: use None as a default for this argument?
+            dec:
+                decimal precision. used only for the numric fields.
+
+        """
+        self.fields.append(self.FieldDefinitionClass(name, typ, len, dec))
+
+    def write(self, filename):
+        """Create empty .DBF file using current structure."""
+        _dbfh = DbfHeader()
+        _dbfh.setCurrentDate()
+        for _fldDef in self.fields:
+            _fldDef.appendToHeader(_dbfh)
+        _dbfStream = file(filename, "wb")
+        _dbfh.write(_dbfStream)
+        _dbfStream.close()
+
+    def write_stream(self, stream):
+        _dbfh = DbfHeader()
+        _dbfh.setCurrentDate()
+        for _fldDef in self.fields:
+            _fldDef.appendToHeader(_dbfh)
+        _dbfh.write(stream)
+
+
+if __name__ == '__main__':
+    # create a new DBF-File
+    dbfn = dbf_new()
+    dbfn.add_field("name", 'C', 80)
+    dbfn.add_field("price", 'N', 10, 2)
+    dbfn.add_field("date", 'D', 8)
+    dbfn.write("tst.dbf")
+    # test new dbf
+    print "*** created tst.dbf: ***"
+    dbft = Dbf('tst.dbf', readOnly=0)
+    print repr(dbft)
+    # add a record
+    rec = DbfRecord(dbft)
+    rec['name'] = 'something'
+    rec['price'] = 10.5
+    rec['date'] = (2000, 1, 12)
+    rec.store()
+    # add another record
+    rec = DbfRecord(dbft)
+    rec['name'] = 'foo and bar'
+    rec['price'] = 12234
+    rec['date'] = (1992, 7, 15)
+    rec.store()
+
+    # show the records
+    print "*** inserted 2 records into tst.dbf: ***"
+    print repr(dbft)
+    for i1 in range(len(dbft)):
+        rec = dbft[i1]
+        for fldName in dbft.fieldNames:
+            print '%s:\t %s' % (fldName, rec[fldName])
+        print
+    dbft.close()
+
+    # vim: set et sts=4 sw=4 :

tablib/packages/dbfpy/fields.py

@@ -0,0 +1,466 @@
+"""DBF fields definitions.
+
+TODO:
+  - make memos work
+"""
+"""History (most recent first):
+26-may-2009 [als]   DbfNumericFieldDef.decodeValue: strip zero bytes
+05-feb-2009 [als]   DbfDateFieldDef.encodeValue: empty arg produces empty date
+16-sep-2008 [als]   DbfNumericFieldDef decoding looks for decimal point
+                    in the value to select float or integer return type
+13-mar-2008 [als]   check field name length in constructor
+11-feb-2007 [als]   handle value conversion errors
+10-feb-2007 [als]   DbfFieldDef: added .rawFromRecord()
+01-dec-2006 [als]   Timestamp columns use None for empty values
+31-oct-2006 [als]   support field types 'F' (float), 'I' (integer)
+                    and 'Y' (currency);
+                    automate export and registration of field classes
+04-jul-2006 [als]   added export declaration
+10-mar-2006 [als]   decode empty values for Date and Logical fields;
+                    show field name in errors
+10-mar-2006 [als]   fix Numeric value decoding: according to spec,
+                    value always is string representation of the number;
+                    ensure that encoded Numeric value fits into the field
+20-dec-2005 [yc]    use field names in upper case
+15-dec-2005 [yc]    field definitions moved from `dbf`.
+"""
+
+__version__ = "$Revision: 1.14 $"[11:-2]
+__date__ = "$Date: 2009/05/26 05:16:51 $"[7:-2]
+
+__all__ = ["lookupFor",] # field classes added at the end of the module
+
+import datetime
+import struct
+import sys
+
+from . import utils
+
+## abstract definitions
+
+class DbfFieldDef(object):
+    """Abstract field definition.
+
+    Child classes must override ``type`` class attribute to provide datatype
+    infromation of the field definition. For more info about types visit
+    `http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
+
+    Also child classes must override ``defaultValue`` field to provide
+    default value for the field value.
+
+    If child class has fixed length ``length`` class attribute must be
+    overriden and set to the valid value. None value means, that field
+    isn't of fixed length.
+
+    Note: ``name`` field must not be changed after instantiation.
+
+    """
+
+    __slots__ = ("name", "length", "decimalCount",
+        "start", "end", "ignoreErrors")
+
+    # length of the field, None in case of variable-length field,
+    # or a number if this field is a fixed-length field
+    length = None
+
+    # field type. for more information about fields types visit
+    # `http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
+    # must be overriden in child classes
+    typeCode = None
+
+    # default value for the field. this field must be
+    # overriden in child classes
+    defaultValue = None
+
+    def __init__(self, name, length=None, decimalCount=None,
+        start=None, stop=None, ignoreErrors=False,
+    ):
+        """Initialize instance."""
+        assert self.typeCode is not None, "Type code must be overriden"
+        assert self.defaultValue is not None, "Default value must be overriden"
+        ## fix arguments
+        if len(name) >10:
+            raise ValueError("Field name \"%s\" is too long" % name)
+        name = str(name).upper()
+        if self.__class__.length is None:
+            if length is None:
+                raise ValueError("[%s] Length isn't specified" % name)
+            length = int(length)
+            if length <= 0:
+                raise ValueError("[%s] Length must be a positive integer"
+                    % name)
+        else:
+            length = self.length
+        if decimalCount is None:
+            decimalCount = 0
+        ## set fields
+        self.name = name
+        # FIXME: validate length according to the specification at
+        # http://www.clicketyclick.dk/databases/xbase/format/data_types.html
+        self.length = length
+        self.decimalCount = decimalCount
+        self.ignoreErrors = ignoreErrors
+        self.start = start
+        self.end = stop
+
+    def __cmp__(self, other):
+        return cmp(self.name, str(other).upper())
+
+    def __hash__(self):
+        return hash(self.name)
+
+    def fromString(cls, string, start, ignoreErrors=False):
+        """Decode dbf field definition from the string data.
+
+        Arguments:
+            string:
+                a string, dbf definition is decoded from. length of
+                the string must be 32 bytes.
+            start:
+                position in the database file.
+            ignoreErrors:
+                initial error processing mode for the new field (boolean)
+
+        """
+        assert len(string) == 32
+        _length = ord(string[16])
+        return cls(utils.unzfill(string)[:11], _length, ord(string[17]),
+            start, start + _length, ignoreErrors=ignoreErrors)
+    fromString = classmethod(fromString)
+
+    def toString(self):
+        """Return encoded field definition.
+
+        Return:
+            Return value is a string object containing encoded
+            definition of this field.
+
+        """
+        if sys.version_info < (2, 4):
+            # earlier versions did not support padding character
+            _name = self.name[:11] + "\0" * (11 - len(self.name))
+        else:
+            _name = self.name.ljust(11, '\0')
+        return (
+            _name +
+            self.typeCode +
+            #data address
+            chr(0) * 4 +
+            chr(self.length) +
+            chr(self.decimalCount) +
+            chr(0) * 14
+        )
+
+    def __repr__(self):
+        return "%-10s %1s %3d %3d" % self.fieldInfo()
+
+    def fieldInfo(self):
+        """Return field information.
+
+        Return:
+            Return value is a (name, type, length, decimals) tuple.
+
+        """
+        return (self.name, self.typeCode, self.length, self.decimalCount)
+
+    def rawFromRecord(self, record):
+        """Return a "raw" field value from the record string."""
+        return record[self.start:self.end]
+
+    def decodeFromRecord(self, record):
+        """Return decoded field value from the record string."""
+        try:
+            return self.decodeValue(self.rawFromRecord(record))
+        except:
+            if self.ignoreErrors:
+                return utils.INVALID_VALUE
+            else:
+                raise
+
+    def decodeValue(self, value):
+        """Return decoded value from string value.
+
+        This method shouldn't be used publicly. It's called from the
+        `decodeFromRecord` method.
+
+        This is an abstract method and it must be overridden in child classes.
+        """
+        raise NotImplementedError
+
+    def encodeValue(self, value):
+        """Return str object containing encoded field value.
+
+        This is an abstract method and it must be overriden in child classes.
+        """
+        raise NotImplementedError
+
+## real classes
+
+class DbfCharacterFieldDef(DbfFieldDef):
+    """Definition of the character field."""
+
+    typeCode = "C"
+    defaultValue = ""
+
+    def decodeValue(self, value):
+        """Return string object.
+
+        Return value is a ``value`` argument with stripped right spaces.
+
+        """
+        return value.rstrip(" ")
+
+    def encodeValue(self, value):
+        """Return raw data string encoded from a ``value``."""
+        return str(value)[:self.length].ljust(self.length)
+
+
+class DbfNumericFieldDef(DbfFieldDef):
+    """Definition of the numeric field."""
+
+    typeCode = "N"
+    # XXX: now I'm not sure it was a good idea to make a class field
+    # `defaultValue` instead of a generic method as it was implemented
+    # previously -- it's ok with all types except number, cuz
+    # if self.decimalCount is 0, we should return 0 and 0.0 otherwise.
+    defaultValue = 0
+
+    def decodeValue(self, value):
+        """Return a number decoded from ``value``.
+
+        If decimals is zero, value will be decoded as an integer;
+        or as a float otherwise.
+
+        Return:
+            Return value is a int (long) or float instance.
+
+        """
+        value = value.strip(" \0")
+        if "." in value:
+            # a float (has decimal separator)
+            return float(value)
+        elif value:
+            # must be an integer
+            return int(value)
+        else:
+            return 0
+
+    def encodeValue(self, value):
+        """Return string containing encoded ``value``."""
+        _rv = ("%*.*f" % (self.length, self.decimalCount, value))
+        if len(_rv) > self.length:
+            _ppos = _rv.find(".")
+            if 0 <= _ppos <= self.length:
+                _rv = _rv[:self.length]
+            else:
+                raise ValueError("[%s] Numeric overflow: %s (field width: %i)"
+                    % (self.name, _rv, self.length))
+        return _rv
+
+class DbfFloatFieldDef(DbfNumericFieldDef):
+    """Definition of the float field - same as numeric."""
+
+    typeCode = "F"
+
+class DbfIntegerFieldDef(DbfFieldDef):
+    """Definition of the integer field."""
+
+    typeCode = "I"
+    length = 4
+    defaultValue = 0
+
+    def decodeValue(self, value):
+        """Return an integer number decoded from ``value``."""
+        return struct.unpack("<i", value)[0]
+
+    def encodeValue(self, value):
+        """Return string containing encoded ``value``."""
+        return struct.pack("<i", int(value))
+
+class DbfCurrencyFieldDef(DbfFieldDef):
+    """Definition of the currency field."""
+
+    typeCode = "Y"
+    length = 8
+    defaultValue = 0.0
+
+    def decodeValue(self, value):
+        """Return float number decoded from ``value``."""
+        return struct.unpack("<q", value)[0] / 10000.
+
+    def encodeValue(self, value):
+        """Return string containing encoded ``value``."""
+        return struct.pack("<q", round(value * 10000))
+
+class DbfLogicalFieldDef(DbfFieldDef):
+    """Definition of the logical field."""
+
+    typeCode = "L"
+    defaultValue = -1
+    length = 1
+
+    def decodeValue(self, value):
+        """Return True, False or -1 decoded from ``value``."""
+        # Note: value always is 1-char string
+        if value == "?":
+            return -1
+        if value in "NnFf ":
+            return False
+        if value in "YyTt":
+            return True
+        raise ValueError("[%s] Invalid logical value %r" % (self.name, value))
+
+    def encodeValue(self, value):
+        """Return a character from the "TF?" set.
+
+        Return:
+            Return value is "T" if ``value`` is True
+            "?" if value is -1 or False otherwise.
+
+        """
+        if value is True:
+            return "T"
+        if value == -1:
+            return "?"
+        return "F"
+
+
+class DbfMemoFieldDef(DbfFieldDef):
+    """Definition of the memo field.
+
+    Note: memos aren't currenly completely supported.
+
+    """
+
+    typeCode = "M"
+    defaultValue = " " * 10
+    length = 10
+
+    def decodeValue(self, value):
+        """Return int .dbt block number decoded from the string object."""
+        #return int(value)
+        raise NotImplementedError
+
+    def encodeValue(self, value):
+        """Return raw data string encoded from a ``value``.
+
+        Note: this is an internal method.
+
+        """
+        #return str(value)[:self.length].ljust(self.length)
+        raise NotImplementedError
+
+
+class DbfDateFieldDef(DbfFieldDef):
+    """Definition of the date field."""
+
+    typeCode = "D"
+    defaultValue = utils.classproperty(lambda cls: datetime.date.today())
+    # "yyyymmdd" gives us 8 characters
+    length = 8
+
+    def decodeValue(self, value):
+        """Return a ``datetime.date`` instance decoded from ``value``."""
+        if value.strip():
+            return utils.getDate(value)
+        else:
+            return None
+
+    def encodeValue(self, value):
+        """Return a string-encoded value.
+
+        ``value`` argument should be a value suitable for the
+        `utils.getDate` call.
+
+        Return:
+            Return value is a string in format "yyyymmdd".
+
+        """
+        if value:
+            return utils.getDate(value).strftime("%Y%m%d")
+        else:
+            return " " * self.length
+
+
+class DbfDateTimeFieldDef(DbfFieldDef):
+    """Definition of the timestamp field."""
+
+    # a difference between JDN (Julian Day Number)
+    # and GDN (Gregorian Day Number). note, that GDN < JDN
+    JDN_GDN_DIFF = 1721425
+    typeCode = "T"
+    defaultValue = utils.classproperty(lambda cls: datetime.datetime.now())
+    # two 32-bits integers representing JDN and amount of
+    # milliseconds respectively gives us 8 bytes.
+    # note, that values must be encoded in LE byteorder.
+    length = 8
+
+    def decodeValue(self, value):
+        """Return a `datetime.datetime` instance."""
+        assert len(value) == self.length
+        # LE byteorder
+        _jdn, _msecs = struct.unpack("<2I", value)
+        if _jdn >= 1:
+            _rv = datetime.datetime.fromordinal(_jdn - self.JDN_GDN_DIFF)
+            _rv += datetime.timedelta(0, _msecs / 1000.0)
+        else:
+            # empty date
+            _rv = None
+        return _rv
+
+    def encodeValue(self, value):
+        """Return a string-encoded ``value``."""
+        if value:
+            value = utils.getDateTime(value)
+            # LE byteorder
+            _rv = struct.pack("<2I", value.toordinal() + self.JDN_GDN_DIFF,
+                (value.hour * 3600 + value.minute * 60 + value.second) * 1000)
+        else:
+            _rv = "\0" * self.length
+        assert len(_rv) == self.length
+        return _rv
+
+
+_fieldsRegistry = {}
+
+def registerField(fieldCls):
+    """Register field definition class.
+
+    ``fieldCls`` should be subclass of the `DbfFieldDef`.
+
+    Use `lookupFor` to retrieve field definition class
+    by the type code.
+
+    """
+    assert fieldCls.typeCode is not None, "Type code isn't defined"
+    # XXX: use fieldCls.typeCode.upper()? in case of any decign
+    # don't forget to look to the same comment in ``lookupFor`` method
+    _fieldsRegistry[fieldCls.typeCode] = fieldCls
+
+
+def lookupFor(typeCode):
+    """Return field definition class for the given type code.
+
+    ``typeCode`` must be a single character. That type should be
+    previously registered.
+
+    Use `registerField` to register new field class.
+
+    Return:
+        Return value is a subclass of the `DbfFieldDef`.
+
+    """
+    # XXX: use typeCode.upper()? in case of any decign don't
+    # forget to look to the same comment in ``registerField``
+    return _fieldsRegistry[typeCode]
+
+## register generic types
+
+for (_name, _val) in globals().items():
+    if isinstance(_val, type) and issubclass(_val, DbfFieldDef) \
+    and (_name != "DbfFieldDef"):
+        __all__.append(_name)
+        registerField(_val)
+del _name, _val
+
+# vim: et sts=4 sw=4 :

tablib/packages/dbfpy/header.py

@@ -0,0 +1,275 @@
+"""DBF header definition.
+
+TODO:
+  - handle encoding of the character fields
+    (encoding information stored in the DBF header)
+
+"""
+"""History (most recent first):
+16-sep-2010 [als]   fromStream: fix century of the last update field
+11-feb-2007 [als]   added .ignoreErrors
+10-feb-2007 [als]   added __getitem__: return field definitions
+                    by field name or field number (zero-based)
+04-jul-2006 [als]   added export declaration
+15-dec-2005 [yc]    created
+"""
+
+__version__ = "$Revision: 1.6 $"[11:-2]
+__date__ = "$Date: 2010/09/16 05:06:39 $"[7:-2]
+
+__all__ = ["DbfHeader"]
+
+try:
+    import cStringIO
+except ImportError:
+    # when we're in python3, we cStringIO has been replaced by io.StringIO
+    import io as cStringIO
+import datetime
+import struct
+import time
+
+from . import fields
+from . import utils
+
+
+class DbfHeader(object):
+    """Dbf header definition.
+
+    For more information about dbf header format visit
+    `http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_STRUCT`
+
+    Examples:
+        Create an empty dbf header and add some field definitions:
+            dbfh = DbfHeader()
+            dbfh.addField(("name", "C", 10))
+            dbfh.addField(("date", "D"))
+            dbfh.addField(DbfNumericFieldDef("price", 5, 2))
+        Create a dbf header with field definitions:
+            dbfh = DbfHeader([
+                ("name", "C", 10),
+                ("date", "D"),
+                DbfNumericFieldDef("price", 5, 2),
+            ])
+
+    """
+
+    __slots__ = ("signature", "fields", "lastUpdate", "recordLength",
+        "recordCount", "headerLength", "changed", "_ignore_errors")
+
+    ## instance construction and initialization methods
+
+    def __init__(self, fields=None, headerLength=0, recordLength=0,
+        recordCount=0, signature=0x03, lastUpdate=None, ignoreErrors=False,
+    ):
+        """Initialize instance.
+
+        Arguments:
+            fields:
+                a list of field definitions;
+            recordLength:
+                size of the records;
+            headerLength:
+                size of the header;
+            recordCount:
+                number of records stored in DBF;
+            signature:
+                version number (aka signature). using 0x03 as a default meaning
+                "File without DBT". for more information about this field visit
+                ``http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_NOTE_1_TARGET``
+            lastUpdate:
+                date of the DBF's update. this could be a string ('yymmdd' or
+                'yyyymmdd'), timestamp (int or float), datetime/date value,
+                a sequence (assuming (yyyy, mm, dd, ...)) or an object having
+                callable ``ticks`` field.
+            ignoreErrors:
+                error processing mode for DBF fields (boolean)
+
+        """
+        self.signature = signature
+        if fields is None:
+            self.fields = []
+        else:
+            self.fields = list(fields)
+        self.lastUpdate = utils.getDate(lastUpdate)
+        self.recordLength = recordLength
+        self.headerLength = headerLength
+        self.recordCount = recordCount
+        self.ignoreErrors = ignoreErrors
+        # XXX: I'm not sure this is safe to
+        # initialize `self.changed` in this way
+        self.changed = bool(self.fields)
+
+    # @classmethod
+    def fromString(cls, string):
+        """Return header instance from the string object."""
+        return cls.fromStream(cStringIO.StringIO(str(string)))
+    fromString = classmethod(fromString)
+
+    # @classmethod
+    def fromStream(cls, stream):
+        """Return header object from the stream."""
+        stream.seek(0)
+        _data = stream.read(32)
+        (_cnt, _hdrLen, _recLen) = struct.unpack("<I2H", _data[4:12])
+        #reserved = _data[12:32]
+        _year = ord(_data[1])
+        if _year < 80:
+            # dBase II started at 1980.  It is quite unlikely
+            # that actual last update date is before that year.
+            _year += 2000
+        else:
+            _year += 1900
+        ## create header object
+        _obj = cls(None, _hdrLen, _recLen, _cnt, ord(_data[0]),
+            (_year, ord(_data[2]), ord(_data[3])))
+        ## append field definitions
+        # position 0 is for the deletion flag
+        _pos = 1
+        _data = stream.read(1)
+
+        # The field definitions are ended either by \x0D OR a newline
+        # character, so we need to handle both when reading from a stream.
+        # When writing, dbfpy appears to write newlines instead of \x0D.
+        while _data[0] not in ["\x0D", "\n"]:
+            _data += stream.read(31)
+            _fld = fields.lookupFor(_data[11]).fromString(_data, _pos)
+            _obj._addField(_fld)
+            _pos = _fld.end
+            _data = stream.read(1)
+        return _obj
+    fromStream = classmethod(fromStream)
+
+    ## properties
+
+    year = property(lambda self: self.lastUpdate.year)
+    month = property(lambda self: self.lastUpdate.month)
+    day = property(lambda self: self.lastUpdate.day)
+
+    def ignoreErrors(self, value):
+        """Update `ignoreErrors` flag on self and all fields"""
+        self._ignore_errors = value = bool(value)
+        for _field in self.fields:
+            _field.ignoreErrors = value
+    ignoreErrors = property(
+        lambda self: self._ignore_errors,
+        ignoreErrors,
+        doc="""Error processing mode for DBF field value conversion
+
+        if set, failing field value conversion will return
+        ``INVALID_VALUE`` instead of raising conversion error.
+
+        """)
+
+    ## object representation
+
+    def __repr__(self):
+        _rv = """\
+Version (signature): 0x%02x
+        Last update: %s
+      Header length: %d
+      Record length: %d
+       Record count: %d
+ FieldName Type Len Dec
+""" % (self.signature, self.lastUpdate, self.headerLength,
+    self.recordLength, self.recordCount)
+        _rv += "\n".join(
+            ["%10s %4s %3s %3s" % _fld.fieldInfo() for _fld in self.fields]
+        )
+        return _rv
+
+    ## internal methods
+
+    def _addField(self, *defs):
+        """Internal variant of the `addField` method.
+
+        This method doesn't set `self.changed` field to True.
+
+        Return value is a length of the appended records.
+        Note: this method doesn't modify ``recordLength`` and
+        ``headerLength`` fields. Use `addField` instead of this
+        method if you don't exactly know what you're doing.
+
+        """
+        # insure we have dbf.DbfFieldDef instances first (instantiation
+        # from the tuple could raise an error, in such a case I don't
+        # wanna add any of the definitions -- all will be ignored)
+        _defs = []
+        _recordLength = 0
+        for _def in defs:
+            if isinstance(_def, fields.DbfFieldDef):
+                _obj = _def
+            else:
+                (_name, _type, _len, _dec) = (tuple(_def) + (None,) * 4)[:4]
+                _cls = fields.lookupFor(_type)
+                _obj = _cls(_name, _len, _dec,
+                    ignoreErrors=self._ignore_errors)
+            _recordLength += _obj.length
+            _defs.append(_obj)
+        # and now extend field definitions and
+        # update record length
+        self.fields += _defs
+        return _recordLength
+
+    ## interface methods
+
+    def addField(self, *defs):
+        """Add field definition to the header.
+
+        Examples:
+            dbfh.addField(
+                ("name", "C", 20),
+                dbf.DbfCharacterFieldDef("surname", 20),
+                dbf.DbfDateFieldDef("birthdate"),
+                ("member", "L"),
+            )
+            dbfh.addField(("price", "N", 5, 2))
+            dbfh.addField(dbf.DbfNumericFieldDef("origprice", 5, 2))
+
+        """
+        _oldLen = self.recordLength
+        self.recordLength += self._addField(*defs)
+        if not _oldLen:
+            self.recordLength += 1
+            # XXX: may be just use:
+            # self.recordeLength += self._addField(*defs) + bool(not _oldLen)
+        # recalculate headerLength
+        self.headerLength = 32 + (32 * len(self.fields)) + 1
+        self.changed = True
+
+    def write(self, stream):
+        """Encode and write header to the stream."""
+        stream.seek(0)
+        stream.write(self.toString())
+        stream.write("".join([_fld.toString() for _fld in self.fields]))
+        stream.write(chr(0x0D))   # cr at end of all hdr data
+        self.changed = False
+
+    def toString(self):
+        """Returned 32 chars length string with encoded header."""
+        return struct.pack("<4BI2H",
+            self.signature,
+            self.year - 1900,
+            self.month,
+            self.day,
+            self.recordCount,
+            self.headerLength,
+            self.recordLength) + "\0" * 20
+
+    def setCurrentDate(self):
+        """Update ``self.lastUpdate`` field with current date value."""
+        self.lastUpdate = datetime.date.today()
+
+    def __getitem__(self, item):
+        """Return a field definition by numeric index or name string"""
+        if isinstance(item, basestring):
+            _name = item.upper()
+            for _field in self.fields:
+                if _field.name == _name:
+                    return _field
+            else:
+                raise KeyError(item)
+        else:
+            # item must be field index
+            return self.fields[item]
+
+# vim: et sts=4 sw=4 :

tablib/packages/dbfpy/record.py

@@ -0,0 +1,262 @@
+"""DBF record definition.
+
+"""
+"""History (most recent first):
+11-feb-2007 [als]   __repr__: added special case for invalid field values
+10-feb-2007 [als]   added .rawFromStream()
+30-oct-2006 [als]   fix record length in .fromStream()
+04-jul-2006 [als]   added export declaration
+20-dec-2005 [yc]    DbfRecord.write() -> DbfRecord._write();
+                    added delete() method.
+16-dec-2005 [yc]    record definition moved from `dbf`.
+"""
+
+__version__ = "$Revision: 1.7 $"[11:-2]
+__date__ = "$Date: 2007/02/11 09:05:49 $"[7:-2]
+
+__all__ = ["DbfRecord"]
+
+from itertools import izip
+
+import utils
+
+class DbfRecord(object):
+    """DBF record.
+
+    Instances of this class shouldn't be created manualy,
+    use `dbf.Dbf.newRecord` instead.
+
+    Class implements mapping/sequence interface, so
+    fields could be accessed via their names or indexes
+    (names is a preffered way to access fields).
+
+    Hint:
+        Use `store` method to save modified record.
+
+    Examples:
+        Add new record to the database:
+            db = Dbf(filename)
+            rec = db.newRecord()
+            rec["FIELD1"] = value1
+            rec["FIELD2"] = value2
+            rec.store()
+        Or the same, but modify existed
+        (second in this case) record:
+            db = Dbf(filename)
+            rec = db[2]
+            rec["FIELD1"] = value1
+            rec["FIELD2"] = value2
+            rec.store()
+
+    """
+
+    __slots__ = "dbf", "index", "deleted", "fieldData"
+
+    ## creation and initialization
+
+    def __init__(self, dbf, index=None, deleted=False, data=None):
+        """Instance initialiation.
+
+        Arguments:
+            dbf:
+                A `Dbf.Dbf` instance this record belonogs to.
+            index:
+                An integer record index or None. If this value is
+                None, record will be appended to the DBF.
+            deleted:
+                Boolean flag indicating whether this record
+                is a deleted record.
+            data:
+                A sequence or None. This is a data of the fields.
+                If this argument is None, default values will be used.
+
+        """
+        self.dbf = dbf
+        # XXX: I'm not sure ``index`` is necessary
+        self.index = index
+        self.deleted = deleted
+        if data is None:
+            self.fieldData = [_fd.defaultValue for _fd in dbf.header.fields]
+        else:
+            self.fieldData = list(data)
+
+    # XXX: validate self.index before calculating position?
+    position = property(lambda self: self.dbf.header.headerLength + \
+        self.index * self.dbf.header.recordLength)
+
+    def rawFromStream(cls, dbf, index):
+        """Return raw record contents read from the stream.
+
+        Arguments:
+            dbf:
+                A `Dbf.Dbf` instance containing the record.
+            index:
+                Index of the record in the records' container.
+                This argument can't be None in this call.
+
+        Return value is a string containing record data in DBF format.
+
+        """
+        # XXX: may be write smth assuming, that current stream
+        # position is the required one? it could save some
+        # time required to calculate where to seek in the file
+        dbf.stream.seek(dbf.header.headerLength +
+            index * dbf.header.recordLength)
+        return dbf.stream.read(dbf.header.recordLength)
+    rawFromStream = classmethod(rawFromStream)
+
+    def fromStream(cls, dbf, index):
+        """Return a record read from the stream.
+
+        Arguments:
+            dbf:
+                A `Dbf.Dbf` instance new record should belong to.
+            index:
+                Index of the record in the records' container.
+                This argument can't be None in this call.
+
+        Return value is an instance of the current class.
+
+        """
+        return cls.fromString(dbf, cls.rawFromStream(dbf, index), index)
+    fromStream = classmethod(fromStream)
+
+    def fromString(cls, dbf, string, index=None):
+        """Return record read from the string object.
+
+        Arguments:
+            dbf:
+                A `Dbf.Dbf` instance new record should belong to.
+            string:
+                A string new record should be created from.
+            index:
+                Index of the record in the container. If this
+                argument is None, record will be appended.
+
+        Return value is an instance of the current class.
+
+        """
+        return cls(dbf, index, string[0]=="*",
+            [_fd.decodeFromRecord(string) for _fd in dbf.header.fields])
+    fromString = classmethod(fromString)
+
+    ## object representation
+
+    def __repr__(self):
+        _template = "%%%ds: %%s (%%s)" % max([len(_fld)
+            for _fld in self.dbf.fieldNames])
+        _rv = []
+        for _fld in self.dbf.fieldNames:
+            _val = self[_fld]
+            if _val is utils.INVALID_VALUE:
+                _rv.append(_template %
+                    (_fld, "None", "value cannot be decoded"))
+            else:
+                _rv.append(_template % (_fld, _val, type(_val)))
+        return "\n".join(_rv)
+
+    ## protected methods
+
+    def _write(self):
+        """Write data to the dbf stream.
+
+        Note:
+            This isn't a public method, it's better to
+            use 'store' instead publically.
+            Be design ``_write`` method should be called
+            only from the `Dbf` instance.
+
+
+        """
+        self._validateIndex(False)
+        self.dbf.stream.seek(self.position)
+        self.dbf.stream.write(self.toString())
+        # FIXME: may be move this write somewhere else?
+        # why we should check this condition for each record?
+        if self.index == len(self.dbf):
+            # this is the last record,
+            # we should write SUB (ASCII 26)
+            self.dbf.stream.write("\x1A")
+
+    ## utility methods
+
+    def _validateIndex(self, allowUndefined=True, checkRange=False):
+        """Valid ``self.index`` value.
+
+        If ``allowUndefined`` argument is True functions does nothing
+        in case of ``self.index`` pointing to None object.
+
+        """
+        if self.index is None:
+            if not allowUndefined:
+                raise ValueError("Index is undefined")
+        elif self.index < 0:
+            raise ValueError("Index can't be negative (%s)" % self.index)
+        elif checkRange and self.index <= self.dbf.header.recordCount:
+            raise ValueError("There are only %d records in the DBF" %
+                self.dbf.header.recordCount)
+
+    ## interface methods
+
+    def store(self):
+        """Store current record in the DBF.
+
+        If ``self.index`` is None, this record will be appended to the
+        records of the DBF this records belongs to; or replaced otherwise.
+
+        """
+        self._validateIndex()
+        if self.index is None:
+            self.index = len(self.dbf)
+            self.dbf.append(self)
+        else:
+            self.dbf[self.index] = self
+
+    def delete(self):
+        """Mark method as deleted."""
+        self.deleted = True
+
+    def toString(self):
+        """Return string packed record values."""
+        return "".join([" *"[self.deleted]] + [
+            _def.encodeValue(_dat)
+            for (_def, _dat) in izip(self.dbf.header.fields, self.fieldData)
+        ])
+
+    def asList(self):
+        """Return a flat list of fields.
+
+        Note:
+            Change of the list's values won't change
+            real values stored in this object.
+
+        """
+        return self.fieldData[:]
+
+    def asDict(self):
+        """Return a dictionary of fields.
+
+        Note:
+            Change of the dicts's values won't change
+            real values stored in this object.
+
+        """
+        return dict([_i for _i in izip(self.dbf.fieldNames, self.fieldData)])
+
+    def __getitem__(self, key):
+        """Return value by field name or field index."""
+        if isinstance(key, (long, int)):
+            # integer index of the field
+            return self.fieldData[key]
+        # assuming string field name
+        return self.fieldData[self.dbf.indexOfFieldName(key)]
+
+    def __setitem__(self, key, value):
+        """Set field value by integer index of the field or string name."""
+        if isinstance(key, (int, long)):
+            # integer index of the field
+            return self.fieldData[key]
+        # assuming string field name
+        self.fieldData[self.dbf.indexOfFieldName(key)] = value
+
+# vim: et sts=4 sw=4 :

tablib/packages/dbfpy/utils.py

@@ -0,0 +1,170 @@
+"""String utilities.
+
+TODO:
+  - allow strings in getDateTime routine;
+"""
+"""History (most recent first):
+11-feb-2007 [als]   added INVALID_VALUE
+10-feb-2007 [als]   allow date strings padded with spaces instead of zeroes
+20-dec-2005 [yc]    handle long objects in getDate/getDateTime
+16-dec-2005 [yc]    created from ``strutil`` module.
+"""
+
+__version__ = "$Revision: 1.4 $"[11:-2]
+__date__ = "$Date: 2007/02/11 08:57:17 $"[7:-2]
+
+import datetime
+import time
+
+
+def unzfill(str):
+    """Return a string without ASCII NULs.
+
+    This function searchers for the first NUL (ASCII 0) occurance
+    and truncates string till that position.
+
+    """
+    try:
+        return str[:str.index('\0')]
+    except ValueError:
+        return str
+
+
+def getDate(date=None):
+    """Return `datetime.date` instance.
+
+    Type of the ``date`` argument could be one of the following:
+        None:
+            use current date value;
+        datetime.date:
+            this value will be returned;
+        datetime.datetime:
+            the result of the date.date() will be returned;
+        string:
+            assuming "%Y%m%d" or "%y%m%dd" format;
+        number:
+            assuming it's a timestamp (returned for example
+            by the time.time() call;
+        sequence:
+            assuming (year, month, day, ...) sequence;
+
+    Additionaly, if ``date`` has callable ``ticks`` attribute,
+    it will be used and result of the called would be treated
+    as a timestamp value.
+
+    """
+    if date is None:
+        # use current value
+        return datetime.date.today()
+    if isinstance(date, datetime.date):
+        return date
+    if isinstance(date, datetime.datetime):
+        return date.date()
+    if isinstance(date, (int, long, float)):
+        # date is a timestamp
+        return datetime.date.fromtimestamp(date)
+    if isinstance(date, basestring):
+        date = date.replace(" ", "0")
+        if len(date) == 6:
+            # yymmdd
+            return datetime.date(*time.strptime(date, "%y%m%d")[:3])
+        # yyyymmdd
+        return datetime.date(*time.strptime(date, "%Y%m%d")[:3])
+    if hasattr(date, "__getitem__"):
+        # a sequence (assuming date/time tuple)
+        return datetime.date(*date[:3])
+    return datetime.date.fromtimestamp(date.ticks())
+
+
+def getDateTime(value=None):
+    """Return `datetime.datetime` instance.
+
+    Type of the ``value`` argument could be one of the following:
+        None:
+            use current date value;
+        datetime.date:
+            result will be converted to the `datetime.datetime` instance
+            using midnight;
+        datetime.datetime:
+            ``value`` will be returned as is;
+        string:
+            *** CURRENTLY NOT SUPPORTED ***;
+        number:
+            assuming it's a timestamp (returned for example
+            by the time.time() call;
+        sequence:
+            assuming (year, month, day, ...) sequence;
+
+    Additionaly, if ``value`` has callable ``ticks`` attribute,
+    it will be used and result of the called would be treated
+    as a timestamp value.
+
+    """
+    if value is None:
+        # use current value
+        return datetime.datetime.today()
+    if isinstance(value, datetime.datetime):
+        return value
+    if isinstance(value, datetime.date):
+        return datetime.datetime.fromordinal(value.toordinal())
+    if isinstance(value, (int, long, float)):
+        # value is a timestamp
+        return datetime.datetime.fromtimestamp(value)
+    if isinstance(value, basestring):
+        raise NotImplementedError("Strings aren't currently implemented")
+    if hasattr(value, "__getitem__"):
+        # a sequence (assuming date/time tuple)
+        return datetime.datetime(*tuple(value)[:6])
+    return datetime.datetime.fromtimestamp(value.ticks())
+
+
+class classproperty(property):
+    """Works in the same way as a ``property``, but for the classes."""
+
+    def __get__(self, obj, cls):
+        return self.fget(cls)
+
+
+class _InvalidValue(object):
+
+    """Value returned from DBF records when field validation fails
+
+    The value is not equal to anything except for itself
+    and equal to all empty values: None, 0, empty string etc.
+    In other words, invalid value is equal to None and not equal
+    to None at the same time.
+
+    This value yields zero upon explicit conversion to a number type,
+    empty string for string types, and False for boolean.
+
+    """
+
+    def __eq__(self, other):
+        return not other
+
+    def __ne__(self, other):
+        return not (other is self)
+
+    def __nonzero__(self):
+        return False
+
+    def __int__(self):
+        return 0
+    __long__ = __int__
+
+    def __float__(self):
+        return 0.0
+
+    def __str__(self):
+        return ""
+
+    def __unicode__(self):
+        return u""
+
+    def __repr__(self):
+        return "<INVALID>"
+
+# invalid value is a constant singleton
+INVALID_VALUE = _InvalidValue()
+
+# vim: set et sts=4 sw=4 :

tablib/packages/dbfpy3/dbf.py

@@ -0,0 +1,298 @@
+#! /usr/bin/env python
+"""DBF accessing helpers.
+
+FIXME: more documentation needed
+
+Examples:
+
+    Create new table, setup structure, add records:
+
+        dbf = Dbf(filename, new=True)
+        dbf.addField(
+            ("NAME", "C", 15),
+            ("SURNAME", "C", 25),
+            ("INITIALS", "C", 10),
+            ("BIRTHDATE", "D"),
+        )
+        for (n, s, i, b) in (
+            ("John", "Miller", "YC", (1980, 10, 11)),
+            ("Andy", "Larkin", "", (1980, 4, 11)),
+        ):
+            rec = dbf.newRecord()
+            rec["NAME"] = n
+            rec["SURNAME"] = s
+            rec["INITIALS"] = i
+            rec["BIRTHDATE"] = b
+            rec.store()
+        dbf.close()
+
+    Open existed dbf, read some data:
+
+        dbf = Dbf(filename, True)
+        for rec in dbf:
+            for fldName in dbf.fieldNames:
+                print '%s:\t %s (%s)' % (fldName, rec[fldName],
+                    type(rec[fldName]))
+            print
+        dbf.close()
+
+"""
+"""History (most recent first):
+11-feb-2007 [als]   export INVALID_VALUE;
+                    Dbf: added .ignoreErrors, .INVALID_VALUE
+04-jul-2006 [als]   added export declaration
+20-dec-2005 [yc]    removed fromStream and newDbf methods:
+                    use argument of __init__ call must be used instead;
+                    added class fields pointing to the header and
+                    record classes.
+17-dec-2005 [yc]    split to several modules; reimplemented
+13-dec-2005 [yc]    adapted to the changes of the `strutil` module.
+13-sep-2002 [als]   support FoxPro Timestamp datatype
+15-nov-1999 [jjk]   documentation updates, add demo
+24-aug-1998 [jjk]   add some encodeValue methods (not tested), other tweaks
+08-jun-1998 [jjk]   fix problems, add more features
+20-feb-1998 [jjk]   fix problems, add more features
+19-feb-1998 [jjk]   add create/write capabilities
+18-feb-1998 [jjk]   from dbfload.py
+"""
+
+__version__ = "$Revision: 1.7 $"[11:-2]
+__date__ = "$Date: 2007/02/11 09:23:13 $"[7:-2]
+__author__ = "Jeff Kunce <kuncej@mail.conservation.state.mo.us>"
+
+__all__ = ["Dbf"]
+
+from . import header
+from . import record
+from .utils import INVALID_VALUE
+
+
+class Dbf(object):
+    """DBF accessor.
+
+    FIXME:
+        docs and examples needed (dont' forget to tell
+        about problems adding new fields on the fly)
+
+    Implementation notes:
+        ``_new`` field is used to indicate whether this is
+        a new data table. `addField` could be used only for
+        the new tables! If at least one record was appended
+        to the table it's structure couldn't be changed.
+
+    """
+
+    __slots__ = ("name", "header", "stream",
+                 "_changed", "_new", "_ignore_errors")
+
+    HeaderClass = header.DbfHeader
+    RecordClass = record.DbfRecord
+    INVALID_VALUE = INVALID_VALUE
+
+    # initialization and creation helpers
+
+    def __init__(self, f, readOnly=False, new=False, ignoreErrors=False):
+        """Initialize instance.
+
+        Arguments:
+            f:
+                Filename or file-like object.
+            new:
+                True if new data table must be created. Assume
+                data table exists if this argument is False.
+            readOnly:
+                if ``f`` argument is a string file will
+                be opend in read-only mode; in other cases
+                this argument is ignored. This argument is ignored
+                even if ``new`` argument is True.
+            headerObj:
+                `header.DbfHeader` instance or None. If this argument
+                is None, new empty header will be used with the
+                all fields set by default.
+            ignoreErrors:
+                if set, failing field value conversion will return
+                ``INVALID_VALUE`` instead of raising conversion error.
+
+        """
+        if isinstance(f, str):
+            # a filename
+            self.name = f
+            if new:
+                # new table (table file must be
+                # created or opened and truncated)
+                self.stream = open(f, "w+b")
+            else:
+                # tabe file must exist
+                self.stream = open(f, ("r+b", "rb")[bool(readOnly)])
+        else:
+            # a stream
+            self.name = getattr(f, "name", "")
+            self.stream = f
+        if new:
+            # if this is a new table, header will be empty
+            self.header = self.HeaderClass()
+        else:
+            # or instantiated using stream
+            self.header = self.HeaderClass.fromStream(self.stream)
+        self.ignoreErrors = ignoreErrors
+        self._new = bool(new)
+        self._changed = False
+
+    # properties
+
+    closed = property(lambda self: self.stream.closed)
+    recordCount = property(lambda self: self.header.recordCount)
+    fieldNames = property(
+        lambda self: [_fld.name for _fld in self.header.fields])
+    fieldDefs = property(lambda self: self.header.fields)
+    changed = property(lambda self: self._changed or self.header.changed)
+
+    def ignoreErrors(self, value):
+        """Update `ignoreErrors` flag on the header object and self"""
+        self.header.ignoreErrors = self._ignore_errors = bool(value)
+
+    ignoreErrors = property(
+        lambda self: self._ignore_errors,
+        ignoreErrors,
+        doc="""Error processing mode for DBF field value conversion
+
+        if set, failing field value conversion will return
+        ``INVALID_VALUE`` instead of raising conversion error.
+
+        """)
+
+    # protected methods
+
+    def _fixIndex(self, index):
+        """Return fixed index.
+
+        This method fails if index isn't a numeric object
+        (long or int). Or index isn't in a valid range
+        (less or equal to the number of records in the db).
+
+        If ``index`` is a negative number, it will be
+        treated as a negative indexes for list objects.
+
+        Return:
+            Return value is numeric object maning valid index.
+
+        """
+        if not isinstance(index, int):
+            raise TypeError("Index must be a numeric object")
+        if index < 0:
+            # index from the right side
+            # fix it to the left-side index
+            index += len(self) + 1
+        if index >= len(self):
+            raise IndexError("Record index out of range")
+        return index
+
+    # iterface methods
+
+    def close(self):
+        self.flush()
+        self.stream.close()
+
+    def flush(self):
+        """Flush data to the associated stream."""
+        if self.changed:
+            self.header.setCurrentDate()
+            self.header.write(self.stream)
+            self.stream.flush()
+            self._changed = False
+
+    def indexOfFieldName(self, name):
+        """Index of field named ``name``."""
+        # FIXME: move this to header class
+        names = [f.name for f in self.header.fields]
+        return names.index(name.upper())
+
+    def newRecord(self):
+        """Return new record, which belong to this table."""
+        return self.RecordClass(self)
+
+    def append(self, record):
+        """Append ``record`` to the database."""
+        record.index = self.header.recordCount
+        record._write()
+        self.header.recordCount += 1
+        self._changed = True
+        self._new = False
+
+    def addField(self, *defs):
+        """Add field definitions.
+
+        For more information see `header.DbfHeader.addField`.
+
+        """
+        if self._new:
+            self.header.addField(*defs)
+        else:
+            raise TypeError("At least one record was added, "
+                            "structure can't be changed")
+
+    # 'magic' methods (representation and sequence interface)
+
+    def __repr__(self):
+        return "Dbf stream '%s'\n" % self.stream + repr(self.header)
+
+    def __len__(self):
+        """Return number of records."""
+        return self.recordCount
+
+    def __getitem__(self, index):
+        """Return `DbfRecord` instance."""
+        return self.RecordClass.fromStream(self, self._fixIndex(index))
+
+    def __setitem__(self, index, record):
+        """Write `DbfRecord` instance to the stream."""
+        record.index = self._fixIndex(index)
+        record._write()
+        self._changed = True
+        self._new = False
+
+        # def __del__(self):
+        #    """Flush stream upon deletion of the object."""
+        #    self.flush()
+
+
+def demo_read(filename):
+    _dbf = Dbf(filename, True)
+    for _rec in _dbf:
+        print()
+        print(repr(_rec))
+    _dbf.close()
+
+
+def demo_create(filename):
+    _dbf = Dbf(filename, new=True)
+    _dbf.addField(
+        ("NAME", "C", 15),
+        ("SURNAME", "C", 25),
+        ("INITIALS", "C", 10),
+        ("BIRTHDATE", "D"),
+    )
+    for (_n, _s, _i, _b) in (
+            ("John", "Miller", "YC", (1981, 1, 2)),
+            ("Andy", "Larkin", "AL", (1982, 3, 4)),
+            ("Bill", "Clinth", "", (1983, 5, 6)),
+            ("Bobb", "McNail", "", (1984, 7, 8)),
+    ):
+        _rec = _dbf.newRecord()
+        _rec["NAME"] = _n
+        _rec["SURNAME"] = _s
+        _rec["INITIALS"] = _i
+        _rec["BIRTHDATE"] = _b
+        _rec.store()
+    print(repr(_dbf))
+    _dbf.close()
+
+
+if __name__ == '__main__':
+    import sys
+
+    _name = len(sys.argv) > 1 and sys.argv[1] or "county.dbf"
+    demo_create(_name)
+    demo_read(_name)
+
+# vim: set et sw=4 sts=4 :

tablib/packages/dbfpy3/dbfnew.py

@@ -0,0 +1,183 @@
+#!/usr/bin/python
+""".DBF creation helpers.
+
+Note: this is a legacy interface.  New code should use Dbf class
+    for table creation (see examples in dbf.py)
+
+TODO:
+  - handle Memo fields.
+  - check length of the fields accoring to the
+    `http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
+
+"""
+"""History (most recent first)
+04-jul-2006 [als]   added export declaration;
+                    updated for dbfpy 2.0
+15-dec-2005 [yc]    define dbf_new.__slots__
+14-dec-2005 [yc]    added vim modeline; retab'd; added doc-strings;
+                    dbf_new now is a new class (inherited from object)
+??-jun-2000 [--]    added by Hans Fiby
+"""
+
+__version__ = "$Revision: 1.4 $"[11:-2]
+__date__ = "$Date: 2006/07/04 08:18:18 $"[7:-2]
+
+__all__ = ["dbf_new"]
+
+from .dbf import *
+from .fields import *
+from .header import *
+from .record import *
+
+
+class _FieldDefinition(object):
+    """Field definition.
+
+    This is a simple structure, which contains ``name``, ``type``,
+    ``len``, ``dec`` and ``cls`` fields.
+
+    Objects also implement get/setitem magic functions, so fields
+    could be accessed via sequence iterface, where 'name' has
+    index 0, 'type' index 1, 'len' index 2, 'dec' index 3 and
+    'cls' could be located at index 4.
+
+    """
+
+    __slots__ = "name", "type", "len", "dec", "cls"
+
+    # WARNING: be attentive - dictionaries are mutable!
+    FLD_TYPES = {
+        # type: (cls, len)
+        "C": (DbfCharacterFieldDef, None),
+        "N": (DbfNumericFieldDef, None),
+        "L": (DbfLogicalFieldDef, 1),
+        # FIXME: support memos
+        # "M": (DbfMemoFieldDef),
+        "D": (DbfDateFieldDef, 8),
+        # FIXME: I'm not sure length should be 14 characters!
+        # but temporary I use it, cuz date is 8 characters
+        # and time 6 (hhmmss)
+        "T": (DbfDateTimeFieldDef, 14),
+    }
+
+    def __init__(self, name, type, len=None, dec=0):
+        _cls, _len = self.FLD_TYPES[type]
+        if _len is None:
+            if len is None:
+                raise ValueError("Field length must be defined")
+            _len = len
+        self.name = name
+        self.type = type
+        self.len = _len
+        self.dec = dec
+        self.cls = _cls
+
+    def getDbfField(self):
+        "Return `DbfFieldDef` instance from the current definition."
+        return self.cls(self.name, self.len, self.dec)
+
+    def appendToHeader(self, dbfh):
+        """Create a `DbfFieldDef` instance and append it to the dbf header.
+
+        Arguments:
+            dbfh: `DbfHeader` instance.
+
+        """
+        _dbff = self.getDbfField()
+        dbfh.addField(_dbff)
+
+
+class dbf_new(object):
+    """New .DBF creation helper.
+
+    Example Usage:
+
+        dbfn = dbf_new()
+        dbfn.add_field("name",'C',80)
+        dbfn.add_field("price",'N',10,2)
+        dbfn.add_field("date",'D',8)
+        dbfn.write("tst.dbf")
+
+    Note:
+        This module cannot handle Memo-fields,
+        they are special.
+
+    """
+
+    __slots__ = ("fields",)
+
+    FieldDefinitionClass = _FieldDefinition
+
+    def __init__(self):
+        self.fields = []
+
+    def add_field(self, name, typ, len, dec=0):
+        """Add field definition.
+
+        Arguments:
+            name:
+                field name (str object). field name must not
+                contain ASCII NULs and it's length shouldn't
+                exceed 10 characters.
+            typ:
+                type of the field. this must be a single character
+                from the "CNLMDT" set meaning character, numeric,
+                logical, memo, date and date/time respectively.
+            len:
+                length of the field. this argument is used only for
+                the character and numeric fields. all other fields
+                have fixed length.
+                FIXME: use None as a default for this argument?
+            dec:
+                decimal precision. used only for the numric fields.
+
+        """
+        self.fields.append(self.FieldDefinitionClass(name, typ, len, dec))
+
+    def write(self, filename):
+        """Create empty .DBF file using current structure."""
+        _dbfh = DbfHeader()
+        _dbfh.setCurrentDate()
+        for _fldDef in self.fields:
+            _fldDef.appendToHeader(_dbfh)
+
+        _dbfStream = open(filename, "wb")
+        _dbfh.write(_dbfStream)
+        _dbfStream.close()
+
+
+if __name__ == '__main__':
+    # create a new DBF-File
+    dbfn = dbf_new()
+    dbfn.add_field("name", 'C', 80)
+    dbfn.add_field("price", 'N', 10, 2)
+    dbfn.add_field("date", 'D', 8)
+    dbfn.write("tst.dbf")
+    # test new dbf
+    print("*** created tst.dbf: ***")
+    dbft = Dbf('tst.dbf', readOnly=0)
+    print(repr(dbft))
+    # add a record
+    rec = DbfRecord(dbft)
+    rec['name'] = 'something'
+    rec['price'] = 10.5
+    rec['date'] = (2000, 1, 12)
+    rec.store()
+    # add another record
+    rec = DbfRecord(dbft)
+    rec['name'] = 'foo and bar'
+    rec['price'] = 12234
+    rec['date'] = (1992, 7, 15)
+    rec.store()
+
+    # show the records
+    print("*** inserted 2 records into tst.dbf: ***")
+    print(repr(dbft))
+    for i1 in range(len(dbft)):
+        rec = dbft[i1]
+        for fldName in dbft.fieldNames:
+            print('%s:\t %s' % (fldName, rec[fldName]))
+        print()
+    dbft.close()
+
+# vim: set et sts=4 sw=4 :

tablib/packages/dbfpy3/fields.py

@@ -0,0 +1,467 @@
+"""DBF fields definitions.
+
+TODO:
+  - make memos work
+"""
+"""History (most recent first):
+26-may-2009 [als]   DbfNumericFieldDef.decodeValue: strip zero bytes
+05-feb-2009 [als]   DbfDateFieldDef.encodeValue: empty arg produces empty date
+16-sep-2008 [als]   DbfNumericFieldDef decoding looks for decimal point
+                    in the value to select float or integer return type
+13-mar-2008 [als]   check field name length in constructor
+11-feb-2007 [als]   handle value conversion errors
+10-feb-2007 [als]   DbfFieldDef: added .rawFromRecord()
+01-dec-2006 [als]   Timestamp columns use None for empty values
+31-oct-2006 [als]   support field types 'F' (float), 'I' (integer)
+                    and 'Y' (currency);
+                    automate export and registration of field classes
+04-jul-2006 [als]   added export declaration
+10-mar-2006 [als]   decode empty values for Date and Logical fields;
+                    show field name in errors
+10-mar-2006 [als]   fix Numeric value decoding: according to spec,
+                    value always is string representation of the number;
+                    ensure that encoded Numeric value fits into the field
+20-dec-2005 [yc]    use field names in upper case
+15-dec-2005 [yc]    field definitions moved from `dbf`.
+"""
+
+__version__ = "$Revision: 1.14 $"[11:-2]
+__date__ = "$Date: 2009/05/26 05:16:51 $"[7:-2]
+
+__all__ = ["lookupFor",] # field classes added at the end of the module
+
+import datetime
+import struct
+import sys
+
+from . import utils
+
+## abstract definitions
+
+class DbfFieldDef(object):
+    """Abstract field definition.
+
+    Child classes must override ``type`` class attribute to provide datatype
+    infromation of the field definition. For more info about types visit
+    `http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
+
+    Also child classes must override ``defaultValue`` field to provide
+    default value for the field value.
+
+    If child class has fixed length ``length`` class attribute must be
+    overriden and set to the valid value. None value means, that field
+    isn't of fixed length.
+
+    Note: ``name`` field must not be changed after instantiation.
+
+    """
+
+
+    __slots__ = ("name", "decimalCount",
+        "start", "end", "ignoreErrors")
+
+    # length of the field, None in case of variable-length field,
+    # or a number if this field is a fixed-length field
+    length = None
+
+    # field type. for more information about fields types visit
+    # `http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
+    # must be overriden in child classes
+    typeCode = None
+
+    # default value for the field. this field must be
+    # overriden in child classes
+    defaultValue = None
+
+    def __init__(self, name, length=None, decimalCount=None,
+        start=None, stop=None, ignoreErrors=False,
+    ):
+        """Initialize instance."""
+        assert self.typeCode is not None, "Type code must be overriden"
+        assert self.defaultValue is not None, "Default value must be overriden"
+        ## fix arguments
+        if len(name) >10:
+            raise ValueError("Field name \"%s\" is too long" % name)
+        name = str(name).upper()
+        if self.__class__.length is None:
+            if length is None:
+                raise ValueError("[%s] Length isn't specified" % name)
+            length = int(length)
+            if length <= 0:
+                raise ValueError("[%s] Length must be a positive integer"
+                    % name)
+        else:
+            length = self.length
+        if decimalCount is None:
+            decimalCount = 0
+        ## set fields
+        self.name = name
+        # FIXME: validate length according to the specification at
+        # http://www.clicketyclick.dk/databases/xbase/format/data_types.html
+        self.length = length
+        self.decimalCount = decimalCount
+        self.ignoreErrors = ignoreErrors
+        self.start = start
+        self.end = stop
+
+    def __cmp__(self, other):
+        return cmp(self.name, str(other).upper())
+
+    def __hash__(self):
+        return hash(self.name)
+
+    def fromString(cls, string, start, ignoreErrors=False):
+        """Decode dbf field definition from the string data.
+
+        Arguments:
+            string:
+                a string, dbf definition is decoded from. length of
+                the string must be 32 bytes.
+            start:
+                position in the database file.
+            ignoreErrors:
+                initial error processing mode for the new field (boolean)
+
+        """
+        assert len(string) == 32
+        _length = string[16]
+        return cls(utils.unzfill(string)[:11].decode('utf-8'), _length,
+            string[17], start, start + _length, ignoreErrors=ignoreErrors)
+    fromString = classmethod(fromString)
+
+    def toString(self):
+        """Return encoded field definition.
+
+        Return:
+            Return value is a string object containing encoded
+            definition of this field.
+
+        """
+        if sys.version_info < (2, 4):
+            # earlier versions did not support padding character
+            _name = self.name[:11] + "\0" * (11 - len(self.name))
+        else:
+            _name = self.name.ljust(11, '\0')
+        return (
+            _name +
+            self.typeCode +
+            #data address
+            chr(0) * 4 +
+            chr(self.length) +
+            chr(self.decimalCount) +
+            chr(0) * 14
+        )
+
+    def __repr__(self):
+        return "%-10s %1s %3d %3d" % self.fieldInfo()
+
+    def fieldInfo(self):
+        """Return field information.
+
+        Return:
+            Return value is a (name, type, length, decimals) tuple.
+
+        """
+        return (self.name, self.typeCode, self.length, self.decimalCount)
+
+    def rawFromRecord(self, record):
+        """Return a "raw" field value from the record string."""
+        return record[self.start:self.end]
+
+    def decodeFromRecord(self, record):
+        """Return decoded field value from the record string."""
+        try:
+            return self.decodeValue(self.rawFromRecord(record))
+        except:
+            if self.ignoreErrors:
+                return utils.INVALID_VALUE
+            else:
+                raise
+
+    def decodeValue(self, value):
+        """Return decoded value from string value.
+
+        This method shouldn't be used publicly. It's called from the
+        `decodeFromRecord` method.
+
+        This is an abstract method and it must be overridden in child classes.
+        """
+        raise NotImplementedError
+
+    def encodeValue(self, value):
+        """Return str object containing encoded field value.
+
+        This is an abstract method and it must be overriden in child classes.
+        """
+        raise NotImplementedError
+
+## real classes
+
+class DbfCharacterFieldDef(DbfFieldDef):
+    """Definition of the character field."""
+
+    typeCode = "C"
+    defaultValue = b''
+
+    def decodeValue(self, value):
+        """Return string object.
+
+        Return value is a ``value`` argument with stripped right spaces.
+
+        """
+        return value.rstrip(b' ').decode('utf-8')
+
+    def encodeValue(self, value):
+        """Return raw data string encoded from a ``value``."""
+        return str(value)[:self.length].ljust(self.length)
+
+
+class DbfNumericFieldDef(DbfFieldDef):
+    """Definition of the numeric field."""
+
+    typeCode = "N"
+    # XXX: now I'm not sure it was a good idea to make a class field
+    # `defaultValue` instead of a generic method as it was implemented
+    # previously -- it's ok with all types except number, cuz
+    # if self.decimalCount is 0, we should return 0 and 0.0 otherwise.
+    defaultValue = 0
+
+    def decodeValue(self, value):
+        """Return a number decoded from ``value``.
+
+        If decimals is zero, value will be decoded as an integer;
+        or as a float otherwise.
+
+        Return:
+            Return value is a int (long) or float instance.
+
+        """
+        value = value.strip(b' \0')
+        if b'.' in value:
+            # a float (has decimal separator)
+            return float(value)
+        elif value:
+            # must be an integer
+            return int(value)
+        else:
+            return 0
+
+    def encodeValue(self, value):
+        """Return string containing encoded ``value``."""
+        _rv = ("%*.*f" % (self.length, self.decimalCount, value))
+        if len(_rv) > self.length:
+            _ppos = _rv.find(".")
+            if 0 <= _ppos <= self.length:
+                _rv = _rv[:self.length]
+            else:
+                raise ValueError("[%s] Numeric overflow: %s (field width: %i)"
+                    % (self.name, _rv, self.length))
+        return _rv
+
+class DbfFloatFieldDef(DbfNumericFieldDef):
+    """Definition of the float field - same as numeric."""
+
+    typeCode = "F"
+
+class DbfIntegerFieldDef(DbfFieldDef):
+    """Definition of the integer field."""
+
+    typeCode = "I"
+    length = 4
+    defaultValue = 0
+
+    def decodeValue(self, value):
+        """Return an integer number decoded from ``value``."""
+        return struct.unpack("<i", value)[0]
+
+    def encodeValue(self, value):
+        """Return string containing encoded ``value``."""
+        return struct.pack("<i", int(value))
+
+class DbfCurrencyFieldDef(DbfFieldDef):
+    """Definition of the currency field."""
+
+    typeCode = "Y"
+    length = 8
+    defaultValue = 0.0
+
+    def decodeValue(self, value):
+        """Return float number decoded from ``value``."""
+        return struct.unpack("<q", value)[0] / 10000.
+
+    def encodeValue(self, value):
+        """Return string containing encoded ``value``."""
+        return struct.pack("<q", round(value * 10000))
+
+class DbfLogicalFieldDef(DbfFieldDef):
+    """Definition of the logical field."""
+
+    typeCode = "L"
+    defaultValue = -1
+    length = 1
+
+    def decodeValue(self, value):
+        """Return True, False or -1 decoded from ``value``."""
+        # Note: value always is 1-char string
+        if value == "?":
+            return -1
+        if value in "NnFf ":
+            return False
+        if value in "YyTt":
+            return True
+        raise ValueError("[%s] Invalid logical value %r" % (self.name, value))
+
+    def encodeValue(self, value):
+        """Return a character from the "TF?" set.
+
+        Return:
+            Return value is "T" if ``value`` is True
+            "?" if value is -1 or False otherwise.
+
+        """
+        if value is True:
+            return "T"
+        if value == -1:
+            return "?"
+        return "F"
+
+
+class DbfMemoFieldDef(DbfFieldDef):
+    """Definition of the memo field.
+
+    Note: memos aren't currenly completely supported.
+
+    """
+
+    typeCode = "M"
+    defaultValue = " " * 10
+    length = 10
+
+    def decodeValue(self, value):
+        """Return int .dbt block number decoded from the string object."""
+        #return int(value)
+        raise NotImplementedError
+
+    def encodeValue(self, value):
+        """Return raw data string encoded from a ``value``.
+
+        Note: this is an internal method.
+
+        """
+        #return str(value)[:self.length].ljust(self.length)
+        raise NotImplementedError
+
+
+class DbfDateFieldDef(DbfFieldDef):
+    """Definition of the date field."""
+
+    typeCode = "D"
+    defaultValue = utils.classproperty(lambda cls: datetime.date.today())
+    # "yyyymmdd" gives us 8 characters
+    length = 8
+
+    def decodeValue(self, value):
+        """Return a ``datetime.date`` instance decoded from ``value``."""
+        if value.strip():
+            return utils.getDate(value)
+        else:
+            return None
+
+    def encodeValue(self, value):
+        """Return a string-encoded value.
+
+        ``value`` argument should be a value suitable for the
+        `utils.getDate` call.
+
+        Return:
+            Return value is a string in format "yyyymmdd".
+
+        """
+        if value:
+            return utils.getDate(value).strftime("%Y%m%d")
+        else:
+            return " " * self.length
+
+
+class DbfDateTimeFieldDef(DbfFieldDef):
+    """Definition of the timestamp field."""
+
+    # a difference between JDN (Julian Day Number)
+    # and GDN (Gregorian Day Number). note, that GDN < JDN
+    JDN_GDN_DIFF = 1721425
+    typeCode = "T"
+    defaultValue = utils.classproperty(lambda cls: datetime.datetime.now())
+    # two 32-bits integers representing JDN and amount of
+    # milliseconds respectively gives us 8 bytes.
+    # note, that values must be encoded in LE byteorder.
+    length = 8
+
+    def decodeValue(self, value):
+        """Return a `datetime.datetime` instance."""
+        assert len(value) == self.length
+        # LE byteorder
+        _jdn, _msecs = struct.unpack("<2I", value)
+        if _jdn >= 1:
+            _rv = datetime.datetime.fromordinal(_jdn - self.JDN_GDN_DIFF)
+            _rv += datetime.timedelta(0, _msecs / 1000.0)
+        else:
+            # empty date
+            _rv = None
+        return _rv
+
+    def encodeValue(self, value):
+        """Return a string-encoded ``value``."""
+        if value:
+            value = utils.getDateTime(value)
+            # LE byteorder
+            _rv = struct.pack("<2I", value.toordinal() + self.JDN_GDN_DIFF,
+                (value.hour * 3600 + value.minute * 60 + value.second) * 1000)
+        else:
+            _rv = "\0" * self.length
+        assert len(_rv) == self.length
+        return _rv
+
+
+_fieldsRegistry = {}
+
+def registerField(fieldCls):
+    """Register field definition class.
+
+    ``fieldCls`` should be subclass of the `DbfFieldDef`.
+
+    Use `lookupFor` to retrieve field definition class
+    by the type code.
+
+    """
+    assert fieldCls.typeCode is not None, "Type code isn't defined"
+    # XXX: use fieldCls.typeCode.upper()? in case of any decign
+    # don't forget to look to the same comment in ``lookupFor`` method
+    _fieldsRegistry[fieldCls.typeCode] = fieldCls
+
+
+def lookupFor(typeCode):
+    """Return field definition class for the given type code.
+
+    ``typeCode`` must be a single character. That type should be
+    previously registered.
+
+    Use `registerField` to register new field class.
+
+    Return:
+        Return value is a subclass of the `DbfFieldDef`.
+
+    """
+    # XXX: use typeCode.upper()? in case of any decign don't
+    # forget to look to the same comment in ``registerField``
+    return _fieldsRegistry[chr(typeCode)]
+
+## register generic types
+
+for (_name, _val) in list(globals().items()):
+    if isinstance(_val, type) and issubclass(_val, DbfFieldDef) \
+    and (_name != "DbfFieldDef"):
+        __all__.append(_name)
+        registerField(_val)
+del _name, _val
+
+# vim: et sts=4 sw=4 :

tablib/packages/dbfpy3/header.py

@@ -0,0 +1,273 @@
+"""DBF header definition.
+
+TODO:
+  - handle encoding of the character fields
+    (encoding information stored in the DBF header)
+
+"""
+"""History (most recent first):
+16-sep-2010 [als]   fromStream: fix century of the last update field
+11-feb-2007 [als]   added .ignoreErrors
+10-feb-2007 [als]   added __getitem__: return field definitions
+                    by field name or field number (zero-based)
+04-jul-2006 [als]   added export declaration
+15-dec-2005 [yc]    created
+"""
+
+__version__ = "$Revision: 1.6 $"[11:-2]
+__date__ = "$Date: 2010/09/16 05:06:39 $"[7:-2]
+
+__all__ = ["DbfHeader"]
+
+import io
+import datetime
+import struct
+import time
+import sys
+
+from . import fields
+from .utils import getDate
+
+
+class DbfHeader(object):
+    """Dbf header definition.
+
+    For more information about dbf header format visit
+    `http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_STRUCT`
+
+    Examples:
+        Create an empty dbf header and add some field definitions:
+            dbfh = DbfHeader()
+            dbfh.addField(("name", "C", 10))
+            dbfh.addField(("date", "D"))
+            dbfh.addField(DbfNumericFieldDef("price", 5, 2))
+        Create a dbf header with field definitions:
+            dbfh = DbfHeader([
+                ("name", "C", 10),
+                ("date", "D"),
+                DbfNumericFieldDef("price", 5, 2),
+            ])
+
+    """
+
+    __slots__ = ("signature", "fields", "lastUpdate", "recordLength",
+        "recordCount", "headerLength", "changed", "_ignore_errors")
+
+    ## instance construction and initialization methods
+
+    def __init__(self, fields=None, headerLength=0, recordLength=0,
+        recordCount=0, signature=0x03, lastUpdate=None, ignoreErrors=False,
+    ):
+        """Initialize instance.
+
+        Arguments:
+            fields:
+                a list of field definitions;
+            recordLength:
+                size of the records;
+            headerLength:
+                size of the header;
+            recordCount:
+                number of records stored in DBF;
+            signature:
+                version number (aka signature). using 0x03 as a default meaning
+                "File without DBT". for more information about this field visit
+                ``http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_NOTE_1_TARGET``
+            lastUpdate:
+                date of the DBF's update. this could be a string ('yymmdd' or
+                'yyyymmdd'), timestamp (int or float), datetime/date value,
+                a sequence (assuming (yyyy, mm, dd, ...)) or an object having
+                callable ``ticks`` field.
+            ignoreErrors:
+                error processing mode for DBF fields (boolean)
+
+        """
+        self.signature = signature
+        if fields is None:
+            self.fields = []
+        else:
+            self.fields = list(fields)
+        self.lastUpdate = getDate(lastUpdate)
+        self.recordLength = recordLength
+        self.headerLength = headerLength
+        self.recordCount = recordCount
+        self.ignoreErrors = ignoreErrors
+        # XXX: I'm not sure this is safe to
+        # initialize `self.changed` in this way
+        self.changed = bool(self.fields)
+
+    # @classmethod
+    def fromString(cls, string):
+        """Return header instance from the string object."""
+        return cls.fromStream(io.StringIO(str(string)))
+    fromString = classmethod(fromString)
+
+    # @classmethod
+    def fromStream(cls, stream):
+        """Return header object from the stream."""
+        stream.seek(0)
+        first_32 = stream.read(32)
+        if type(first_32) != bytes:
+            _data = bytes(first_32, sys.getfilesystemencoding())
+        _data = first_32
+        (_cnt, _hdrLen, _recLen) = struct.unpack("<I2H", _data[4:12])
+        #reserved = _data[12:32]
+        _year = _data[1]
+        if _year < 80:
+            # dBase II started at 1980.  It is quite unlikely
+            # that actual last update date is before that year.
+            _year += 2000
+        else:
+            _year += 1900
+        ## create header object
+        _obj = cls(None, _hdrLen, _recLen, _cnt, _data[0],
+            (_year, _data[2], _data[3]))
+        ## append field definitions
+        # position 0 is for the deletion flag
+        _pos = 1
+        _data = stream.read(1)
+        while _data != b'\r':
+            _data += stream.read(31)
+            _fld = fields.lookupFor(_data[11]).fromString(_data, _pos)
+            _obj._addField(_fld)
+            _pos = _fld.end
+            _data = stream.read(1)
+        return _obj
+    fromStream = classmethod(fromStream)
+
+    ## properties
+
+    year = property(lambda self: self.lastUpdate.year)
+    month = property(lambda self: self.lastUpdate.month)
+    day = property(lambda self: self.lastUpdate.day)
+
+    def ignoreErrors(self, value):
+        """Update `ignoreErrors` flag on self and all fields"""
+        self._ignore_errors = value = bool(value)
+        for _field in self.fields:
+            _field.ignoreErrors = value
+    ignoreErrors = property(
+        lambda self: self._ignore_errors,
+        ignoreErrors,
+        doc="""Error processing mode for DBF field value conversion
+
+        if set, failing field value conversion will return
+        ``INVALID_VALUE`` instead of raising conversion error.
+
+        """)
+
+    ## object representation
+
+    def __repr__(self):
+        _rv = """\
+Version (signature): 0x%02x
+        Last update: %s
+      Header length: %d
+      Record length: %d
+       Record count: %d
+ FieldName Type Len Dec
+""" % (self.signature, self.lastUpdate, self.headerLength,
+    self.recordLength, self.recordCount)
+        _rv += "\n".join(
+            ["%10s %4s %3s %3s" % _fld.fieldInfo() for _fld in self.fields]
+        )
+        return _rv
+
+    ## internal methods
+
+    def _addField(self, *defs):
+        """Internal variant of the `addField` method.
+
+        This method doesn't set `self.changed` field to True.
+
+        Return value is a length of the appended records.
+        Note: this method doesn't modify ``recordLength`` and
+        ``headerLength`` fields. Use `addField` instead of this
+        method if you don't exactly know what you're doing.
+
+        """
+        # insure we have dbf.DbfFieldDef instances first (instantiation
+        # from the tuple could raise an error, in such a case I don't
+        # wanna add any of the definitions -- all will be ignored)
+        _defs = []
+        _recordLength = 0
+        for _def in defs:
+            if isinstance(_def, fields.DbfFieldDef):
+                _obj = _def
+            else:
+                (_name, _type, _len, _dec) = (tuple(_def) + (None,) * 4)[:4]
+                _cls = fields.lookupFor(_type)
+                _obj = _cls(_name, _len, _dec,
+                    ignoreErrors=self._ignore_errors)
+            _recordLength += _obj.length
+            _defs.append(_obj)
+        # and now extend field definitions and
+        # update record length
+        self.fields += _defs
+        return _recordLength
+
+    ## interface methods
+
+    def addField(self, *defs):
+        """Add field definition to the header.
+
+        Examples:
+            dbfh.addField(
+                ("name", "C", 20),
+                dbf.DbfCharacterFieldDef("surname", 20),
+                dbf.DbfDateFieldDef("birthdate"),
+                ("member", "L"),
+            )
+            dbfh.addField(("price", "N", 5, 2))
+            dbfh.addField(dbf.DbfNumericFieldDef("origprice", 5, 2))
+
+        """
+        _oldLen = self.recordLength
+        self.recordLength += self._addField(*defs)
+        if not _oldLen:
+            self.recordLength += 1
+            # XXX: may be just use:
+            # self.recordeLength += self._addField(*defs) + bool(not _oldLen)
+        # recalculate headerLength
+        self.headerLength = 32 + (32 * len(self.fields)) + 1
+        self.changed = True
+
+    def write(self, stream):
+        """Encode and write header to the stream."""
+        stream.seek(0)
+        stream.write(self.toString())
+        fields = [_fld.toString() for _fld in self.fields]
+        stream.write(''.join(fields).encode(sys.getfilesystemencoding()))
+        stream.write(b'\x0D')   # cr at end of all header data
+        self.changed = False
+
+    def toString(self):
+        """Returned 32 chars length string with encoded header."""
+        return struct.pack("<4BI2H",
+            self.signature,
+            self.year - 1900,
+            self.month,
+            self.day,
+            self.recordCount,
+            self.headerLength,
+            self.recordLength) + (b'\x00' * 20)
+        #TODO: figure out if bytes(utf-8) is correct here.
+
+    def setCurrentDate(self):
+        """Update ``self.lastUpdate`` field with current date value."""
+        self.lastUpdate = datetime.date.today()
+
+    def __getitem__(self, item):
+        """Return a field definition by numeric index or name string"""
+        if isinstance(item, str):
+            _name = item.upper()
+            for _field in self.fields:
+                if _field.name == _name:
+                    return _field
+            else:
+                raise KeyError(item)
+        else:
+            # item must be field index
+            return self.fields[item]
+
+# vim: et sts=4 sw=4 :

tablib/packages/dbfpy3/record.py

@@ -0,0 +1,266 @@
+"""DBF record definition.
+
+"""
+"""History (most recent first):
+11-feb-2007 [als]   __repr__: added special case for invalid field values
+10-feb-2007 [als]   added .rawFromStream()
+30-oct-2006 [als]   fix record length in .fromStream()
+04-jul-2006 [als]   added export declaration
+20-dec-2005 [yc]    DbfRecord.write() -> DbfRecord._write();
+                    added delete() method.
+16-dec-2005 [yc]    record definition moved from `dbf`.
+"""
+
+__version__ = "$Revision: 1.7 $"[11:-2]
+__date__ = "$Date: 2007/02/11 09:05:49 $"[7:-2]
+
+__all__ = ["DbfRecord"]
+
+import sys
+
+from . import utils
+
+class DbfRecord(object):
+    """DBF record.
+
+    Instances of this class shouldn't be created manualy,
+    use `dbf.Dbf.newRecord` instead.
+
+    Class implements mapping/sequence interface, so
+    fields could be accessed via their names or indexes
+    (names is a preffered way to access fields).
+
+    Hint:
+        Use `store` method to save modified record.
+
+    Examples:
+        Add new record to the database:
+            db = Dbf(filename)
+            rec = db.newRecord()
+            rec["FIELD1"] = value1
+            rec["FIELD2"] = value2
+            rec.store()
+        Or the same, but modify existed
+        (second in this case) record:
+            db = Dbf(filename)
+            rec = db[2]
+            rec["FIELD1"] = value1
+            rec["FIELD2"] = value2
+            rec.store()
+
+    """
+
+    __slots__ = "dbf", "index", "deleted", "fieldData"
+
+    ## creation and initialization
+
+    def __init__(self, dbf, index=None, deleted=False, data=None):
+        """Instance initialiation.
+
+        Arguments:
+            dbf:
+                A `Dbf.Dbf` instance this record belonogs to.
+            index:
+                An integer record index or None. If this value is
+                None, record will be appended to the DBF.
+            deleted:
+                Boolean flag indicating whether this record
+                is a deleted record.
+            data:
+                A sequence or None. This is a data of the fields.
+                If this argument is None, default values will be used.
+
+        """
+        self.dbf = dbf
+        # XXX: I'm not sure ``index`` is necessary
+        self.index = index
+        self.deleted = deleted
+        if data is None:
+            self.fieldData = [_fd.defaultValue for _fd in dbf.header.fields]
+        else:
+            self.fieldData = list(data)
+
+    # XXX: validate self.index before calculating position?
+    position = property(lambda self: self.dbf.header.headerLength + \
+        self.index * self.dbf.header.recordLength)
+
+    def rawFromStream(cls, dbf, index):
+        """Return raw record contents read from the stream.
+
+        Arguments:
+            dbf:
+                A `Dbf.Dbf` instance containing the record.
+            index:
+                Index of the record in the records' container.
+                This argument can't be None in this call.
+
+        Return value is a string containing record data in DBF format.
+
+        """
+        # XXX: may be write smth assuming, that current stream
+        # position is the required one? it could save some
+        # time required to calculate where to seek in the file
+        dbf.stream.seek(dbf.header.headerLength +
+            index * dbf.header.recordLength)
+        return dbf.stream.read(dbf.header.recordLength)
+    rawFromStream = classmethod(rawFromStream)
+
+    def fromStream(cls, dbf, index):
+        """Return a record read from the stream.
+
+        Arguments:
+            dbf:
+                A `Dbf.Dbf` instance new record should belong to.
+            index:
+                Index of the record in the records' container.
+                This argument can't be None in this call.
+
+        Return value is an instance of the current class.
+
+        """
+        return cls.fromString(dbf, cls.rawFromStream(dbf, index), index)
+    fromStream = classmethod(fromStream)
+
+    def fromString(cls, dbf, string, index=None):
+        """Return record read from the string object.
+
+        Arguments:
+            dbf:
+                A `Dbf.Dbf` instance new record should belong to.
+            string:
+                A string new record should be created from.
+            index:
+                Index of the record in the container. If this
+                argument is None, record will be appended.
+
+        Return value is an instance of the current class.
+
+        """
+        return cls(dbf, index, string[0]=="*",
+            [_fd.decodeFromRecord(string) for _fd in dbf.header.fields])
+    fromString = classmethod(fromString)
+
+    ## object representation
+
+    def __repr__(self):
+        _template = "%%%ds: %%s (%%s)" % max([len(_fld)
+            for _fld in self.dbf.fieldNames])
+        _rv = []
+        for _fld in self.dbf.fieldNames:
+            _val = self[_fld]
+            if _val is utils.INVALID_VALUE:
+                _rv.append(_template %
+                    (_fld, "None", "value cannot be decoded"))
+            else:
+                _rv.append(_template % (_fld, _val, type(_val)))
+        return "\n".join(_rv)
+
+    ## protected methods
+
+    def _write(self):
+        """Write data to the dbf stream.
+
+        Note:
+            This isn't a public method, it's better to
+            use 'store' instead publically.
+            Be design ``_write`` method should be called
+            only from the `Dbf` instance.
+
+
+        """
+        self._validateIndex(False)
+        self.dbf.stream.seek(self.position)
+        self.dbf.stream.write(bytes(self.toString(),
+            sys.getfilesystemencoding()))
+        # FIXME: may be move this write somewhere else?
+        # why we should check this condition for each record?
+        if self.index == len(self.dbf):
+            # this is the last record,
+            # we should write SUB (ASCII 26)
+            self.dbf.stream.write(b"\x1A")
+
+    ## utility methods
+
+    def _validateIndex(self, allowUndefined=True, checkRange=False):
+        """Valid ``self.index`` value.
+
+        If ``allowUndefined`` argument is True functions does nothing
+        in case of ``self.index`` pointing to None object.
+
+        """
+        if self.index is None:
+            if not allowUndefined:
+                raise ValueError("Index is undefined")
+        elif self.index < 0:
+            raise ValueError("Index can't be negative (%s)" % self.index)
+        elif checkRange and self.index <= self.dbf.header.recordCount:
+            raise ValueError("There are only %d records in the DBF" %
+                self.dbf.header.recordCount)
+
+    ## interface methods
+
+    def store(self):
+        """Store current record in the DBF.
+
+        If ``self.index`` is None, this record will be appended to the
+        records of the DBF this records belongs to; or replaced otherwise.
+
+        """
+        self._validateIndex()
+        if self.index is None:
+            self.index = len(self.dbf)
+            self.dbf.append(self)
+        else:
+            self.dbf[self.index] = self
+
+    def delete(self):
+        """Mark method as deleted."""
+        self.deleted = True
+
+    def toString(self):
+        """Return string packed record values."""
+#        for (_def, _dat) in zip(self.dbf.header.fields, self.fieldData):
+#            
+
+        return "".join([" *"[self.deleted]] + [
+           _def.encodeValue(_dat)
+            for (_def, _dat) in zip(self.dbf.header.fields, self.fieldData)
+        ])
+
+    def asList(self):
+        """Return a flat list of fields.
+
+        Note:
+            Change of the list's values won't change
+            real values stored in this object.
+
+        """
+        return self.fieldData[:]
+
+    def asDict(self):
+        """Return a dictionary of fields.
+
+        Note:
+            Change of the dicts's values won't change
+            real values stored in this object.
+
+        """
+        return dict([_i for _i in zip(self.dbf.fieldNames, self.fieldData)])
+
+    def __getitem__(self, key):
+        """Return value by field name or field index."""
+        if isinstance(key, int):
+            # integer index of the field
+            return self.fieldData[key]
+        # assuming string field name
+        return self.fieldData[self.dbf.indexOfFieldName(key)]
+
+    def __setitem__(self, key, value):
+        """Set field value by integer index of the field or string name."""
+        if isinstance(key, int):
+            # integer index of the field
+            return self.fieldData[key]
+        # assuming string field name
+        self.fieldData[self.dbf.indexOfFieldName(key)] = value
+
+# vim: et sts=4 sw=4 :

tablib/packages/dbfpy3/utils.py

@@ -0,0 +1,170 @@
+"""String utilities.
+
+TODO:
+  - allow strings in getDateTime routine;
+"""
+"""History (most recent first):
+11-feb-2007 [als]   added INVALID_VALUE
+10-feb-2007 [als]   allow date strings padded with spaces instead of zeroes
+20-dec-2005 [yc]    handle long objects in getDate/getDateTime
+16-dec-2005 [yc]    created from ``strutil`` module.
+"""
+
+__version__ = "$Revision: 1.4 $"[11:-2]
+__date__ = "$Date: 2007/02/11 08:57:17 $"[7:-2]
+
+import datetime
+import time
+
+
+def unzfill(str):
+    """Return a string without ASCII NULs.
+
+    This function searchers for the first NUL (ASCII 0) occurance
+    and truncates string till that position.
+
+    """
+    try:
+        return str[:str.index(b'\0')]
+    except ValueError:
+        return str
+
+
+def getDate(date=None):
+    """Return `datetime.date` instance.
+
+    Type of the ``date`` argument could be one of the following:
+        None:
+            use current date value;
+        datetime.date:
+            this value will be returned;
+        datetime.datetime:
+            the result of the date.date() will be returned;
+        string:
+            assuming "%Y%m%d" or "%y%m%dd" format;
+        number:
+            assuming it's a timestamp (returned for example
+            by the time.time() call;
+        sequence:
+            assuming (year, month, day, ...) sequence;
+
+    Additionaly, if ``date`` has callable ``ticks`` attribute,
+    it will be used and result of the called would be treated
+    as a timestamp value.
+
+    """
+    if date is None:
+        # use current value
+        return datetime.date.today()
+    if isinstance(date, datetime.date):
+        return date
+    if isinstance(date, datetime.datetime):
+        return date.date()
+    if isinstance(date, (int, float)):
+        # date is a timestamp
+        return datetime.date.fromtimestamp(date)
+    if isinstance(date, str):
+        date = date.replace(" ", "0")
+        if len(date) == 6:
+            # yymmdd
+            return datetime.date(*time.strptime(date, "%y%m%d")[:3])
+        # yyyymmdd
+        return datetime.date(*time.strptime(date, "%Y%m%d")[:3])
+    if hasattr(date, "__getitem__"):
+        # a sequence (assuming date/time tuple)
+        return datetime.date(*date[:3])
+    return datetime.date.fromtimestamp(date.ticks())
+
+
+def getDateTime(value=None):
+    """Return `datetime.datetime` instance.
+
+    Type of the ``value`` argument could be one of the following:
+        None:
+            use current date value;
+        datetime.date:
+            result will be converted to the `datetime.datetime` instance
+            using midnight;
+        datetime.datetime:
+            ``value`` will be returned as is;
+        string:
+            *** CURRENTLY NOT SUPPORTED ***;
+        number:
+            assuming it's a timestamp (returned for example
+            by the time.time() call;
+        sequence:
+            assuming (year, month, day, ...) sequence;
+
+    Additionaly, if ``value`` has callable ``ticks`` attribute,
+    it will be used and result of the called would be treated
+    as a timestamp value.
+
+    """
+    if value is None:
+        # use current value
+        return datetime.datetime.today()
+    if isinstance(value, datetime.datetime):
+        return value
+    if isinstance(value, datetime.date):
+        return datetime.datetime.fromordinal(value.toordinal())
+    if isinstance(value, (int, float)):
+        # value is a timestamp
+        return datetime.datetime.fromtimestamp(value)
+    if isinstance(value, str):
+        raise NotImplementedError("Strings aren't currently implemented")
+    if hasattr(value, "__getitem__"):
+        # a sequence (assuming date/time tuple)
+        return datetime.datetime(*tuple(value)[:6])
+    return datetime.datetime.fromtimestamp(value.ticks())
+
+
+class classproperty(property):
+    """Works in the same way as a ``property``, but for the classes."""
+
+    def __get__(self, obj, cls):
+        return self.fget(cls)
+
+
+class _InvalidValue(object):
+
+    """Value returned from DBF records when field validation fails
+
+    The value is not equal to anything except for itself
+    and equal to all empty values: None, 0, empty string etc.
+    In other words, invalid value is equal to None and not equal
+    to None at the same time.
+
+    This value yields zero upon explicit conversion to a number type,
+    empty string for string types, and False for boolean.
+
+    """
+
+    def __eq__(self, other):
+        return not other
+
+    def __ne__(self, other):
+        return not (other is self)
+
+    def __bool__(self):
+        return False
+
+    def __int__(self):
+        return 0
+    __long__ = __int__
+
+    def __float__(self):
+        return 0.0
+
+    def __str__(self):
+        return ""
+
+    def __unicode__(self):
+        return ""
+
+    def __repr__(self):
+        return "<INVALID>"
+
+# invalid value is a constant singleton
+INVALID_VALUE = _InvalidValue()
+
+# vim: set et sts=4 sw=4 :

tablib/packages/markup.py

@@ -33,7 +33,7 @@ class element:
             self.tag = tag.lower( )
         else:
             self.tag = tag.upper( )
-    
+
     def __call__( self, *args, **kwargs ):
         if len( args ) > 1:
             raise ArgumentError( self.tag )
@@ -42,14 +42,14 @@ class element:
         if self.parent is not None and self.parent.class_ is not None:
             if 'class_' not in kwargs:
                 kwargs['class_'] = self.parent.class_
-            
+
         if self.parent is None and len( args ) == 1:
             x = [ self.render( self.tag, False, myarg, mydict ) for myarg, mydict in _argsdicts( args, kwargs ) ]
             return '\n'.join( x )
         elif self.parent is None and len( args ) == 0:
             x = [ self.render( self.tag, True, myarg, mydict ) for myarg, mydict in _argsdicts( args, kwargs ) ]
             return '\n'.join( x )
-            
+
         if self.tag in self.parent.twotags:
             for myarg, mydict in _argsdicts( args, kwargs ):
                 self.render( self.tag, False, myarg, mydict )
@@ -63,33 +63,31 @@ class element:
             raise DeprecationError( self.tag )
         else:
             raise InvalidElementError( self.tag, self.parent.mode )
-    
+
     def render( self, tag, single, between, kwargs ):
         """Append the actual tags to content."""
 
-        out = "<%s" % tag
+        out = u"<%s" % tag
         for key, value in kwargs.iteritems( ):
             if value is not None:               # when value is None that means stuff like <... checked>
                 key = key.strip('_')            # strip this so class_ will mean class, etc.
-                if key == 'http_equiv':         # special cases, maybe change _ to - overall?
-                    key = 'http-equiv'
-                elif key == 'accept_charset':
-                    key = 'accept-charset'
-                out = "%s %s=\"%s\"" % ( out, key, escape( value ) )
+                if key in ['http_equiv', 'accept_charset']:
+                    key.replace('_','-')
+                out = u"%s %s=\"%s\"" % ( out, key, escape( value ) )
             else:
-                out = "%s %s" % ( out, key )
+                out = u"%s %s" % ( out, key )
         if between is not None:
-            out = "%s>%s</%s>" % ( out, between, tag )
+            out = u"%s>%s</%s>" % ( out, between, tag )
         else:
             if single:
-                out = "%s />" % out
+                out = u"%s />" % out
             else:
-                out = "%s>" % out
+                out = u"%s>" % out
         if self.parent is not None:
             self.parent.content.append( out )
         else:
             return out
-    
+
     def close( self ):
         """Append a closing tag unless element has only opening tag."""
 
@@ -128,11 +126,11 @@ class page:
                                 these two keyword arguments may be used to select
                                 the set of valid elements in 'xml' mode
                                 invalid elements will raise appropriate exceptions
-        
+
         separator --            string to place between added elements, defaults to newline
-        
+
         class_ --               a class that will be added to every element if defined"""
-        
+
         valid_onetags = [ "AREA", "BASE", "BR", "COL", "FRAME", "HR", "IMG", "INPUT", "LINK", "META", "PARAM" ]
         valid_twotags = [ "A", "ABBR", "ACRONYM", "ADDRESS", "B", "BDO", "BIG", "BLOCKQUOTE", "BODY", "BUTTON",
                 "CAPTION", "CITE", "CODE", "COLGROUP", "DD", "DEL", "DFN", "DIV", "DL", "DT", "EM", "FIELDSET",
@@ -163,7 +161,7 @@ class page:
             self.deptags += map( string.lower, self.deptags )
             self.mode = 'strict_html'
         elif mode == 'loose_html':
-            self.onetags = valid_onetags + deprecated_onetags 
+            self.onetags = valid_onetags + deprecated_onetags
             self.onetags += map( string.lower, self.onetags )
             self.twotags = valid_twotags + deprecated_twotags
             self.twotags += map( string.lower, self.twotags )
@@ -183,16 +181,16 @@ class page:
 
     def __getattr__( self, attr ):
         if attr.startswith("__") and attr.endswith("__"):
-            raise AttributeError, attr
+            raise AttributeError(attr)
         return element( attr, case=self.case, parent=self )
 
     def __str__( self ):
-        
+
         if self._full and ( self.mode == 'strict_html' or self.mode == 'loose_html' ):
             end = [ '</body>', '</html>' ]
         else:
             end = [ ]
-        
+
         return self.separator.join( self.header + self.content + self.footer + end )
 
     def __call__( self, escape=False ):
@@ -232,7 +230,7 @@ class page:
 
         lang --     language, usually a two character string, will appear
                     as <html lang='en'> in html mode (ignored in xml mode)
-        
+
         css --      Cascading Style Sheet filename as a string or a list of
                     strings for multiple css files (ignored in xml mode)
 
@@ -306,7 +304,7 @@ class page:
     def css( self, filelist ):
         """This convenience function is only useful for html.
         It adds css stylesheet(s) to the document via the <link> element."""
-      
+
         if isinstance( filelist, basestring ):
             self.link( href=filelist, rel='stylesheet', type='text/css', media='all' )
         else:
@@ -322,7 +320,7 @@ class page:
             for name, content in mydict.iteritems( ):
                 self.meta( name=name, content=content )
         else:
-            raise TypeError, "Metainfo should be called with a dictionary argument of name:content pairs."
+            raise TypeError ("Metainfo should be called with a dictionary argument of name:content pairs.")
 
     def scripts( self, mydict ):
         """Only useful in html, mydict is dictionary of src:type pairs will
@@ -332,20 +330,20 @@ class page:
             for src, type in mydict.iteritems( ):
                 self.script( '', src=src, type='text/%s' % type )
         else:
-            raise TypeError, "Script should be given a dictionary of src:type pairs."
+            raise TypeError ("Script should be given a dictionary of src:type pairs.")
 
 
 class _oneliner:
     """An instance of oneliner returns a string corresponding to one element.
     This class can be used to write 'oneliners' that return a string
     immediately so there is no need to instantiate the page class."""
-    
+
     def __init__( self, case='lower' ):
         self.case = case
-    
+
     def __getattr__( self, attr ):
         if attr.startswith("__") and attr.endswith("__"):
-            raise AttributeError, attr
+            raise AttributeError(attr)
         return element( attr, case=self.case, parent=None )
 
 oneliner = _oneliner( case='lower' )
@@ -353,13 +351,13 @@ upper_oneliner = _oneliner( case='upper' )
 
 def _argsdicts( args, mydict ):
     """A utility generator that pads argument list and dictionary values, will only be called with len( args ) = 0, 1."""
-    
+
     if len( args ) == 0:
-        args = None, 
+        args = None,
     elif len( args ) == 1:
         args = _totuple( args[0] )
     else:
-        raise Exception, "We should have never gotten here."
+        raise Exception("We should have never gotten here.")
 
     mykeys = mydict.keys( )
     myvalues = map( _totuple, mydict.values( ) )
@@ -418,7 +416,7 @@ _escape = escape
 
 def unescape( text ):
     """Inverse of escape."""
-    
+
     if isinstance( text, basestring ):
         if '&amp;' in text:
             text = text.replace( '&amp;', '&' )
@@ -481,4 +479,4 @@ class CustomizationError( MarkupError ):
         self.message = "If you customize the allowed elements, you must define both types 'onetags' and 'twotags'."
 
 if __name__ == '__main__':
-    print __doc__
+    print (__doc__)

tablib/packages/odf3/opendocument.py

@@ -39,7 +39,7 @@ UNIXPERMS = 0o100644 << 16  # -rw-r--r--
 IS_FILENAME = 0
 IS_IMAGE = 1
 # We need at least Python 2.2
-assert sys.version_info[0]>=2 and sys.version_info[1] >= 2
+assert sys.version_info[0]>=2
 
 #sys.setrecursionlimit(100)
 #The recursion limit is set conservative so mistakes like

tablib/packages/openpyxl3/reader/excel.py

@@ -58,11 +58,15 @@ def load_workbook(filename, use_iterators = False):
 
     """
 
-    if isinstance(filename, file):
+    try:
         # fileobject must have been opened with 'rb' flag
         # it is required by zipfile
         if 'b' not in filename.mode:
             raise OpenModeError("File-object must be opened in binary mode")
+    except AttributeError:
+        # filename is not an object
+        # it doesn't have mode attribute
+        pass
 
     try:
         archive = ZipFile(filename, 'r', ZIP_DEFLATED)

tablib/packages/unicodecsv/__init__.py

@@ -1,22 +1,65 @@
 # -*- coding: utf-8 -*-
 import csv
-from csv import *
+try:
+    from itertools import izip
+except ImportError:
+    izip = zip
 
 #http://semver.org/
-VERSION = (0, 8, 0)
+VERSION = (0, 10, 1)
 __version__ = ".".join(map(str,VERSION))
 
-def _stringify(s, encoding):
-    if type(s)==unicode:
-        return s.encode(encoding)
+pass_throughs = [
+    'register_dialect',
+    'unregister_dialect',
+    'get_dialect',
+    'list_dialects',
+    'field_size_limit',
+    'Dialect',
+    'excel',
+    'excel_tab',
+    'Sniffer',
+    'QUOTE_ALL',
+    'QUOTE_MINIMAL',
+    'QUOTE_NONNUMERIC',
+    'QUOTE_NONE',
+    'Error'
+]
+__all__ = [
+    'reader',
+    'writer',
+    'DictReader',
+    'DictWriter',
+] + pass_throughs
+
+for prop in pass_throughs:
+    globals()[prop]=getattr(csv, prop)
+
+def _stringify(s, encoding, errors):
+    if s is None:
+        return ''
+    if isinstance(s, unicode):
+        return s.encode(encoding, errors)
     elif isinstance(s, (int , float)):
         pass #let csv.QUOTE_NONNUMERIC do its thing.
-    elif type(s) != str:
+    elif not isinstance(s, str):
         s=str(s)
     return s
 
-def _stringify_list(l, encoding):
-    return [_stringify(s, encoding) for s in l]
+def _stringify_list(l, encoding, errors='strict'):
+    try:
+        return [_stringify(s, encoding, errors) for s in iter(l)]
+    except TypeError as e:
+        raise csv.Error(str(e))
+
+def _unicodify(s, encoding):
+    if s is None:
+        return None
+    if isinstance(s, (unicode, int, float)):
+        return s
+    elif isinstance(s, str):
+        return s.decode(encoding)
+    return s
 
 class UnicodeWriter(object):
     """
@@ -28,78 +71,127 @@ class UnicodeWriter(object):
     >>> f.seek(0)
     >>> r = unicodecsv.reader(f, encoding='utf-8')
     >>> row = r.next()
-    >>> print row[0], row[1]
-    é ñ
+    >>> row[0] == u'é'
+    True
+    >>> row[1] == u'ñ'
+    True
     """
-    def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
-        self.writer = csv.writer(f)
-        self.dialect = dialect
+    def __init__(self, f, dialect=csv.excel, encoding='utf-8', errors='strict',
+                 *args, **kwds):
         self.encoding = encoding
-        self.writer = csv.writer(f, dialect=dialect, **kwds)
+        self.writer = csv.writer(f, dialect, *args, **kwds)
+        self.encoding_errors = errors
 
     def writerow(self, row):
-        self.writer.writerow(_stringify_list(row, self.encoding))
+        self.writer.writerow(_stringify_list(row, self.encoding, self.encoding_errors))
 
     def writerows(self, rows):
         for row in rows:
           self.writerow(row)
+
+    @property
+    def dialect(self):
+        return self.writer.dialect
 writer = UnicodeWriter
 
 class UnicodeReader(object):
-    def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
-        self.reader = csv.reader(f, dialect=dialect, **kwds)
+    def __init__(self, f, dialect=None, encoding='utf-8', errors='strict',
+                 **kwds):
+        format_params = ['delimiter', 'doublequote', 'escapechar', 'lineterminator', 'quotechar', 'quoting', 'skipinitialspace']
+        if dialect is None:
+            if not any([kwd_name in format_params for kwd_name in kwds.keys()]):
+                dialect = csv.excel
+        self.reader = csv.reader(f, dialect, **kwds)
         self.encoding = encoding
+        self.encoding_errors = errors
 
     def next(self):
         row = self.reader.next()
-        return [unicode(s, self.encoding) for s in row]
+        encoding = self.encoding
+        encoding_errors = self.encoding_errors
+        float_ = float
+        unicode_ = unicode
+        return [(value if isinstance(value, float_) else
+                 unicode_(value, encoding, encoding_errors)) for value in row]
 
     def __iter__(self):
         return self
+
+    @property
+    def dialect(self):
+        return self.reader.dialect
+
+    @property
+    def line_num(self):
+        return self.reader.line_num
 reader = UnicodeReader
 
 class DictWriter(csv.DictWriter):
     """
     >>> from cStringIO import StringIO
     >>> f = StringIO()
-    >>> w = DictWriter(f, ['a', 'b'], restval=u'î')
-    >>> w.writerow({'a':'1'})
-    >>> w.writerow({'a':'1', 'b':u'ø'})
-    >>> w.writerow({'a':u'é'})
+    >>> w = DictWriter(f, ['a', u'ñ', 'b'], restval=u'î')
+    >>> w.writerow({'a':'1', u'ñ':'2'})
+    >>> w.writerow({'a':'1', u'ñ':'2', 'b':u'ø'})
+    >>> w.writerow({'a':u'é', u'ñ':'2'})
     >>> f.seek(0)
-    >>> r = DictReader(f, fieldnames=['a'], restkey='r')
-    >>> r.next() == {'a':u'1', 'r':[u"î"]}
+    >>> r = DictReader(f, fieldnames=['a', u'ñ'], restkey='r')
+    >>> r.next() == {'a': u'1', u'ñ':'2', 'r': [u'î']}
     True
-    >>> r.next() == {'a':u'1', 'r':[u"ø"]}
+    >>> r.next() == {'a': u'1', u'ñ':'2', 'r': [u'\xc3\xb8']}
+    True
+    >>> r.next() == {'a': u'\xc3\xa9', u'ñ':'2', 'r': [u'\xc3\xae']}
     True
-    >>> r.next() == {'a':u'é', 'r':[u"î"]}
     """
-    def __init__(self, csvfile, fieldnames, restval='', extrasaction='raise', dialect='excel', encoding='utf-8', *args, **kwds):
-        self.fieldnames = fieldnames
+    def __init__(self, csvfile, fieldnames, restval='', extrasaction='raise', dialect='excel', encoding='utf-8', errors='strict', *args, **kwds):
         self.encoding = encoding
-        self.restval = restval
-        self.writer = csv.DictWriter(csvfile, fieldnames, restval, extrasaction, dialect, *args, **kwds)
-    def writerow(self, d):
-        for fieldname in self.fieldnames:
-            if fieldname in d:
-                d[fieldname] = _stringify(d[fieldname], self.encoding)
-            else:
-                d[fieldname] = _stringify(self.restval, self.encoding)
-        self.writer.writerow(d)
+        csv.DictWriter.__init__(self, csvfile, fieldnames, restval, extrasaction, dialect, *args, **kwds)
+        self.writer = UnicodeWriter(csvfile, dialect, encoding=encoding, errors=errors, *args, **kwds)
+        self.encoding_errors = errors
+
+    def writeheader(self):
+        fieldnames = _stringify_list(self.fieldnames, self.encoding, self.encoding_errors)
+        header = dict(zip(self.fieldnames, self.fieldnames))
+        self.writerow(header)
 
 class DictReader(csv.DictReader):
-    def __init__(self, csvfile, fieldnames=None, restkey=None, restval=None, dialect='excel', encoding='utf-8', *args, **kwds):
-        self.restkey = restkey
-        self.encoding = encoding
-        self.reader = csv.DictReader(csvfile, fieldnames, restkey, restval, dialect, *args, **kwds)
+    """
+    >>> from cStringIO import StringIO
+    >>> f = StringIO()
+    >>> w = DictWriter(f, fieldnames=['name', 'place'])
+    >>> w.writerow({'name': 'Cary Grant', 'place': 'hollywood'})
+    >>> w.writerow({'name': 'Nathan Brillstone', 'place': u'øLand'})
+    >>> w.writerow({'name': u'Willam ø. Unicoder', 'place': u'éSpandland'})
+    >>> f.seek(0)
+    >>> r = DictReader(f, fieldnames=['name', 'place'])
+    >>> print r.next() == {'name': 'Cary Grant', 'place': 'hollywood'}
+    True
+    >>> print r.next() == {'name': 'Nathan Brillstone', 'place': u'øLand'}
+    True
+    >>> print r.next() == {'name': u'Willam ø. Unicoder', 'place': u'éSpandland'}
+    True
+    """
+    def __init__(self, csvfile, fieldnames=None, restkey=None, restval=None,
+                 dialect='excel', encoding='utf-8', errors='strict', *args,
+                 **kwds):
+        if fieldnames is not None:
+            fieldnames = _stringify_list(fieldnames, encoding)
+        csv.DictReader.__init__(self, csvfile, fieldnames, restkey, restval, dialect, *args, **kwds)
+        self.reader = UnicodeReader(csvfile, dialect, encoding=encoding,
+                                    errors=errors, *args, **kwds)
+        if fieldnames is None and not hasattr(csv.DictReader, 'fieldnames'):
+            # Python 2.5 fieldnames workaround. (http://bugs.python.org/issue3436)
+            reader = UnicodeReader(csvfile, dialect, encoding=encoding, *args, **kwds)
+            self.fieldnames = _stringify_list(reader.next(), reader.encoding)
+        self.unicode_fieldnames = [_unicodify(f, encoding) for f in
+                                   self.fieldnames]
+        self.unicode_restkey = _unicodify(restkey, encoding)
+
     def next(self):
-        d = self.reader.next()
-        for k, v in d.items():
-            if k == self.restkey:
-                rest = v
-                if rest:
-                    d[self.restkey] = [unicode(v, self.encoding) for v in rest]
-            else:
-                if v is not None:
-                    d[k] = unicode(v, self.encoding)
-        return d
+        row = csv.DictReader.next(self)
+        result = dict((uni_key, row[str_key]) for (str_key, uni_key) in
+                      izip(self.fieldnames, self.unicode_fieldnames))
+        rest = row.get(self.restkey)
+        if rest:
+            result[self.unicode_restkey] = rest
+        return result

tablib/packages/xlrd/biffh.py

@@ -0,0 +1,639 @@
+# -*- coding: cp1252 -*-
+
+##
+# Support module for the xlrd package.
+#
+# <p>Portions copyright © 2005-2008 Stephen John Machin, Lingfo Pty Ltd</p>
+# <p>This module is part of the xlrd package, which is released under a BSD-style licence.</p>
+##
+
+# 2008-02-10 SJM BIFF2 BLANK record
+# 2008-02-08 SJM Preparation for Excel 2.0 support
+# 2008-02-02 SJM Added suffixes (_B2, _B2_ONLY, etc) on record names for biff_dump & biff_count
+# 2007-12-04 SJM Added support for Excel 2.x (BIFF2) files.
+# 2007-09-08 SJM Avoid crash when zero-length Unicode string missing options byte.
+# 2007-04-22 SJM Remove experimental "trimming" facility.
+
+DEBUG = 0
+
+from struct import unpack
+import sys
+from timemachine import *
+
+class XLRDError(Exception):
+    pass
+
+##
+# Parent of almost all other classes in the package. Defines a common "dump" method
+# for debugging.
+
+class BaseObject(object):
+
+    _repr_these = []
+
+    ##
+    # @param f open file object, to which the dump is written
+    # @param header text to write before the dump
+    # @param footer text to write after the dump
+    # @param indent number of leading spaces (for recursive calls)
+
+    def dump(self, f=None, header=None, footer=None, indent=0):
+        if f is None:
+            f = sys.stderr
+        alist = self.__dict__.items()
+        alist.sort()
+        pad = " " * indent
+        if header is not None: print >> f, header
+        list_type = type([])
+        dict_type = type({})
+        for attr, value in alist:
+            if getattr(value, 'dump', None) and attr != 'book':
+                value.dump(f,
+                    header="%s%s (%s object):" % (pad, attr, value.__class__.__name__),
+                    indent=indent+4)
+            elif attr not in self._repr_these and (
+                isinstance(value, list_type) or isinstance(value, dict_type)
+                ):
+                print >> f, "%s%s: %s, len = %d" % (pad, attr, type(value), len(value))
+            else:
+                print >> f, "%s%s: %r" % (pad, attr, value)
+        if footer is not None: print >> f, footer
+
+FUN, FDT, FNU, FGE, FTX = range(5) # unknown, date, number, general, text
+DATEFORMAT = FDT
+NUMBERFORMAT = FNU
+
+(
+    XL_CELL_EMPTY,
+    XL_CELL_TEXT,
+    XL_CELL_NUMBER,
+    XL_CELL_DATE,
+    XL_CELL_BOOLEAN,
+    XL_CELL_ERROR,
+    XL_CELL_BLANK, # for use in debugging, gathering stats, etc
+) = range(7)
+
+biff_text_from_num = {
+    0:  "(not BIFF)",
+    20: "2.0",
+    21: "2.1",
+    30: "3",
+    40: "4S",
+    45: "4W",
+    50: "5",
+    70: "7",
+    80: "8",
+    85: "8X",
+    }
+
+##
+# <p>This dictionary can be used to produce a text version of the internal codes
+# that Excel uses for error cells. Here are its contents:
+# <pre>
+# 0x00: '#NULL!',  # Intersection of two cell ranges is empty
+# 0x07: '#DIV/0!', # Division by zero
+# 0x0F: '#VALUE!', # Wrong type of operand
+# 0x17: '#REF!',   # Illegal or deleted cell reference
+# 0x1D: '#NAME?',  # Wrong function or range name
+# 0x24: '#NUM!',   # Value range overflow
+# 0x2A: '#N/A!',   # Argument or function not available
+# </pre></p>
+
+error_text_from_code = {
+    0x00: '#NULL!',  # Intersection of two cell ranges is empty
+    0x07: '#DIV/0!', # Division by zero
+    0x0F: '#VALUE!', # Wrong type of operand
+    0x17: '#REF!',   # Illegal or deleted cell reference
+    0x1D: '#NAME?',  # Wrong function or range name
+    0x24: '#NUM!',   # Value range overflow
+    0x2A: '#N/A!',   # Argument or function not available
+}
+
+BIFF_FIRST_UNICODE = 80
+
+XL_WORKBOOK_GLOBALS = WBKBLOBAL = 0x5
+XL_WORKBOOK_GLOBALS_4W = 0x100
+XL_WORKSHEET = WRKSHEET = 0x10
+
+XL_BOUNDSHEET_WORKSHEET = 0x00
+XL_BOUNDSHEET_CHART     = 0x02
+XL_BOUNDSHEET_VB_MODULE = 0x06
+
+# XL_RK2 = 0x7e
+XL_ARRAY  = 0x0221
+XL_ARRAY2 = 0x0021
+XL_BLANK = 0x0201
+XL_BLANK_B2 = 0x01
+XL_BOF = 0x809
+XL_BOOLERR = 0x205
+XL_BOOLERR_B2 = 0x5
+XL_BOUNDSHEET = 0x85
+XL_BUILTINFMTCOUNT = 0x56
+XL_CF = 0x01B1
+XL_CODEPAGE = 0x42
+XL_COLINFO = 0x7D
+XL_COLUMNDEFAULT = 0x20 # BIFF2 only
+XL_COLWIDTH = 0x24 # BIFF2 only
+XL_CONDFMT = 0x01B0
+XL_CONTINUE = 0x3c
+XL_COUNTRY = 0x8C
+XL_DATEMODE = 0x22
+XL_DEFAULTROWHEIGHT = 0x0225
+XL_DEFCOLWIDTH = 0x55
+XL_DIMENSION = 0x200
+XL_DIMENSION2 = 0x0
+XL_EFONT = 0x45
+XL_EOF = 0x0a
+XL_EXTERNNAME = 0x23
+XL_EXTERNSHEET = 0x17
+XL_EXTSST = 0xff
+XL_FEAT11 = 0x872
+XL_FILEPASS = 0x2f
+XL_FONT = 0x31
+XL_FONT_B3B4 = 0x231
+XL_FORMAT = 0x41e
+XL_FORMAT2 = 0x1E # BIFF2, BIFF3
+XL_FORMULA = 0x6
+XL_FORMULA3 = 0x206
+XL_FORMULA4 = 0x406
+XL_GCW = 0xab
+XL_INDEX = 0x20b
+XL_INTEGER = 0x2 # BIFF2 only
+XL_IXFE = 0x44 # BIFF2 only
+XL_LABEL = 0x204
+XL_LABEL_B2 = 0x04
+XL_LABELRANGES = 0x15f
+XL_LABELSST = 0xfd
+XL_MERGEDCELLS = 0xE5
+XL_MSO_DRAWING = 0x00EC
+XL_MSO_DRAWING_GROUP = 0x00EB
+XL_MSO_DRAWING_SELECTION = 0x00ED
+XL_MULRK = 0xbd
+XL_MULBLANK = 0xbe
+XL_NAME = 0x18
+XL_NOTE = 0x1c
+XL_NUMBER = 0x203
+XL_NUMBER_B2 = 0x3
+XL_OBJ = 0x5D
+XL_PALETTE = 0x92
+XL_RK = 0x27e
+XL_ROW = 0x208
+XL_ROW_B2 = 0x08
+XL_RSTRING = 0xd6
+XL_SHEETHDR = 0x8F # BIFF4W only
+XL_SHEETSOFFSET = 0x8E # BIFF4W only
+XL_SHRFMLA = 0x04bc
+XL_SST = 0xfc
+XL_STANDARDWIDTH = 0x99
+XL_STRING = 0x207
+XL_STRING_B2 = 0x7
+XL_STYLE = 0x293
+XL_SUPBOOK = 0x1AE
+XL_TABLEOP = 0x236
+XL_TABLEOP2 = 0x37
+XL_TABLEOP_B2 = 0x36
+XL_TXO = 0x1b6
+XL_UNCALCED = 0x5e
+XL_UNKNOWN = 0xffff
+XL_WINDOW2 = 0x023E
+XL_WRITEACCESS = 0x5C
+XL_XF = 0xe0
+XL_XF2 = 0x0043 # BIFF2 version of XF record
+XL_XF3 = 0x0243 # BIFF3 version of XF record
+XL_XF4 = 0x0443 # BIFF4 version of XF record
+
+boflen = {0x0809: 8, 0x0409: 6, 0x0209: 6, 0x0009: 4}
+bofcodes = (0x0809, 0x0409, 0x0209, 0x0009)
+
+XL_FORMULA_OPCODES = (0x0006, 0x0406, 0x0206)
+
+_cell_opcode_list = [
+    XL_BOOLERR,
+    XL_FORMULA,
+    XL_FORMULA3,
+    XL_FORMULA4,
+    XL_LABEL,
+    XL_LABELSST,
+    XL_MULRK,
+    XL_NUMBER,
+    XL_RK,
+    XL_RSTRING,
+    ]
+_cell_opcode_dict = {}
+for _cell_opcode in _cell_opcode_list:
+    _cell_opcode_dict[_cell_opcode] = 1
+is_cell_opcode = _cell_opcode_dict.has_key
+
+# def fprintf(f, fmt, *vargs): f.write(fmt % vargs)
+
+def fprintf(f, fmt, *vargs):
+    if fmt.endswith('\n'):
+        print >> f, fmt[:-1] % vargs
+    else:
+        print >> f, fmt % vargs,
+
+def upkbits(tgt_obj, src, manifest, local_setattr=setattr):
+    for n, mask, attr in manifest:
+        local_setattr(tgt_obj, attr, (src & mask) >> n)
+
+def upkbitsL(tgt_obj, src, manifest, local_setattr=setattr, local_int=int):
+    for n, mask, attr in manifest:
+        local_setattr(tgt_obj, attr, local_int((src & mask) >> n))
+
+def unpack_string(data, pos, encoding, lenlen=1):
+    nchars = unpack('<' + 'BH'[lenlen-1], data[pos:pos+lenlen])[0]
+    pos += lenlen
+    return unicode(data[pos:pos+nchars], encoding)
+
+def unpack_string_update_pos(data, pos, encoding, lenlen=1, known_len=None):
+    if known_len is not None:
+        # On a NAME record, the length byte is detached from the front of the string.
+        nchars = known_len
+    else:
+        nchars = unpack('<' + 'BH'[lenlen-1], data[pos:pos+lenlen])[0]
+        pos += lenlen
+    newpos = pos + nchars
+    return (unicode(data[pos:newpos], encoding), newpos)
+
+def unpack_unicode(data, pos, lenlen=2):
+    "Return unicode_strg"
+    nchars = unpack('<' + 'BH'[lenlen-1], data[pos:pos+lenlen])[0]
+    if not nchars:
+        # Ambiguous whether 0-length string should have an "options" byte.
+        # Avoid crash if missing.
+        return u""
+    pos += lenlen
+    options = ord(data[pos])
+    pos += 1
+    # phonetic = options & 0x04
+    # richtext = options & 0x08
+    if options & 0x08:
+        # rt = unpack('<H', data[pos:pos+2])[0] # unused
+        pos += 2
+    if options & 0x04:
+        # sz = unpack('<i', data[pos:pos+4])[0] # unused
+        pos += 4
+    if options & 0x01:
+        # Uncompressed UTF-16-LE
+        rawstrg = data[pos:pos+2*nchars]
+        # if DEBUG: print "nchars=%d pos=%d rawstrg=%r" % (nchars, pos, rawstrg)
+        strg = unicode(rawstrg, 'utf_16_le')
+        # pos += 2*nchars
+    else:
+        # Note: this is COMPRESSED (not ASCII!) encoding!!!
+        # Merely returning the raw bytes would work OK 99.99% of the time
+        # if the local codepage was cp1252 -- however this would rapidly go pear-shaped
+        # for other codepages so we grit our Anglocentric teeth and return Unicode :-)
+
+        strg = unicode(data[pos:pos+nchars], "latin_1")
+        # pos += nchars
+    # if richtext:
+    #     pos += 4 * rt
+    # if phonetic:
+    #     pos += sz
+    # return (strg, pos)
+    return strg
+
+def unpack_unicode_update_pos(data, pos, lenlen=2, known_len=None):
+    "Return (unicode_strg, updated value of pos)"
+    if known_len is not None:
+        # On a NAME record, the length byte is detached from the front of the string.
+        nchars = known_len
+    else:
+        nchars = unpack('<' + 'BH'[lenlen-1], data[pos:pos+lenlen])[0]
+        pos += lenlen
+    if not nchars and not data[pos:]:
+        # Zero-length string with no options byte
+        return (u"", pos)
+    options = ord(data[pos])
+    pos += 1
+    phonetic = options & 0x04
+    richtext = options & 0x08
+    if richtext:
+        rt = unpack('<H', data[pos:pos+2])[0]
+        pos += 2
+    if phonetic:
+        sz = unpack('<i', data[pos:pos+4])[0]
+        pos += 4
+    if options & 0x01:
+        # Uncompressed UTF-16-LE
+        strg = unicode(data[pos:pos+2*nchars], 'utf_16_le')
+        pos += 2*nchars
+    else:
+        # Note: this is COMPRESSED (not ASCII!) encoding!!!
+        strg = unicode(data[pos:pos+nchars], "latin_1")
+        pos += nchars
+    if richtext:
+        pos += 4 * rt
+    if phonetic:
+        pos += sz
+    return (strg, pos)
+
+def unpack_cell_range_address_list_update_pos(
+    output_list, data, pos, biff_version, addr_size=6):
+    # output_list is updated in situ
+    if biff_version < 80:
+        assert addr_size == 6
+    else:
+        assert addr_size in (6, 8)
+    n, = unpack("<H", data[pos:pos+2])
+    pos += 2
+    if n:
+        if addr_size == 6:
+            fmt = "<HHBB"
+        else:
+            fmt = "<HHHH"
+        for _unused in xrange(n):
+            ra, rb, ca, cb = unpack(fmt, data[pos:pos+addr_size])
+            output_list.append((ra, rb+1, ca, cb+1))
+            pos += addr_size
+    return pos
+
+_brecstrg = """\
+0000 DIMENSIONS_B2
+0001 BLANK_B2
+0002 INTEGER_B2_ONLY
+0003 NUMBER_B2
+0004 LABEL_B2
+0005 BOOLERR_B2
+0006 FORMULA
+0007 STRING_B2
+0008 ROW_B2
+0009 BOF_B2
+000A EOF
+000B INDEX_B2_ONLY
+000C CALCCOUNT
+000D CALCMODE
+000E PRECISION
+000F REFMODE
+0010 DELTA
+0011 ITERATION
+0012 PROTECT
+0013 PASSWORD
+0014 HEADER
+0015 FOOTER
+0016 EXTERNCOUNT
+0017 EXTERNSHEET
+0018 NAME_B2,5+
+0019 WINDOWPROTECT
+001A VERTICALPAGEBREAKS
+001B HORIZONTALPAGEBREAKS
+001C NOTE
+001D SELECTION
+001E FORMAT_B2-3
+001F BUILTINFMTCOUNT_B2
+0020 COLUMNDEFAULT_B2_ONLY
+0021 ARRAY_B2_ONLY
+0022 DATEMODE
+0023 EXTERNNAME
+0024 COLWIDTH_B2_ONLY
+0025 DEFAULTROWHEIGHT_B2_ONLY
+0026 LEFTMARGIN
+0027 RIGHTMARGIN
+0028 TOPMARGIN
+0029 BOTTOMMARGIN
+002A PRINTHEADERS
+002B PRINTGRIDLINES
+002F FILEPASS
+0031 FONT
+0032 FONT2_B2_ONLY
+0036 TABLEOP_B2
+0037 TABLEOP2_B2
+003C CONTINUE
+003D WINDOW1
+003E WINDOW2_B2
+0040 BACKUP
+0041 PANE
+0042 CODEPAGE
+0043 XF_B2
+0044 IXFE_B2_ONLY
+0045 EFONT_B2_ONLY
+004D PLS
+0051 DCONREF
+0055 DEFCOLWIDTH
+0056 BUILTINFMTCOUNT_B3-4
+0059 XCT
+005A CRN
+005B FILESHARING
+005C WRITEACCESS
+005D OBJECT
+005E UNCALCED
+005F SAVERECALC
+0063 OBJECTPROTECT
+007D COLINFO
+007E RK2_mythical_?
+0080 GUTS
+0081 WSBOOL
+0082 GRIDSET
+0083 HCENTER
+0084 VCENTER
+0085 BOUNDSHEET
+0086 WRITEPROT
+008C COUNTRY
+008D HIDEOBJ
+008E SHEETSOFFSET
+008F SHEETHDR
+0090 SORT
+0092 PALETTE
+0099 STANDARDWIDTH
+009B FILTERMODE
+009C FNGROUPCOUNT
+009D AUTOFILTERINFO
+009E AUTOFILTER
+00A0 SCL
+00A1 SETUP
+00AB GCW
+00BD MULRK
+00BE MULBLANK
+00C1 MMS
+00D6 RSTRING
+00D7 DBCELL
+00DA BOOKBOOL
+00DD SCENPROTECT
+00E0 XF
+00E1 INTERFACEHDR
+00E2 INTERFACEEND
+00E5 MERGEDCELLS
+00E9 BITMAP
+00EB MSO_DRAWING_GROUP
+00EC MSO_DRAWING
+00ED MSO_DRAWING_SELECTION
+00EF PHONETIC
+00FC SST
+00FD LABELSST
+00FF EXTSST
+013D TABID
+015F LABELRANGES
+0160 USESELFS
+0161 DSF
+01AE SUPBOOK
+01AF PROTECTIONREV4
+01B0 CONDFMT
+01B1 CF
+01B2 DVAL
+01B6 TXO
+01B7 REFRESHALL
+01B8 HLINK
+01BC PASSWORDREV4
+01BE DV
+01C0 XL9FILE
+01C1 RECALCID
+0200 DIMENSIONS
+0201 BLANK
+0203 NUMBER
+0204 LABEL
+0205 BOOLERR
+0206 FORMULA_B3
+0207 STRING
+0208 ROW
+0209 BOF
+020B INDEX_B3+
+0218 NAME
+0221 ARRAY
+0223 EXTERNNAME_B3-4
+0225 DEFAULTROWHEIGHT
+0231 FONT_B3B4
+0236 TABLEOP
+023E WINDOW2
+0243 XF_B3
+027E RK
+0293 STYLE
+0406 FORMULA_B4
+0409 BOF
+041E FORMAT
+0443 XF_B4
+04BC SHRFMLA
+0800 QUICKTIP
+0809 BOF
+0862 SHEETLAYOUT
+0867 SHEETPROTECTION
+0868 RANGEPROTECTION
+"""
+
+biff_rec_name_dict = {}
+for _buff in _brecstrg.splitlines():
+    _numh, _name = _buff.split()
+    biff_rec_name_dict[int(_numh, 16)] = _name
+del _buff, _name, _brecstrg
+
+def hex_char_dump(strg, ofs, dlen, base=0, fout=sys.stdout, unnumbered=False):
+    endpos = min(ofs + dlen, len(strg))
+    pos = ofs
+    numbered = not unnumbered
+    num_prefix = ''
+    while pos < endpos:
+        endsub = min(pos + 16, endpos)
+        substrg = strg[pos:endsub]
+        lensub = endsub - pos
+        if lensub <= 0 or lensub != len(substrg):
+            fprintf(
+                sys.stdout,
+                '??? hex_char_dump: ofs=%d dlen=%d base=%d -> endpos=%d pos=%d endsub=%d substrg=%r\n',
+                ofs, dlen, base, endpos, pos, endsub, substrg)
+            break
+        hexd = ''.join(["%02x " % ord(c) for c in substrg])
+        chard = ''
+        for c in substrg:
+            if c == '\0':
+                c = '~'
+            elif not (' ' <= c <= '~'):
+                c = '?'
+            chard += c
+        if numbered:
+            num_prefix = "%5d: " %  (base+pos-ofs)
+        fprintf(fout, "%s     %-48s %s\n", num_prefix, hexd, chard)
+        pos = endsub
+
+def biff_dump(mem, stream_offset, stream_len, base=0, fout=sys.stdout, unnumbered=False):
+    pos = stream_offset
+    stream_end = stream_offset + stream_len
+    adj = base - stream_offset
+    dummies = 0
+    numbered = not unnumbered
+    num_prefix = ''
+    while stream_end - pos >= 4:
+        rc, length = unpack('<HH', mem[pos:pos+4])
+        if rc == 0 and length == 0:
+            if mem[pos:] == '\0' * (stream_end - pos):
+                dummies = stream_end - pos
+                savpos = pos
+                pos = stream_end
+                break
+            if dummies:
+                dummies += 4
+            else:
+                savpos = pos
+                dummies = 4
+            pos += 4
+        else:
+            if dummies:
+                if numbered:
+                    num_prefix =  "%5d: " % (adj + savpos)
+                fprintf(fout, "%s---- %d zero bytes skipped ----\n", num_prefix, dummies)
+                dummies = 0
+            recname = biff_rec_name_dict.get(rc, '<UNKNOWN>')
+            if numbered:
+                num_prefix = "%5d: " % (adj + pos)
+            fprintf(fout, "%s%04x %s len = %04x (%d)\n", num_prefix, rc, recname, length, length)
+            pos += 4
+            hex_char_dump(mem, pos, length, adj+pos, fout, unnumbered)
+            pos += length
+    if dummies:
+        if numbered:
+            num_prefix =  "%5d: " % (adj + savpos)
+        fprintf(fout, "%s---- %d zero bytes skipped ----\n", num_prefix, dummies)
+    if pos < stream_end:
+        if numbered:
+            num_prefix = "%5d: " % (adj + pos)
+        fprintf(fout, "%s---- Misc bytes at end ----\n", num_prefix)
+        hex_char_dump(mem, pos, stream_end-pos, adj + pos, fout, unnumbered)
+    elif pos > stream_end:
+        fprintf(fout, "Last dumped record has length (%d) that is too large\n", length)
+
+def biff_count_records(mem, stream_offset, stream_len, fout=sys.stdout):
+    pos = stream_offset
+    stream_end = stream_offset + stream_len
+    tally = {}
+    while stream_end - pos >= 4:
+        rc, length = unpack('<HH', mem[pos:pos+4])
+        if rc == 0 and length == 0:
+            if mem[pos:] == '\0' * (stream_end - pos):
+                break
+            recname = "<Dummy (zero)>"
+        else:
+            recname = biff_rec_name_dict.get(rc, None)
+            if recname is None:
+                recname = "Unknown_0x%04X" % rc
+        if tally.has_key(recname):
+            tally[recname] += 1
+        else:
+            tally[recname] = 1
+        pos += length + 4
+    slist = tally.items()
+    slist.sort()
+    for recname, count in slist:
+        print >> fout, "%8d %s" % (count, recname)
+
+encoding_from_codepage = {
+    1200 : 'utf_16_le',
+    10000: 'mac_roman',
+    10006: 'mac_greek', # guess
+    10007: 'mac_cyrillic', # guess
+    10029: 'mac_latin2', # guess
+    10079: 'mac_iceland', # guess
+    10081: 'mac_turkish', # guess
+    32768: 'mac_roman',
+    32769: 'cp1252',
+    }
+# some more guessing, for Indic scripts
+# codepage 57000 range:
+# 2 Devanagari [0]
+# 3 Bengali [1]
+# 4 Tamil [5]
+# 5 Telegu [6]
+# 6 Assamese [1] c.f. Bengali
+# 7 Oriya [4]
+# 8 Kannada [7]
+# 9 Malayalam [8]
+# 10 Gujarati [3]
+# 11 Gurmukhi [2]

tablib/packages/xlrd/compdoc.py

@@ -0,0 +1,358 @@
+# -*- coding: cp1252 -*-
+
+##
+# Implements the minimal functionality required
+# to extract a "Workbook" or "Book" stream (as one big string)
+# from an OLE2 Compound Document file.
+# <p>Copyright © 2005-2008 Stephen John Machin, Lingfo Pty Ltd</p>
+# <p>This module is part of the xlrd package, which is released under a BSD-style licence.</p>
+##
+
+# No part of the content of this file was derived from the works of David Giffin.
+
+# 2008-11-04 SJM Avoid assertion error when -1 used instead of -2 for first_SID of empty SCSS [Frank Hoffsuemmer]
+# 2007-09-08 SJM Warning message if sector sizes are extremely large.
+# 2007-05-07 SJM Meaningful exception instead of IndexError if a SAT (sector allocation table) is corrupted.
+# 2007-04-22 SJM Missing "<" in a struct.unpack call => can't open files on bigendian platforms.
+
+
+import sys
+from struct import unpack
+from timemachine import *
+
+##
+# Magic cookie that should appear in the first 8 bytes of the file.
+SIGNATURE = "\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1"
+
+EOCSID = -2
+FREESID = -1
+SATSID = -3
+MSATSID = -4
+
+class CompDocError(Exception):
+    pass
+
+class DirNode(object):
+
+    def __init__(self, DID, dent, DEBUG=0):
+        # dent is the 128-byte directory entry
+        self.DID = DID
+        # (cbufsize, self.etype, self.colour, self.left_DID, self.right_DID,
+        # self.root_DID,
+        # self.first_SID,
+        # self.tot_size) = \
+        #     unpack('<HBBiii16x4x8x8xii4x', dent[64:128])
+        (cbufsize, self.etype, self.colour, self.left_DID, self.right_DID,
+        self.root_DID) = \
+            unpack('<HBBiii', dent[64:80])
+        (self.first_SID, self.tot_size) = \
+            unpack('<ii', dent[116:124])
+        if cbufsize == 0:
+            self.name = u''
+        else:
+            self.name = unicode(dent[0:cbufsize-2], 'utf_16_le') # omit the trailing U+0000
+        self.children = [] # filled in later
+        self.parent = -1 # indicates orphan; fixed up later
+        self.tsinfo = unpack('<IIII', dent[100:116])
+        if DEBUG:
+            self.dump(DEBUG)
+
+    def dump(self, DEBUG=1):
+        print "DID=%d name=%r etype=%d DIDs(left=%d right=%d root=%d parent=%d kids=%r) first_SID=%d tot_size=%d" \
+            % (self.DID, self.name, self.etype, self.left_DID,
+            self.right_DID, self.root_DID, self.parent, self.children, self.first_SID, self.tot_size)
+        if DEBUG == 2:
+            # cre_lo, cre_hi, mod_lo, mod_hi = tsinfo
+            print "timestamp info", self.tsinfo
+
+def _build_family_tree(dirlist, parent_DID, child_DID):
+    if child_DID < 0: return
+    _build_family_tree(dirlist, parent_DID, dirlist[child_DID].left_DID)
+    dirlist[parent_DID].children.append(child_DID)
+    dirlist[child_DID].parent = parent_DID
+    _build_family_tree(dirlist, parent_DID, dirlist[child_DID].right_DID)
+    if dirlist[child_DID].etype == 1: # storage
+        _build_family_tree(dirlist, child_DID, dirlist[child_DID].root_DID)
+
+##
+# Compound document handler.
+# @param mem The raw contents of the file, as a string, or as an mmap.mmap() object. The
+# only operation it needs to support is slicing.
+
+class CompDoc(object):
+
+    def __init__(self, mem, logfile=sys.stdout, DEBUG=0):
+        self.logfile = logfile
+        if mem[0:8] != SIGNATURE:
+            raise CompDocError('Not an OLE2 compound document')
+        if mem[28:30] != '\xFE\xFF':
+            raise CompDocError('Expected "little-endian" marker, found %r' % mem[28:30])
+        revision, version = unpack('<HH', mem[24:28])
+        if DEBUG:
+            print >> logfile, "\nCompDoc format: version=0x%04x revision=0x%04x" % (version, revision)
+        self.mem = mem
+        ssz, sssz = unpack('<HH', mem[30:34])
+        if ssz > 20: # allows for 2**20 bytes i.e. 1MB
+            print >> logfile, \
+                "WARNING: sector size (2**%d) is preposterous; assuming 512 and continuing ..." \
+                % ssz
+            ssz = 9
+        if sssz > ssz:
+            print >> logfile, \
+                "WARNING: short stream sector size (2**%d) is preposterous; assuming 64 and continuing ..." \
+                % sssz
+            sssz = 6
+        self.sec_size = sec_size = 1 << ssz
+        self.short_sec_size = 1 << sssz
+        (
+            SAT_tot_secs, self.dir_first_sec_sid, _unused, self.min_size_std_stream,
+            SSAT_first_sec_sid, SSAT_tot_secs,
+            MSAT_first_sec_sid, MSAT_tot_secs,
+        # ) = unpack('<ii4xiiiii', mem[44:76])
+        ) = unpack('<iiiiiiii', mem[44:76])
+        mem_data_len = len(mem) - 512
+        mem_data_secs, left_over = divmod(mem_data_len, sec_size)
+        if left_over:
+            #### raise CompDocError("Not a whole number of sectors")
+            print >> logfile, \
+                "WARNING *** file size (%d) not 512 + multiple of sector size (%d)" \
+                % (len(mem), sec_size)
+        if DEBUG:
+            print >> logfile, 'sec sizes', ssz, sssz, sec_size, self.short_sec_size
+            print >> logfile, "mem data: %d bytes == %d sectors" % (mem_data_len, mem_data_secs)
+            print >> logfile, "SAT_tot_secs=%d, dir_first_sec_sid=%d, min_size_std_stream=%d" \
+                % (SAT_tot_secs, self.dir_first_sec_sid, self.min_size_std_stream,)
+            print >> logfile, "SSAT_first_sec_sid=%d, SSAT_tot_secs=%d" % (SSAT_first_sec_sid, SSAT_tot_secs,)
+            print >> logfile, "MSAT_first_sec_sid=%d, MSAT_tot_secs=%d" % (MSAT_first_sec_sid, MSAT_tot_secs,)
+        nent = int_floor_div(sec_size, 4) # number of SID entries in a sector
+        fmt = "<%di" % nent
+        trunc_warned = 0
+        #
+        # === build the MSAT ===
+        #
+        MSAT = list(unpack('<109i', mem[76:512]))
+        sid = MSAT_first_sec_sid
+        while sid >= 0:
+            if sid >= mem_data_secs:
+                raise CompDocError(
+                    "MSAT extension: accessing sector %d but only %d in file" % (sid, mem_data_secs)
+                    )
+            offset = 512 + sec_size * sid
+            news = list(unpack(fmt, mem[offset:offset+sec_size]))
+            sid = news.pop()
+            MSAT.extend(news)
+        if DEBUG:
+            print >> logfile, "MSAT: len =", len(MSAT)
+            print >> logfile, MSAT
+        #
+        # === build the SAT ===
+        #
+        self.SAT = []
+        for msid in MSAT:
+            if msid == FREESID: continue
+            if msid >= mem_data_secs:
+                if not trunc_warned:
+                    print >> logfile, "WARNING *** File is truncated, or OLE2 MSAT is corrupt!!"
+                    print >> logfile, \
+                        "INFO: Trying to access sector %d but only %d available" \
+                        % (msid, mem_data_secs)
+                    trunc_warned = 1
+                continue
+            offset = 512 + sec_size * msid
+            news = list(unpack(fmt, mem[offset:offset+sec_size]))
+            self.SAT.extend(news)
+        if DEBUG:
+            print >> logfile, "SAT: len =", len(self.SAT)
+            print >> logfile, self.SAT
+            # print >> logfile, "SAT ",
+            # for i, s in enumerate(self.SAT):
+                # print >> logfile, "entry: %4d offset: %6d, next entry: %4d" % (i, 512 + sec_size * i, s)
+                # print >> logfile, "%d:%d " % (i, s),
+            print
+
+        # === build the directory ===
+        #
+        dbytes = self._get_stream(
+            self.mem, 512, self.SAT, self.sec_size, self.dir_first_sec_sid,
+            name="directory")
+        dirlist = []
+        did = -1
+        for pos in xrange(0, len(dbytes), 128):
+            did += 1
+            dirlist.append(DirNode(did, dbytes[pos:pos+128], 0))
+        self.dirlist = dirlist
+        _build_family_tree(dirlist, 0, dirlist[0].root_DID) # and stand well back ...
+        if DEBUG:
+            for d in dirlist:
+                d.dump(DEBUG)
+        #
+        # === get the SSCS ===
+        #
+        sscs_dir = self.dirlist[0]
+        assert sscs_dir.etype == 5 # root entry
+        if sscs_dir.first_SID < 0 and sscs_dir.tot_size == 0:
+            # Problem reported by Frank Hoffsuemmer: some software was
+            # writing -1 instead of -2 (EOCSID) for the first_SID
+            # when the SCCS was empty. Not having EOCSID caused assertion
+            # failure in _get_stream.
+            # Solution: avoid calling _get_stream in any case when the
+            # SCSS appears to be empty.
+            self.SSCS = ""
+        else:
+            self.SSCS = self._get_stream(
+                self.mem, 512, self.SAT, sec_size, sscs_dir.first_SID,
+                sscs_dir.tot_size, name="SSCS")
+        # if DEBUG: print >> logfile, "SSCS", repr(self.SSCS)
+        #
+        # === build the SSAT ===
+        #
+        self.SSAT = []
+        if SSAT_tot_secs > 0 and sscs_dir.tot_size == 0:
+            print >> logfile, \
+                "WARNING *** OLE2 inconsistency: SSCS size is 0 but SSAT size is non-zero"
+        if sscs_dir.tot_size > 0:
+            sid = SSAT_first_sec_sid
+            nsecs = SSAT_tot_secs
+            while sid >= 0 and nsecs > 0:
+                nsecs -= 1
+                start_pos = 512 + sid * sec_size
+                news = list(unpack(fmt, mem[start_pos:start_pos+sec_size]))
+                self.SSAT.extend(news)
+                sid = self.SAT[sid]
+            # assert SSAT_tot_secs == 0 or sid == EOCSID
+            if DEBUG: print >> logfile, "SSAT last sid %d; remaining sectors %d" % (sid, nsecs)
+            assert nsecs == 0 and sid == EOCSID
+        if DEBUG: print >> logfile, "SSAT", self.SSAT
+
+    def _get_stream(self, mem, base, sat, sec_size, start_sid, size=None, name=''):
+        # print >> self.logfile, "_get_stream", base, sec_size, start_sid, size
+        sectors = []
+        s = start_sid
+        if size is None:
+            # nothing to check against
+            while s >= 0:
+                start_pos = base + s * sec_size
+                sectors.append(mem[start_pos:start_pos+sec_size])
+                try:
+                    s = sat[s]
+                except IndexError:
+                    raise CompDocError(
+                        "OLE2 stream %r: sector allocation table invalid entry (%d)" %
+                        (name, s)
+                        )
+            assert s == EOCSID
+        else:
+            todo = size
+            while s >= 0:
+                start_pos = base + s * sec_size
+                grab = sec_size
+                if grab > todo:
+                    grab = todo
+                todo -= grab
+                sectors.append(mem[start_pos:start_pos+grab])
+                try:
+                    s = sat[s]
+                except IndexError:
+                    raise CompDocError(
+                        "OLE2 stream %r: sector allocation table invalid entry (%d)" %
+                        (name, s)
+                        )
+            assert s == EOCSID
+            if todo != 0:
+                print >> self.logfile, \
+                    "WARNING *** OLE2 stream %r: expected size %d, actual size %d" \
+                    % (name, size, size - todo)
+        return ''.join(sectors)
+
+    def _dir_search(self, path, storage_DID=0):
+        # Return matching DirNode instance, or None
+        head = path[0]
+        tail = path[1:]
+        dl = self.dirlist
+        for child in dl[storage_DID].children:
+            if dl[child].name.lower() == head.lower():
+                et = dl[child].etype
+                if et == 2:
+                    return dl[child]
+                if et == 1:
+                    if not tail:
+                        raise CompDocError("Requested component is a 'storage'")
+                    return self._dir_search(tail, child)
+                dl[child].dump(1)
+                raise CompDocError("Requested stream is not a 'user stream'")
+        return None
+
+    ##
+    # Interrogate the compound document's directory; return the stream as a string if found, otherwise
+    # return None.
+    # @param qname Name of the desired stream e.g. u'Workbook'. Should be in Unicode or convertible thereto.
+
+    def get_named_stream(self, qname):
+        d = self._dir_search(qname.split("/"))
+        if d is None:
+            return None
+        if d.tot_size >= self.min_size_std_stream:
+            return self._get_stream(
+                self.mem, 512, self.SAT, self.sec_size, d.first_SID,
+                d.tot_size, name=qname)
+        else:
+            return self._get_stream(
+                self.SSCS, 0, self.SSAT, self.short_sec_size, d.first_SID,
+                d.tot_size, name=qname + " (from SSCS)")
+
+    ##
+    # Interrogate the compound document's directory.
+    # If the named stream is not found, (None, 0, 0) will be returned.
+    # If the named stream is found and is contiguous within the original byte sequence ("mem")
+    # used when the document was opened,
+    # then (mem, offset_to_start_of_stream, length_of_stream) is returned.
+    # Otherwise a new string is built from the fragments and (new_string, 0, length_of_stream) is returned.
+    # @param qname Name of the desired stream e.g. u'Workbook'. Should be in Unicode or convertible thereto.
+
+    def locate_named_stream(self, qname):
+        d = self._dir_search(qname.split("/"))
+        if d is None:
+            return (None, 0, 0)
+        if d.tot_size >= self.min_size_std_stream:
+            return self._locate_stream(self.mem, 512, self.SAT, self.sec_size, d.first_SID, d.tot_size)
+        else:
+            return (
+                self._get_stream(
+                    self.SSCS, 0, self.SSAT, self.short_sec_size, d.first_SID,
+                    d.tot_size, qname + " (from SSCS)"),
+                0,
+                d.tot_size
+                )
+        return (None, 0, 0) # not found
+
+    def _locate_stream(self, mem, base, sat, sec_size, start_sid, size):
+        # print >> self.logfile, "_locate_stream", base, sec_size, start_sid, size
+        s = start_sid
+        if s < 0:
+            raise CompDocError("_locate_stream: start_sid (%d) is -ve" % start_sid)
+        p = -99 # dummy previous SID
+        start_pos = -9999
+        end_pos = -8888
+        slices = []
+        while s >= 0:
+            if s == p+1:
+                # contiguous sectors
+                end_pos += sec_size
+            else:
+                # start new slice
+                if p >= 0:
+                    # not first time
+                    slices.append((start_pos, end_pos))
+                start_pos = base + s * sec_size
+                end_pos = start_pos + sec_size
+            p = s
+            s = sat[s]
+        assert s == EOCSID
+        # print >> self.logfile, len(slices) + 1, "slices"
+        if not slices:
+            # The stream is contiguous ... just what we like!
+            return (mem, start_pos, size)
+        slices.append((start_pos, end_pos))
+        return (''.join([mem[start_pos:end_pos] for start_pos, end_pos in slices]), 0, size)
+
+# ==========================================================================================

tablib/packages/xlrd/doc/compdoc.html

@@ -0,0 +1,69 @@
+<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Strict//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd'>
+<html>
+<head>
+<meta http-equiv='Content-Type' content='text/html; charset=us-ascii' />
+<title>The compdoc Module</title>
+</head>
+<body>
+<h1>The compdoc Module</h1>
+<p>Implements the minimal functionality required
+to extract a "Workbook" or "Book" stream (as one big string)
+from an OLE2 Compound Document file.
+</p><p>Copyright &#169; 2005-2008 Stephen John Machin, Lingfo Pty Ltd</p>
+<p>This module is part of the xlrd package, which is released under a BSD-style licence.</p>
+<h2>Module Contents</h2>
+<dl>
+<dt><b>CompDoc(mem, logfile=sys.stdout, DEBUG=0)</b> (class) [<a href='#compdoc.CompDoc-class'>#</a>]</dt>
+<dd>
+<p>Compound document handler.</p>
+<dl>
+<dt><i>mem</i></dt>
+<dd>
+The raw contents of the file, as a string, or as an mmap.mmap() object. The
+only operation it needs to support is slicing.</dd>
+</dl><br />
+<p>For more information about this class, see <a href='#compdoc.CompDoc-class'><i>The CompDoc Class</i></a>.</p>
+</dd>
+<dt><a id='compdoc.SIGNATURE-variable' name='compdoc.SIGNATURE-variable'><b>SIGNATURE</b></a> (variable) [<a href='#compdoc.SIGNATURE-variable'>#</a>]</dt>
+<dd>
+<p>Magic cookie that should appear in the first 8 bytes of the file.</p>
+</dd>
+</dl>
+<h2><a id='compdoc.CompDoc-class' name='compdoc.CompDoc-class'>The CompDoc Class</a></h2>
+<dl>
+<dt><b>CompDoc(mem, logfile=sys.stdout, DEBUG=0)</b> (class) [<a href='#compdoc.CompDoc-class'>#</a>]</dt>
+<dd>
+<p>Compound document handler.</p>
+<dl>
+<dt><i>mem</i></dt>
+<dd>
+The raw contents of the file, as a string, or as an mmap.mmap() object. The
+only operation it needs to support is slicing.</dd>
+</dl><br />
+</dd>
+<dt><a id='compdoc.CompDoc.get_named_stream-method' name='compdoc.CompDoc.get_named_stream-method'><b>get_named_stream(qname)</b></a> [<a href='#compdoc.CompDoc.get_named_stream-method'>#</a>]</dt>
+<dd>
+<p>Interrogate the compound document's directory; return the stream as a string if found, otherwise
+return None.</p>
+<dl>
+<dt><i>qname</i></dt>
+<dd>
+Name of the desired stream e.g. u'Workbook'. Should be in Unicode or convertible thereto.</dd>
+</dl><br />
+</dd>
+<dt><a id='compdoc.CompDoc.locate_named_stream-method' name='compdoc.CompDoc.locate_named_stream-method'><b>locate_named_stream(qname)</b></a> [<a href='#compdoc.CompDoc.locate_named_stream-method'>#</a>]</dt>
+<dd>
+<p>Interrogate the compound document's directory.
+If the named stream is not found, (None, 0, 0) will be returned.
+If the named stream is found and is contiguous within the original byte sequence (&quot;mem&quot;)
+used when the document was opened,
+then (mem, offset_to_start_of_stream, length_of_stream) is returned.
+Otherwise a new string is built from the fragments and (new_string, 0, length_of_stream) is returned.</p>
+<dl>
+<dt><i>qname</i></dt>
+<dd>
+Name of the desired stream e.g. u'Workbook'. Should be in Unicode or convertible thereto.</dd>
+</dl><br />
+</dd>
+</dl>
+</body></html>

tablib/packages/xlrd/examples/xlrdnameAPIdemo.py

@@ -0,0 +1,178 @@
+# -*- coding: cp1252 -*-
+
+##
+# Module/script example of the xlrd API for extracting information
+# about named references, named constants, etc.
+#
+# <p>Copyright © 2006 Stephen John Machin, Lingfo Pty Ltd</p>
+# <p>This module is part of the xlrd package, which is released under a BSD-style licence.</p>
+##
+
+import xlrd
+import sys
+import glob
+
+def scope_as_string(book, scope):
+    if 0 <= scope < book.nsheets:
+        return "sheet #%d (%r)" % (scope, book.sheet_names()[scope])
+    if scope == -1:
+        return "Global"
+    if scope == -2:
+        return "Macro/VBA"
+    return "Unknown scope value (%r)" % scope
+
+def do_scope_query(book, scope_strg, show_contents=0, f=sys.stdout):
+    try:
+        qscope = int(scope_strg)
+    except ValueError:
+        if scope_strg == "*":
+            qscope = None # means "all'
+        else:
+            # so assume it's a sheet name ...
+            qscope = book.sheet_names().index(scope_strg)
+            print >> f, "%r => %d" % (scope_strg, qscope)
+    for nobj in book.name_obj_list:
+        if qscope is None or nobj.scope == qscope:
+            show_name_object(book, nobj, show_contents, f)
+
+def show_name_details(book, name, show_contents=0, f=sys.stdout):
+    """
+    book -- Book object obtained from xlrd.open_workbook().
+    name -- The name that's being investigated.
+    show_contents -- 0: Don't; 1: Non-empty cells only; 2: All cells
+    f -- Open output file handle.
+    """
+    name_lcase = name.lower() # Excel names are case-insensitive.
+    nobj_list = book.name_map.get(name_lcase)
+    if not nobj_list:
+        print >> f, "%r: unknown name" % name
+        return
+    for nobj in nobj_list:
+        show_name_object(book, nobj, show_contents, f)
+
+def show_name_details_in_scope(
+    book, name, scope_strg, show_contents=0, f=sys.stdout,
+    ):
+    try:
+        scope = int(scope_strg)
+    except ValueError:
+        # so assume it's a sheet name ...
+        scope = book.sheet_names().index(scope_strg)
+        print >> f, "%r => %d" % (scope_strg, scope)
+    name_lcase = name.lower() # Excel names are case-insensitive.
+    while 1:
+        nobj = book.name_and_scope_map.get((name_lcase, scope))
+        if nobj:
+            break
+        print >> f, "Name %r not found in scope %d" % (name, scope)
+        if scope == -1:
+            return
+        scope = -1 # Try again with global scope
+    print >> f, "Name %r found in scope %d" % (name, scope)
+    show_name_object(book, nobj, show_contents, f)
+
+def showable_cell_value(celltype, cellvalue, datemode):
+    if celltype == xlrd.XL_CELL_DATE:
+        try:
+            showval = xlrd.xldate_as_tuple(cellvalue, datemode)
+        except xlrd.XLDateError:
+            e1, e2 = sys.exc_info()[:2]
+            showval = "%s:%s" % (e1.__name__, e2)
+    elif celltype == xlrd.XL_CELL_ERROR:
+        showval = xlrd.error_text_from_code.get(
+            cellvalue, '<Unknown error code 0x%02x>' % cellvalue)
+    else:
+        showval = cellvalue
+    return showval
+
+def show_name_object(book, nobj, show_contents=0, f=sys.stdout):
+    print >> f, "\nName: %r, scope: %r (%s)" \
+        % (nobj.name, nobj.scope, scope_as_string(book, nobj.scope))
+    res = nobj.result
+    print >> f, "Formula eval result: %r" % res
+    if res is None:
+        return
+    # result should be an instance of the Operand class
+    kind = res.kind
+    value = res.value
+    if kind >= 0:
+        # A scalar, or unknown ... you've seen all there is to see.
+        pass
+    elif kind == xlrd.oREL:
+        # A list of Ref3D objects representing *relative* ranges
+        for i in xrange(len(value)):
+            ref3d = value[i]
+            print >> f, "Range %d: %r ==> %s"% (i, ref3d.coords, xlrd.rangename3drel(book, ref3d))
+    elif kind == xlrd.oREF:
+        # A list of Ref3D objects
+        for i in xrange(len(value)):
+            ref3d = value[i]
+            print >> f, "Range %d: %r ==> %s"% (i, ref3d.coords, xlrd.rangename3d(book, ref3d))
+            if not show_contents:
+                continue
+            datemode = book.datemode
+            for shx in xrange(ref3d.shtxlo, ref3d.shtxhi):
+                sh = book.sheet_by_index(shx)
+                print >> f, "   Sheet #%d (%s)" % (shx, sh.name)
+                rowlim = min(ref3d.rowxhi, sh.nrows)
+                collim = min(ref3d.colxhi, sh.ncols)
+                for rowx in xrange(ref3d.rowxlo, rowlim):
+                    for colx in xrange(ref3d.colxlo, collim):
+                        cty = sh.cell_type(rowx, colx)
+                        if cty == xlrd.XL_CELL_EMPTY and show_contents == 1:
+                            continue
+                        cval = sh.cell_value(rowx, colx)
+                        sval = showable_cell_value(cty, cval, datemode)
+                        print >> f, "      (%3d,%3d) %-5s: %r" \
+                            % (rowx, colx, xlrd.cellname(rowx, colx), sval)
+
+if __name__ == "__main__":
+    def usage():
+        text = """
+usage: xlrdnameAIPdemo.py glob_pattern name scope show_contents
+
+where:
+    "glob_pattern" designates a set of files
+    "name" is a name or '*' (all names)
+    "scope" is -1 (global) or a sheet number
+        or a sheet name or * (all scopes)
+    "show_contents" is one of 0 (no show),
+       1 (only non-empty cells), or 2 (all cells)
+
+Examples (script name and glob_pattern arg omitted for brevity)
+    [Searching through book.name_obj_list]
+    * * 0 lists all names
+    * * 1 lists all names, showing referenced non-empty cells
+    * 1 0 lists all names local to the 2nd sheet
+    * Northern 0 lists all names local to the 'Northern' sheet
+    * -1 0 lists all names with global scope
+    [Initial direct access through book.name_map]
+    Sales * 0 lists all occurrences of "Sales" in any scope
+    [Direct access through book.name_and_scope_map]
+    Revenue -1 0 checks if "Revenue" exists in global scope
+
+"""
+        sys.stdout.write(text)
+    
+    if len(sys.argv) != 5:
+        usage()
+        sys.exit(0)
+    arg_pattern = sys.argv[1] # glob pattern e.g. "foo*.xls"
+    arg_name = sys.argv[2]    # see below
+    arg_scope = sys.argv[3]   # see below
+    arg_show_contents = int(sys.argv[4]) # 0: no show, 1: only non-empty cells,
+                                         # 2: all cells
+    for fname in glob.glob(arg_pattern):
+        book = xlrd.open_workbook(fname)
+        if arg_name == "*":
+            # Examine book.name_obj_list to find all names
+            # in a given scope ("*" => all scopes)
+            do_scope_query(book, arg_scope, arg_show_contents)
+        elif arg_scope == "*":
+            # Using book.name_map to find all usage of a name.
+            show_name_details(book, arg_name, arg_show_contents)
+        else:
+            # Using book.name_and_scope_map to find which if any instances
+            # of a name are visible in the given scope, which can be supplied
+            # as -1 (global) or a sheet number or a sheet name.
+            show_name_details_in_scope(book, arg_name, arg_scope, arg_show_contents)

tablib/packages/xlrd/licences.py

@@ -0,0 +1,77 @@
+# -*- coding: cp1252 -*-
+
+"""
+Portions copyright © 2005-2009, Stephen John Machin, Lingfo Pty Ltd
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice,
+this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+this list of conditions and the following disclaimer in the documentation
+and/or other materials provided with the distribution.
+
+3. None of the names of Stephen John Machin, Lingfo Pty Ltd and any
+contributors may be used to endorse or promote products derived from this
+software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS
+BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+THE POSSIBILITY OF SUCH DAMAGE.
+"""
+
+"""
+/*-
+ * Copyright (c) 2001 David Giffin.
+ * All rights reserved.
+ *
+ * Based on the the Java version: Andrew Khan Copyright (c) 2000.
+ *
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * 3. All advertising materials mentioning features or use of this
+ *    software must display the following acknowledgment:
+ *    "This product includes software developed by
+ *     David Giffin <david@giffin.org>."
+ *
+ * 4. Redistributions of any form whatsoever must retain the following
+ *    acknowledgment:
+ *    "This product includes software developed by
+ *     David Giffin <david@giffin.org>."
+ *
+ * THIS SOFTWARE IS PROVIDED BY DAVID GIFFIN ``AS IS'' AND ANY
+ * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL DAVID GIFFIN OR
+ * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ * OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+"""

tablib/packages/xlrd/timemachine.py

@@ -0,0 +1,44 @@
+# -*- coding: cp1252 -*-
+
+##
+# <p>Copyright © 2006-2008 Stephen John Machin, Lingfo Pty Ltd</p>
+# <p>This module is part of the xlrd package, which is released under a BSD-style licence.</p>
+##
+
+# timemachine.py -- adaptation for earlier Pythons e.g. 2.1
+# usage: from timemachine import *
+
+# 2008-02-08 SJM Generalised method of detecting IronPython
+
+import sys
+
+python_version = sys.version_info[:2] # e.g. version 2.4 -> (2, 4)
+
+CAN_PICKLE_ARRAY = python_version >= (2, 5)
+CAN_SUBCLASS_BUILTIN = python_version >= (2, 2)
+
+if sys.version.find("IronPython") >= 0:
+    array_array = None
+else:
+    from array import array as array_array
+
+if python_version < (2, 2):
+    class object:
+        pass
+    False = 0
+    True = 1
+
+def int_floor_div(x, y):
+    return divmod(x, y)[0]
+
+def intbool(x):
+    if x:
+        return 1
+    return 0
+
+if python_version < (2, 3):
+    def sum(sequence, start=0):
+        tot = start
+        for item in aseq:
+            tot += item
+        return tot

tablib/packages/xlrd/xldate.py

@@ -0,0 +1,171 @@
+# -*- coding: cp1252 -*-
+
+# No part of the content of this file was derived from the works of David Giffin.
+
+##
+# <p>Copyright © 2005-2008 Stephen John Machin, Lingfo Pty Ltd</p>
+# <p>This module is part of the xlrd package, which is released under a BSD-style licence.</p>
+#
+# <p>Provides function(s) for dealing with Microsoft Excel ™ dates.</p>
+##
+
+# 2008-10-18 SJM Fix bug in xldate_from_date_tuple (affected some years after 2099)
+
+# The conversion from days to (year, month, day) starts with
+# an integral "julian day number" aka JDN.
+# FWIW, JDN 0 corresponds to noon on Monday November 24 in Gregorian year -4713.
+# More importantly:
+#    Noon on Gregorian 1900-03-01 (day 61 in the 1900-based system) is JDN 2415080.0
+#    Noon on Gregorian 1904-01-02 (day  1 in the 1904-based system) is JDN 2416482.0
+
+from timemachine import int_floor_div as ifd
+
+_JDN_delta = (2415080 - 61, 2416482 - 1)
+assert _JDN_delta[1] - _JDN_delta[0] == 1462
+
+class XLDateError(ValueError): pass
+
+class XLDateNegative(XLDateError): pass
+class XLDateAmbiguous(XLDateError): pass
+class XLDateTooLarge(XLDateError): pass
+class XLDateBadDatemode(XLDateError): pass
+class XLDateBadTuple(XLDateError): pass
+
+_XLDAYS_TOO_LARGE = (2958466, 2958466 - 1462) # This is equivalent to 10000-01-01
+
+##
+# Convert an Excel number (presumed to represent a date, a datetime or a time) into
+# a tuple suitable for feeding to datetime or mx.DateTime constructors.
+# @param xldate The Excel number
+# @param datemode 0: 1900-based, 1: 1904-based.
+# <br>WARNING: when using this function to
+# interpret the contents of a workbook, you should pass in the Book.datemode
+# attribute of that workbook. Whether
+# the workbook has ever been anywhere near a Macintosh is irrelevant.
+# @return Gregorian (year, month, day, hour, minute, nearest_second).
+# <br>Special case: if 0.0 <= xldate < 1.0, it is assumed to represent a time;
+# (0, 0, 0, hour, minute, second) will be returned.
+# <br>Note: 1904-01-01 is not regarded as a valid date in the datemode 1 system; its "serial number"
+# is zero.
+# @throws XLDateNegative xldate < 0.00
+# @throws XLDateAmbiguous The 1900 leap-year problem (datemode == 0 and 1.0 <= xldate < 61.0)
+# @throws XLDateTooLarge Gregorian year 10000 or later
+# @throws XLDateBadDatemode datemode arg is neither 0 nor 1
+# @throws XLDateError Covers the 4 specific errors
+
+def xldate_as_tuple(xldate, datemode):
+    if datemode not in (0, 1):
+        raise XLDateBadDatemode(datemode)
+    if xldate == 0.00:
+        return (0, 0, 0, 0, 0, 0)
+    if xldate < 0.00:
+        raise XLDateNegative(xldate)
+    xldays = int(xldate)
+    frac = xldate - xldays
+    seconds = int(round(frac * 86400.0))
+    assert 0 <= seconds <= 86400
+    if seconds == 86400:
+        hour = minute = second = 0
+        xldays += 1
+    else:
+        # second = seconds % 60; minutes = seconds // 60
+        minutes, second = divmod(seconds, 60)
+        # minute = minutes % 60; hour    = minutes // 60
+        hour, minute = divmod(minutes, 60)
+    if xldays >= _XLDAYS_TOO_LARGE[datemode]:
+        raise XLDateTooLarge(xldate)
+
+    if xldays == 0:
+        return (0, 0, 0, hour, minute, second)
+
+    if xldays < 61 and datemode == 0:
+        raise XLDateAmbiguous(xldate)
+
+    jdn = xldays + _JDN_delta[datemode]
+    yreg = (ifd(ifd(jdn * 4 + 274277, 146097) * 3, 4) + jdn + 1363) * 4 + 3
+    mp = ifd(yreg % 1461, 4) * 535 + 333
+    d = ifd(mp % 16384, 535) + 1
+    # mp /= 16384
+    mp >>= 14
+    if mp >= 10:
+        return (ifd(yreg, 1461) - 4715, mp - 9, d, hour, minute, second)
+    else:
+        return (ifd(yreg, 1461) - 4716, mp + 3, d, hour, minute, second)
+
+# === conversions from date/time to xl numbers
+
+def _leap(y):
+    if y % 4: return 0
+    if y % 100: return 1
+    if y % 400: return 0
+    return 1
+
+_days_in_month = (None, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31)
+
+##
+# Convert a date tuple (year, month, day) to an Excel date.
+# @param year Gregorian year.
+# @param month 1 <= month <= 12
+# @param day 1 <= day <= last day of that (year, month)
+# @param datemode 0: 1900-based, 1: 1904-based.
+# @throws XLDateAmbiguous The 1900 leap-year problem (datemode == 0 and 1.0 <= xldate < 61.0)
+# @throws XLDateBadDatemode datemode arg is neither 0 nor 1
+# @throws XLDateBadTuple (year, month, day) is too early/late or has invalid component(s)
+# @throws XLDateError Covers the specific errors
+
+def xldate_from_date_tuple((year, month, day), datemode):
+
+    if datemode not in (0, 1):
+        raise XLDateBadDatemode(datemode)
+
+    if year == 0 and month == 0 and day == 0:
+        return 0.00
+
+    if not (1900 <= year <= 9999):
+        raise XLDateBadTuple("Invalid year: %r" % ((year, month, day),))
+    if not (1 <= month <= 12):
+        raise XLDateBadTuple("Invalid month: %r" % ((year, month, day),))
+    if  day < 1 \
+    or (day > _days_in_month[month] and not(day == 29 and month == 2 and _leap(year))):
+        raise XLDateBadTuple("Invalid day: %r" % ((year, month, day),))
+
+    Yp = year + 4716
+    M = month
+    if M <= 2:
+        Yp = Yp - 1
+        Mp = M + 9
+    else:
+        Mp = M - 3
+    jdn = ifd(1461 * Yp, 4) + ifd(979 * Mp + 16, 32) + \
+        day - 1364 - ifd(ifd(Yp + 184, 100) * 3, 4)
+    xldays = jdn - _JDN_delta[datemode]
+    if xldays <= 0:
+        raise XLDateBadTuple("Invalid (year, month, day): %r" % ((year, month, day),))
+    if xldays < 61 and datemode == 0:
+        raise XLDateAmbiguous("Before 1900-03-01: %r" % ((year, month, day),))
+    return float(xldays)
+
+##
+# Convert a time tuple (hour, minute, second) to an Excel "date" value (fraction of a day).
+# @param hour 0 <= hour < 24
+# @param minute 0 <= minute < 60
+# @param second 0 <= second < 60
+# @throws XLDateBadTuple Out-of-range hour, minute, or second
+
+def xldate_from_time_tuple((hour, minute, second)):
+    if 0 <= hour < 24 and 0 <= minute < 60 and 0 <= second < 60:
+        return ((second / 60.0 + minute) / 60.0 + hour) / 24.0
+    raise XLDateBadTuple("Invalid (hour, minute, second): %r" % ((hour, minute, second),))
+
+##
+# Convert a datetime tuple (year, month, day, hour, minute, second) to an Excel date value.
+# For more details, refer to other xldate_from_*_tuple functions.
+# @param datetime_tuple (year, month, day, hour, minute, second)
+# @param datemode 0: 1900-based, 1: 1904-based.
+
+def xldate_from_datetime_tuple(datetime_tuple, datemode):
+    return (
+        xldate_from_date_tuple(datetime_tuple[:3], datemode)
+        +
+        xldate_from_time_tuple(datetime_tuple[3:])
+        )

tablib/packages/xlrd3/biffh.py

@@ -0,0 +1,620 @@
+# Support module for the xlrd3 package.
+#
+# Portions copyright (c) 2005-2008 Stephen John Machin, Lingfo Pty Ltd
+# This module is part of the xlrd package, which is released under a
+# BSD-style licence.
+#
+# 2010-12-08 mozman refactoring for python 3
+# 2008-02-10 SJM BIFF2 BLANK record
+# 2008-02-08 SJM Preparation for Excel 2.0 support
+# 2008-02-02 SJM Added suffixes (_B2, _B2_ONLY, etc) on record names for
+#                biff_dump & biff_count
+# 2007-12-04 SJM Added support for Excel 2.x (BIFF2) files.
+# 2007-09-08 SJM Avoid crash when zero-length Unicode string missing options byte.
+# 2007-04-22 SJM Remove experimental "trimming" facility.
+
+import sys
+from struct import unpack
+
+encoding_from_codepage = {
+    1200 : 'utf_16_le',
+    10000: 'mac_roman',
+    10006: 'mac_greek', # guess
+    10007: 'mac_cyrillic', # guess
+    10029: 'mac_latin2', # guess
+    10079: 'mac_iceland', # guess
+    10081: 'mac_turkish', # guess
+    32768: 'mac_roman',
+    32769: 'cp1252',
+    }
+
+# some more guessing, for Indic scripts
+# codepage 57000 range:
+# 2 Devanagari [0]
+# 3 Bengali [1]
+# 4 Tamil [5]
+# 5 Telegu [6]
+# 6 Assamese [1] c.f. Bengali
+# 7 Oriya [4]
+# 8 Kannada [7]
+# 9 Malayalam [8]
+# 10 Gujarati [3]
+# 11 Gurmukhi [2]
+
+FUN = 0 # unknown
+FDT = 1 # date
+FNU = 2 # number
+FGE = 3 # general
+FTX = 4 # text
+
+DATEFORMAT = FDT
+NUMBERFORMAT = FNU
+
+XL_CELL_EMPTY = 0
+XL_CELL_TEXT = 1
+XL_CELL_NUMBER = 2
+XL_CELL_DATE = 3
+XL_CELL_BOOLEAN = 4
+XL_CELL_ERROR = 5
+XL_CELL_BLANK = 6 # for use in debugging, gathering stats, etc
+
+biff_text_from_num = {
+    0:  "(not BIFF)",
+    20: "2.0",
+    21: "2.1",
+    30: "3",
+    40: "4S",
+    45: "4W",
+    50: "5",
+    70: "7",
+    80: "8",
+    85: "8X",
+}
+
+# This dictionary can be used to produce a text version of the internal codes
+# that Excel uses for error cells. Here are its contents:
+error_text_from_code = {
+    0x00: '#NULL!',  # Intersection of two cell ranges is empty
+    0x07: '#DIV/0!', # Division by zero
+    0x0F: '#VALUE!', # Wrong type of operand
+    0x17: '#REF!',   # Illegal or deleted cell reference
+    0x1D: '#NAME?',  # Wrong function or range name
+    0x24: '#NUM!',   # Value range overflow
+    0x2A: '#N/A!',   # Argument or function not available
+}
+
+BIFF_FIRST_UNICODE = 80
+
+XL_WORKBOOK_GLOBALS = WBKBLOBAL = 0x5
+XL_WORKBOOK_GLOBALS_4W = 0x100
+XL_WORKSHEET = WRKSHEET = 0x10
+
+XL_BOUNDSHEET_WORKSHEET = 0x00
+XL_BOUNDSHEET_CHART     = 0x02
+XL_BOUNDSHEET_VB_MODULE = 0x06
+
+# XL_RK2 = 0x7e
+XL_ARRAY  = 0x0221
+XL_ARRAY2 = 0x0021
+XL_BLANK = 0x0201
+XL_BLANK_B2 = 0x01
+XL_BOF = 0x809
+XL_BOOLERR = 0x205
+XL_BOOLERR_B2 = 0x5
+XL_BOUNDSHEET = 0x85
+XL_BUILTINFMTCOUNT = 0x56
+XL_CF = 0x01B1
+XL_CODEPAGE = 0x42
+XL_COLINFO = 0x7D
+XL_COLUMNDEFAULT = 0x20 # BIFF2 only
+XL_COLWIDTH = 0x24 # BIFF2 only
+XL_CONDFMT = 0x01B0
+XL_CONTINUE = 0x3c
+XL_COUNTRY = 0x8C
+XL_DATEMODE = 0x22
+XL_DEFAULTROWHEIGHT = 0x0225
+XL_DEFCOLWIDTH = 0x55
+XL_DIMENSION = 0x200
+XL_DIMENSION2 = 0x0
+XL_EFONT = 0x45
+XL_EOF = 0x0a
+XL_EXTERNNAME = 0x23
+XL_EXTERNSHEET = 0x17
+XL_EXTSST = 0xff
+XL_FEAT11 = 0x872
+XL_FILEPASS = 0x2f
+XL_FONT = 0x31
+XL_FONT_B3B4 = 0x231
+XL_FORMAT = 0x41e
+XL_FORMAT2 = 0x1E # BIFF2, BIFF3
+XL_FORMULA = 0x6
+XL_FORMULA3 = 0x206
+XL_FORMULA4 = 0x406
+XL_GCW = 0xab
+XL_INDEX = 0x20b
+XL_INTEGER = 0x2 # BIFF2 only
+XL_IXFE = 0x44 # BIFF2 only
+XL_LABEL = 0x204
+XL_LABEL_B2 = 0x04
+XL_LABELRANGES = 0x15f
+XL_LABELSST = 0xfd
+XL_MERGEDCELLS = 0xE5
+XL_MSO_DRAWING = 0x00EC
+XL_MSO_DRAWING_GROUP = 0x00EB
+XL_MSO_DRAWING_SELECTION = 0x00ED
+XL_MULRK = 0xbd
+XL_MULBLANK = 0xbe
+XL_NAME = 0x18
+XL_NOTE = 0x1c
+XL_NUMBER = 0x203
+XL_NUMBER_B2 = 0x3
+XL_OBJ = 0x5D
+XL_PALETTE = 0x92
+XL_RK = 0x27e
+XL_ROW = 0x208
+XL_ROW_B2 = 0x08
+XL_RSTRING = 0xd6
+XL_SHEETHDR = 0x8F # BIFF4W only
+XL_SHEETSOFFSET = 0x8E # BIFF4W only
+XL_SHRFMLA = 0x04bc
+XL_SST = 0xfc
+XL_STANDARDWIDTH = 0x99
+XL_STRING = 0x207
+XL_STRING_B2 = 0x7
+XL_STYLE = 0x293
+XL_SUPBOOK = 0x1AE
+XL_TABLEOP = 0x236
+XL_TABLEOP2 = 0x37
+XL_TABLEOP_B2 = 0x36
+XL_TXO = 0x1b6
+XL_UNCALCED = 0x5e
+XL_UNKNOWN = 0xffff
+XL_WINDOW2 = 0x023E
+XL_WRITEACCESS = 0x5C
+XL_XF = 0xe0
+XL_XF2 = 0x0043 # BIFF2 version of XF record
+XL_XF3 = 0x0243 # BIFF3 version of XF record
+XL_XF4 = 0x0443 # BIFF4 version of XF record
+
+boflen = {
+    0x0809: 8,
+    0x0409: 6,
+    0x0209: 6,
+    0x0009: 4,
+}
+
+bofcodes = (0x0809, 0x0409, 0x0209, 0x0009)
+
+XL_FORMULA_OPCODES = (0x0006, 0x0406, 0x0206)
+
+_cell_opcode_list = (
+    XL_BOOLERR,
+    XL_FORMULA,
+    XL_FORMULA3,
+    XL_FORMULA4,
+    XL_LABEL,
+    XL_LABELSST,
+    XL_MULRK,
+    XL_NUMBER,
+    XL_RK,
+    XL_RSTRING,
+)
+
+biff_rec_name_dict = {
+    0x0000: 'DIMENSIONS_B2',
+    0x0001: 'BLANK_B2',
+    0x0002: 'INTEGER_B2_ONLY',
+    0x0003: 'NUMBER_B2',
+    0x0004: 'LABEL_B2',
+    0x0005: 'BOOLERR_B2',
+    0x0006: 'FORMULA',
+    0x0007: 'STRING_B2',
+    0x0008: 'ROW_B2',
+    0x0009: 'BOF_B2',
+    0x000A: 'EOF',
+    0x000B: 'INDEX_B2_ONLY',
+    0x000C: 'CALCCOUNT',
+    0x000D: 'CALCMODE',
+    0x000E: 'PRECISION',
+    0x000F: 'REFMODE',
+    0x0010: 'DELTA',
+    0x0011: 'ITERATION',
+    0x0012: 'PROTECT',
+    0x0013: 'PASSWORD',
+    0x0014: 'HEADER',
+    0x0015: 'FOOTER',
+    0x0016: 'EXTERNCOUNT',
+    0x0017: 'EXTERNSHEET',
+    0x0018: 'NAME_B2,5+',
+    0x0019: 'WINDOWPROTECT',
+    0x001A: 'VERTICALPAGEBREAKS',
+    0x001B: 'HORIZONTALPAGEBREAKS',
+    0x001C: 'NOTE',
+    0x001D: 'SELECTION',
+    0x001E: 'FORMAT_B2-3',
+    0x001F: 'BUILTINFMTCOUNT_B2',
+    0x0020: 'COLUMNDEFAULT_B2_ONLY',
+    0x0021: 'ARRAY_B2_ONLY',
+    0x0022: 'DATEMODE',
+    0x0023: 'EXTERNNAME',
+    0x0024: 'COLWIDTH_B2_ONLY',
+    0x0025: 'DEFAULTROWHEIGHT_B2_ONLY',
+    0x0026: 'LEFTMARGIN',
+    0x0027: 'RIGHTMARGIN',
+    0x0028: 'TOPMARGIN',
+    0x0029: 'BOTTOMMARGIN',
+    0x002A: 'PRINTHEADERS',
+    0x002B: 'PRINTGRIDLINES',
+    0x002F: 'FILEPASS',
+    0x0031: 'FONT',
+    0x0032: 'FONT2_B2_ONLY',
+    0x0036: 'TABLEOP_B2',
+    0x0037: 'TABLEOP2_B2',
+    0x003C: 'CONTINUE',
+    0x003D: 'WINDOW1',
+    0x003E: 'WINDOW2_B2',
+    0x0040: 'BACKUP',
+    0x0041: 'PANE',
+    0x0042: 'CODEPAGE',
+    0x0043: 'XF_B2',
+    0x0044: 'IXFE_B2_ONLY',
+    0x0045: 'EFONT_B2_ONLY',
+    0x004D: 'PLS',
+    0x0051: 'DCONREF',
+    0x0055: 'DEFCOLWIDTH',
+    0x0056: 'BUILTINFMTCOUNT_B3-4',
+    0x0059: 'XCT',
+    0x005A: 'CRN',
+    0x005B: 'FILESHARING',
+    0x005C: 'WRITEACCESS',
+    0x005D: 'OBJECT',
+    0x005E: 'UNCALCED',
+    0x005F: 'SAVERECALC',
+    0x0063: 'OBJECTPROTECT',
+    0x007D: 'COLINFO',
+    0x007E: 'RK2_mythical_?',
+    0x0080: 'GUTS',
+    0x0081: 'WSBOOL',
+    0x0082: 'GRIDSET',
+    0x0083: 'HCENTER',
+    0x0084: 'VCENTER',
+    0x0085: 'BOUNDSHEET',
+    0x0086: 'WRITEPROT',
+    0x008C: 'COUNTRY',
+    0x008D: 'HIDEOBJ',
+    0x008E: 'SHEETSOFFSET',
+    0x008F: 'SHEETHDR',
+    0x0090: 'SORT',
+    0x0092: 'PALETTE',
+    0x0099: 'STANDARDWIDTH',
+    0x009B: 'FILTERMODE',
+    0x009C: 'FNGROUPCOUNT',
+    0x009D: 'AUTOFILTERINFO',
+    0x009E: 'AUTOFILTER',
+    0x00A0: 'SCL',
+    0x00A1: 'SETUP',
+    0x00AB: 'GCW',
+    0x00BD: 'MULRK',
+    0x00BE: 'MULBLANK',
+    0x00C1: 'MMS',
+    0x00D6: 'RSTRING',
+    0x00D7: 'DBCELL',
+    0x00DA: 'BOOKBOOL',
+    0x00DD: 'SCENPROTECT',
+    0x00E0: 'XF',
+    0x00E1: 'INTERFACEHDR',
+    0x00E2: 'INTERFACEEND',
+    0x00E5: 'MERGEDCELLS',
+    0x00E9: 'BITMAP',
+    0x00EB: 'MSO_DRAWING_GROUP',
+    0x00EC: 'MSO_DRAWING',
+    0x00ED: 'MSO_DRAWING_SELECTION',
+    0x00EF: 'PHONETIC',
+    0x00FC: 'SST',
+    0x00FD: 'LABELSST',
+    0x00FF: 'EXTSST',
+    0x013D: 'TABID',
+    0x015F: 'LABELRANGES',
+    0x0160: 'USESELFS',
+    0x0161: 'DSF',
+    0x01AE: 'SUPBOOK',
+    0x01AF: 'PROTECTIONREV4',
+    0x01B0: 'CONDFMT',
+    0x01B1: 'CF',
+    0x01B2: 'DVAL',
+    0x01B6: 'TXO',
+    0x01B7: 'REFRESHALL',
+    0x01B8: 'HLINK',
+    0x01BC: 'PASSWORDREV4',
+    0x01BE: 'DV',
+    0x01C0: 'XL9FILE',
+    0x01C1: 'RECALCID',
+    0x0200: 'DIMENSIONS',
+    0x0201: 'BLANK',
+    0x0203: 'NUMBER',
+    0x0204: 'LABEL',
+    0x0205: 'BOOLERR',
+    0x0206: 'FORMULA_B3',
+    0x0207: 'STRING',
+    0x0208: 'ROW',
+    0x0209: 'BOF',
+    0x020B: 'INDEX_B3+',
+    0x0218: 'NAME',
+    0x0221: 'ARRAY',
+    0x0223: 'EXTERNNAME_B3-4',
+    0x0225: 'DEFAULTROWHEIGHT',
+    0x0231: 'FONT_B3B4',
+    0x0236: 'TABLEOP',
+    0x023E: 'WINDOW2',
+    0x0243: 'XF_B3',
+    0x027E: 'RK',
+    0x0293: 'STYLE',
+    0x0406: 'FORMULA_B4',
+    0x0409: 'BOF',
+    0x041E: 'FORMAT',
+    0x0443: 'XF_B4',
+    0x04BC: 'SHRFMLA',
+    0x0800: 'QUICKTIP',
+    0x0809: 'BOF',
+    0x0862: 'SHEETLAYOUT',
+    0x0867: 'SHEETPROTECTION',
+    0x0868: 'RANGEPROTECTION',
+}
+
+class XLRDError(Exception):
+    pass
+
+class BaseObject:
+    """
+    Parent of almost all other classes in the package. Defines a common
+    'dump' method for debugging.
+    """
+    _repr_these = []
+
+    def dump(self, f=None, header=None, footer=None, indent=0):
+        """
+        :param f: open file object, to which the dump is written
+        :param header: text to write before the dump
+        :param footer: text to write after the dump
+        :param indent: number of leading spaces (for recursive calls)
+        """
+        if f is None:
+            f = sys.stderr
+        pad = " " * indent
+
+        if header is not None:
+            print(header, file=f)
+
+        for attr, value in sorted(self.__dict__.items()):
+            if getattr(value, 'dump', None) and attr != 'book':
+                value.dump(f,
+                    header="%s%s (%s object):" % (pad, attr, value.__class__.__name__),
+                    indent=indent+4)
+            elif attr not in self._repr_these and \
+                 (isinstance(value, list) or
+                  isinstance(value, dict)):
+                print("%s%s: %s, len = %d" % (pad, attr, type(value), len(value)), file=f)
+            else:
+                print("%s%s: %r" % (pad, attr, value), file=f)
+        if footer is not None:
+            print(footer, file=f)
+
+def fprintf(f, fmt, *vargs):
+    print(fmt.rstrip('\n') % vargs, file=f)
+
+def upkbits(tgt_obj, src, manifest, local_setattr=setattr):
+    for n, mask, attr in manifest:
+        local_setattr(tgt_obj, attr, (src & mask) >> n)
+
+def upkbitsL(tgt_obj, src, manifest, local_setattr=setattr, local_int=int):
+    for n, mask, attr in manifest:
+        local_setattr(tgt_obj, attr, local_int((src & mask) >> n))
+
+def unpack_string(data, pos, encoding, lenlen=1):
+    nchars = unpack('<' + 'BH'[lenlen-1], data[pos:pos+lenlen])[0]
+    pos += lenlen
+    return str(data[pos:pos+nchars], encoding)
+
+def unpack_string_update_pos(data, pos, encoding, lenlen=1, known_len=None):
+    if known_len is not None:
+        # On a NAME record, the length byte is detached from the front of the string.
+        nchars = known_len
+    else:
+        nchars = unpack('<' + 'BH'[lenlen-1], data[pos:pos+lenlen])[0]
+        pos += lenlen
+
+    newpos = pos + nchars
+    return (str(data[pos:newpos], encoding), newpos)
+
+def unpack_unicode(data, pos, lenlen=2):
+    """ Return unicode_strg """
+    nchars = unpack('<' + 'BH'[lenlen-1], data[pos:pos+lenlen])[0]
+    if not nchars:
+        # Ambiguous whether 0-length string should have an "options" byte.
+        # Avoid crash if missing.
+        return ""
+    pos += lenlen
+    options = data[pos]
+    pos += 1
+
+    if options & 0x08: # richtext
+        pos += 2
+
+    if options & 0x04: # phonetic
+        pos += 4
+
+    if options & 0x01:
+        # Uncompressed UTF-16-LE
+        rawstrg = data[pos:pos+2*nchars]
+        strg = str(rawstrg, 'utf_16_le')
+    else:
+        # Note: this is COMPRESSED (not ASCII!) encoding!!!
+        # Merely returning the raw bytes would work OK 99.99% of the time
+        # if the local codepage was cp1252 -- however this would rapidly go pear-shaped
+        # for other codepages so we grit our Anglocentric teeth and return Unicode :-)
+        strg = str(data[pos:pos+nchars], "latin_1")
+    return strg
+
+def unpack_unicode_update_pos(data, pos, lenlen=2, known_len=None):
+    """ Return (unicode_strg, updated value of pos) """
+    if known_len is not None:
+        # On a NAME record, the length byte is detached from the front of the string.
+        nchars = known_len
+    else:
+        nchars = unpack('<' + 'BH'[lenlen-1], data[pos:pos+lenlen])[0]
+        pos += lenlen
+
+    if not nchars and not data[pos:]:
+        # Zero-length string with no options byte
+        return ("", pos)
+
+    options = data[pos]
+    pos += 1
+    phonetic = options & 0x04
+    richtext = options & 0x08
+
+    if richtext:
+        rt = unpack('<H', data[pos:pos+2])[0]
+        pos += 2
+
+    if phonetic:
+        sz = unpack('<i', data[pos:pos+4])[0]
+        pos += 4
+
+    if options & 0x01:
+        # Uncompressed UTF-16-LE
+        strg = str(data[pos:pos+2*nchars], 'utf_16_le')
+        pos += 2*nchars
+    else:
+        # Note: this is COMPRESSED (not ASCII!) encoding!!!
+        strg = str(data[pos:pos+nchars], "latin_1")
+        pos += nchars
+
+    if richtext:
+        pos += 4 * rt
+
+    if phonetic:
+        pos += sz
+
+    return (strg, pos)
+
+def unpack_cell_range_address_list_update_pos(
+    output_list, data, pos, biff_version, addr_size=6):
+    # output_list is updated in situ
+    if biff_version < 80:
+        assert addr_size == 6
+    else:
+        assert addr_size in (6, 8)
+    n, = unpack("<H", data[pos:pos+2])
+    pos += 2
+    if n:
+        fmt = "<HHBB" if addr_size == 6 else "<HHHH"
+        for _unused in range(n):
+            ra, rb, ca, cb = unpack(fmt, data[pos:pos+addr_size])
+            output_list.append((ra, rb+1, ca, cb+1))
+            pos += addr_size
+    return pos
+
+def hex_char_dump(strg, ofs, dlen, base=0, fout=sys.stdout, unnumbered=False):
+    endpos = min(ofs + dlen, len(strg))
+    pos = ofs
+    numbered = not unnumbered
+    num_prefix = ''
+    while pos < endpos:
+        endsub = min(pos + 16, endpos)
+        substrg = strg[pos:endsub]
+        lensub = endsub - pos
+        if lensub <= 0 or lensub != len(substrg):
+            fprintf(
+                sys.stdout,
+                '??? hex_char_dump: ofs=%d dlen=%d base=%d -> endpos=%d pos=%d endsub=%d substrg=%r\n',
+                ofs, dlen, base, endpos, pos, endsub, substrg)
+            break
+        hexd = ''.join(["%02x " % c for c in substrg])
+        chard = ''
+        for c in substrg:
+            if c == ord('\0'):
+                c = '~'
+            elif not (' ' <= chr(c) <= '~'):
+                c = '?'
+            if isinstance(c, int):
+                c = chr(c)
+            chard += c
+        if numbered:
+            num_prefix = "%5d: " %  (base+pos-ofs)
+        fprintf(fout, "%s     %-48s %s\n", num_prefix, hexd, chard)
+        pos = endsub
+
+def biff_dump(mem, stream_offset, stream_len, base=0, fout=sys.stdout,
+              unnumbered=False):
+    pos = stream_offset
+    stream_end = stream_offset + stream_len
+    adj = base - stream_offset
+    dummies = 0
+    numbered = not unnumbered
+    num_prefix = ''
+    while stream_end - pos >= 4:
+        rc, length = unpack('<HH', mem[pos:pos+4])
+        if rc == 0 and length == 0:
+            if mem[pos:] == '\0' * (stream_end - pos):
+                dummies = stream_end - pos
+                savpos = pos
+                pos = stream_end
+                break
+
+            if dummies:
+                dummies += 4
+            else:
+                savpos = pos
+                dummies = 4
+            pos += 4
+        else:
+            if dummies:
+                if numbered:
+                    num_prefix =  "%5d: " % (adj + savpos)
+                fprintf(fout, "%s---- %d zero bytes skipped ----\n",
+                        num_prefix, dummies)
+                dummies = 0
+
+            recname = biff_rec_name_dict.get(rc, '<UNKNOWN>')
+            if numbered:
+                num_prefix = "%5d: " % (adj + pos)
+            fprintf(fout, "%s%04x %s len = %04x (%d)\n",
+                    num_prefix, rc, recname, length, length)
+            pos += 4
+            hex_char_dump(mem, pos, length, adj+pos, fout, unnumbered)
+            pos += length
+    if dummies:
+        if numbered:
+            num_prefix =  "%5d: " % (adj + savpos)
+        fprintf(fout, "%s---- %d zero bytes skipped ----\n", num_prefix, dummies)
+
+    if pos < stream_end:
+        if numbered:
+            num_prefix = "%5d: " % (adj + pos)
+        fprintf(fout, "%s---- Misc bytes at end ----\n", num_prefix)
+        hex_char_dump(mem, pos, stream_end-pos, adj + pos, fout, unnumbered)
+    elif pos > stream_end:
+        fprintf(fout, "Last dumped record has length (%d) that is too large\n", length)
+
+def biff_count_records(mem, stream_offset, stream_len, fout=sys.stdout):
+    pos = stream_offset
+    stream_end = stream_offset + stream_len
+    tally = {}
+    while stream_end - pos >= 4:
+        rc, length = unpack('<HH', mem[pos:pos+4])
+        if rc == 0 and length == 0:
+            if mem[pos:] == '\0' * (stream_end - pos):
+                break
+            recname = "<Dummy (zero)>"
+        else:
+            recname = biff_rec_name_dict.get(rc, None)
+            if recname is None:
+                recname = "Unknown_0x%04X" % rc
+        if recname in tally:
+            tally[recname] += 1
+        else:
+            tally[recname] = 1
+        pos += length + 4
+    for recname, count in sorted(tally.items()):
+        fprintf(fout, "%8d %s", count, recname)

tablib/packages/xlrd3/compdoc.py

@@ -0,0 +1,346 @@
+# Implements the minimal functionality required
+# to extract a "Workbook" or "Book" stream (as one big string)
+# from an OLE2 Compound Document file.
+#
+# Copyright © 2005-2008 Stephen John Machin, Lingfo Pty Ltd
+# This module is part of the xlrd3 package, which is released under a BSD-style licence.
+
+# No part of the content of this file was derived from the works of David Giffin.
+
+# 2008-11-04 SJM Avoid assertion error when -1 used instead of -2 for first_SID of empty SCSS [Frank Hoffsuemmer]
+# 2007-09-08 SJM Warning message if sector sizes are extremely large.
+# 2007-05-07 SJM Meaningful exception instead of IndexError if a SAT (sector allocation table) is corrupted.
+# 2007-04-22 SJM Missing "<" in a struct.unpack call => can't open files on bigendian platforms.
+
+
+import sys
+from struct import unpack
+
+# Magic cookie that should appear in the first 8 bytes of the file.
+SIGNATURE = b"\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1"
+
+EOCSID = -2
+FREESID = -1
+SATSID = -3
+MSATSID = -4
+
+def int_floor_div(x, y):
+    return divmod(x, y)[0]
+
+class CompDocError(Exception):
+    pass
+
+class DirNode(object):
+
+    def __init__(self, DID, dent, DEBUG=0):
+        # dent is the 128-byte directory entry
+        self.DID = DID
+        # (cbufsize, self.etype, self.colour, self.left_DID, self.right_DID,
+        # self.root_DID,
+        # self.first_SID,
+        # self.tot_size) = \
+        #     unpack('<HBBiii16x4x8x8xii4x', dent[64:128])
+        (cbufsize, self.etype, self.colour, self.left_DID, self.right_DID,
+        self.root_DID) = \
+            unpack('<HBBiii', dent[64:80])
+        (self.first_SID, self.tot_size) = \
+            unpack('<ii', dent[116:124])
+        if cbufsize == 0:
+            self.name = ''
+        else:
+            self.name = str(dent[0:cbufsize-2], 'utf_16_le') # omit the trailing U+0000
+        self.children = [] # filled in later
+        self.parent = -1 # indicates orphan; fixed up later
+        self.tsinfo = unpack('<IIII', dent[100:116])
+        if DEBUG:
+            self.dump(DEBUG)
+
+    def dump(self, DEBUG=1):
+        print("DID=%d name=%r etype=%d DIDs(left=%d right=%d root=%d parent=%d kids=%r) first_SID=%d tot_size=%d" \
+            % (self.DID, self.name, self.etype, self.left_DID,
+            self.right_DID, self.root_DID, self.parent, self.children, self.first_SID, self.tot_size))
+        if DEBUG == 2:
+            # cre_lo, cre_hi, mod_lo, mod_hi = tsinfo
+            print("timestamp info", self.tsinfo)
+
+def _build_family_tree(dirlist, parent_DID, child_DID):
+    if child_DID < 0: return
+    _build_family_tree(dirlist, parent_DID, dirlist[child_DID].left_DID)
+    dirlist[parent_DID].children.append(child_DID)
+    dirlist[child_DID].parent = parent_DID
+    _build_family_tree(dirlist, parent_DID, dirlist[child_DID].right_DID)
+    if dirlist[child_DID].etype == 1: # storage
+        _build_family_tree(dirlist, child_DID, dirlist[child_DID].root_DID)
+
+# Compound document handler.
+# @param mem The raw contents of the file, as a string, or as an mmap.mmap() object. The
+# only operation it needs to support is slicing.
+
+class CompDoc(object):
+
+    def __init__(self, mem, logfile=sys.stdout, DEBUG=0):
+        self.logfile = logfile
+        if mem[0:8] != SIGNATURE:
+            raise CompDocError('Not an OLE2 compound document')
+        if mem[28:30] != b'\xFE\xFF':
+            raise CompDocError('Expected "little-endian" marker, found %r' % mem[28:30])
+        revision, version = unpack('<HH', mem[24:28])
+        if DEBUG:
+            print("\nCompDoc format: version=0x%04x revision=0x%04x" % (version, revision), file=logfile)
+        self.mem = mem
+        ssz, sssz = unpack('<HH', mem[30:34])
+        if ssz > 20: # allows for 2**20 bytes i.e. 1MB
+            print("WARNING: sector size (2**%d) is preposterous; assuming 512 and continuing ..." \
+                % ssz, file=logfile)
+            ssz = 9
+        if sssz > ssz:
+            print("WARNING: short stream sector size (2**%d) is preposterous; assuming 64 and continuing ..." \
+                % sssz, file=logfile)
+            sssz = 6
+        self.sec_size = sec_size = 1 << ssz
+        self.short_sec_size = 1 << sssz
+        (
+            SAT_tot_secs, self.dir_first_sec_sid, _unused, self.min_size_std_stream,
+            SSAT_first_sec_sid, SSAT_tot_secs,
+            MSAT_first_sec_sid, MSAT_tot_secs,
+        # ) = unpack('<ii4xiiiii', mem[44:76])
+        ) = unpack('<iiiiiiii', mem[44:76])
+        mem_data_len = len(mem) - 512
+        mem_data_secs, left_over = divmod(mem_data_len, sec_size)
+        if left_over:
+            #### raise CompDocError("Not a whole number of sectors")
+            print("WARNING *** file size (%d) not 512 + multiple of sector size (%d)" \
+                % (len(mem), sec_size), file=logfile)
+        if DEBUG:
+            print('sec sizes', ssz, sssz, sec_size, self.short_sec_size, file=logfile)
+            print("mem data: %d bytes == %d sectors" % (mem_data_len, mem_data_secs), file=logfile)
+            print("SAT_tot_secs=%d, dir_first_sec_sid=%d, min_size_std_stream=%d" \
+                % (SAT_tot_secs, self.dir_first_sec_sid, self.min_size_std_stream,), file=logfile)
+            print("SSAT_first_sec_sid=%d, SSAT_tot_secs=%d" % (SSAT_first_sec_sid, SSAT_tot_secs,), file=logfile)
+            print("MSAT_first_sec_sid=%d, MSAT_tot_secs=%d" % (MSAT_first_sec_sid, MSAT_tot_secs,), file=logfile)
+        nent = int_floor_div(sec_size, 4) # number of SID entries in a sector
+        fmt = "<%di" % nent
+        trunc_warned = 0
+        #
+        # === build the MSAT ===
+        #
+        MSAT = list(unpack('<109i', mem[76:512]))
+        sid = MSAT_first_sec_sid
+        while sid >= 0:
+            if sid >= mem_data_secs:
+                raise CompDocError(
+                    "MSAT extension: accessing sector %d but only %d in file" % (sid, mem_data_secs)
+                    )
+            offset = 512 + sec_size * sid
+            news = list(unpack(fmt, mem[offset:offset+sec_size]))
+            sid = news.pop()
+            MSAT.extend(news)
+        if DEBUG:
+            print("MSAT: len =", len(MSAT), file=logfile)
+            print(MSAT, file=logfile)
+        #
+        # === build the SAT ===
+        #
+        self.SAT = []
+        for msid in MSAT:
+            if msid == FREESID: continue
+            if msid >= mem_data_secs:
+                if not trunc_warned:
+                    print("WARNING *** File is truncated, or OLE2 MSAT is corrupt!!", file=logfile)
+                    print("INFO: Trying to access sector %d but only %d available" \
+                        % (msid, mem_data_secs), file=logfile)
+                    trunc_warned = 1
+                continue
+            offset = 512 + sec_size * msid
+            news = list(unpack(fmt, mem[offset:offset+sec_size]))
+            self.SAT.extend(news)
+        if DEBUG:
+            print("SAT: len =", len(self.SAT), file=logfile)
+            print(self.SAT, file=logfile)
+            # print >> logfile, "SAT ",
+            # for i, s in enumerate(self.SAT):
+                # print >> logfile, "entry: %4d offset: %6d, next entry: %4d" % (i, 512 + sec_size * i, s)
+                # print >> logfile, "%d:%d " % (i, s),
+            print()
+
+        # === build the directory ===
+        #
+        dbytes = self._get_stream(
+            self.mem, 512, self.SAT, self.sec_size, self.dir_first_sec_sid,
+            name="directory")
+        dirlist = []
+        did = -1
+        for pos in range(0, len(dbytes), 128):
+            did += 1
+            dirlist.append(DirNode(did, dbytes[pos:pos+128], 0))
+        self.dirlist = dirlist
+        _build_family_tree(dirlist, 0, dirlist[0].root_DID) # and stand well back ...
+        if DEBUG:
+            for d in dirlist:
+                d.dump(DEBUG)
+        #
+        # === get the SSCS ===
+        #
+        sscs_dir = self.dirlist[0]
+        assert sscs_dir.etype == 5 # root entry
+        if sscs_dir.first_SID < 0 and sscs_dir.tot_size == 0:
+            # Problem reported by Frank Hoffsuemmer: some software was
+            # writing -1 instead of -2 (EOCSID) for the first_SID
+            # when the SCCS was empty. Not having EOCSID caused assertion
+            # failure in _get_stream.
+            # Solution: avoid calling _get_stream in any case when the
+            # SCSS appears to be empty.
+            self.SSCS = ""
+        else:
+            self.SSCS = self._get_stream(
+                self.mem, 512, self.SAT, sec_size, sscs_dir.first_SID,
+                sscs_dir.tot_size, name="SSCS")
+        # if DEBUG: print >> logfile, "SSCS", repr(self.SSCS)
+        #
+        # === build the SSAT ===
+        #
+        self.SSAT = []
+        if SSAT_tot_secs > 0 and sscs_dir.tot_size == 0:
+            print("WARNING *** OLE2 inconsistency: SSCS size is 0 but SSAT size is non-zero", file=logfile)
+        if sscs_dir.tot_size > 0:
+            sid = SSAT_first_sec_sid
+            nsecs = SSAT_tot_secs
+            while sid >= 0 and nsecs > 0:
+                nsecs -= 1
+                start_pos = 512 + sid * sec_size
+                news = list(unpack(fmt, mem[start_pos:start_pos+sec_size]))
+                self.SSAT.extend(news)
+                sid = self.SAT[sid]
+            # assert SSAT_tot_secs == 0 or sid == EOCSID
+            if DEBUG: print("SSAT last sid %d; remaining sectors %d" % (sid, nsecs), file=logfile)
+            assert nsecs == 0 and sid == EOCSID
+        if DEBUG: print("SSAT", self.SSAT, file=logfile)
+
+    def _get_stream(self, mem, base, sat, sec_size, start_sid, size=None, name=''):
+        # print >> self.logfile, "_get_stream", base, sec_size, start_sid, size
+        sectors = []
+        s = start_sid
+        if size is None:
+            # nothing to check against
+            while s >= 0:
+                start_pos = base + s * sec_size
+                sectors.append(mem[start_pos:start_pos+sec_size])
+                try:
+                    s = sat[s]
+                except IndexError:
+                    raise CompDocError(
+                        "OLE2 stream %r: sector allocation table invalid entry (%d)" %
+                        (name, s)
+                        )
+            assert s == EOCSID
+        else:
+            todo = size
+            while s >= 0:
+                start_pos = base + s * sec_size
+                grab = sec_size
+                if grab > todo:
+                    grab = todo
+                todo -= grab
+                sectors.append(mem[start_pos:start_pos+grab])
+                try:
+                    s = sat[s]
+                except IndexError:
+                    raise CompDocError(
+                        "OLE2 stream %r: sector allocation table invalid entry (%d)" %
+                        (name, s)
+                        )
+            assert s == EOCSID
+            if todo != 0:
+                print("WARNING *** OLE2 stream %r: expected size %d, actual size %d" \
+                    % (name, size, size - todo), file=self.logfile)
+        return b''.join(sectors)
+
+    def _dir_search(self, path, storage_DID=0):
+        # Return matching DirNode instance, or None
+        head = path[0]
+        tail = path[1:]
+        dl = self.dirlist
+        for child in dl[storage_DID].children:
+            if dl[child].name.lower() == head.lower():
+                et = dl[child].etype
+                if et == 2:
+                    return dl[child]
+                if et == 1:
+                    if not tail:
+                        raise CompDocError("Requested component is a 'storage'")
+                    return self._dir_search(tail, child)
+                dl[child].dump(1)
+                raise CompDocError("Requested stream is not a 'user stream'")
+        return None
+
+    ##
+    # Interrogate the compound document's directory; return the stream as a string if found, otherwise
+    # return None.
+    # @param qname Name of the desired stream e.g. u'Workbook'. Should be in Unicode or convertible thereto.
+
+    def get_named_stream(self, qname):
+        d = self._dir_search(qname.split("/"))
+        if d is None:
+            return None
+        if d.tot_size >= self.min_size_std_stream:
+            return self._get_stream(
+                self.mem, 512, self.SAT, self.sec_size, d.first_SID,
+                d.tot_size, name=qname)
+        else:
+            return self._get_stream(
+                self.SSCS, 0, self.SSAT, self.short_sec_size, d.first_SID,
+                d.tot_size, name=qname + " (from SSCS)")
+
+    # Interrogate the compound document's directory.
+    # If the named stream is not found, (None, 0, 0) will be returned.
+    # If the named stream is found and is contiguous within the original byte sequence ("mem")
+    # used when the document was opened,
+    # then (mem, offset_to_start_of_stream, length_of_stream) is returned.
+    # Otherwise a new string is built from the fragments and (new_string, 0, length_of_stream) is returned.
+    # @param qname Name of the desired stream e.g. u'Workbook'. Should be in Unicode or convertible thereto.
+
+    def locate_named_stream(self, qname):
+        d = self._dir_search(qname.split("/"))
+        if d is None:
+            return (None, 0, 0)
+        if d.tot_size >= self.min_size_std_stream:
+            return self._locate_stream(self.mem, 512, self.SAT, self.sec_size, d.first_SID, d.tot_size)
+        else:
+            return (
+                self._get_stream(
+                    self.SSCS, 0, self.SSAT, self.short_sec_size, d.first_SID,
+                    d.tot_size, qname + " (from SSCS)"),
+                0,
+                d.tot_size
+                )
+        return (None, 0, 0) # not found
+
+    def _locate_stream(self, mem, base, sat, sec_size, start_sid, size):
+        # print >> self.logfile, "_locate_stream", base, sec_size, start_sid, size
+        s = start_sid
+        if s < 0:
+            raise CompDocError("_locate_stream: start_sid (%d) is -ve" % start_sid)
+        p = -99 # dummy previous SID
+        start_pos = -9999
+        end_pos = -8888
+        slices = []
+        while s >= 0:
+            if s == p+1:
+                # contiguous sectors
+                end_pos += sec_size
+            else:
+                # start new slice
+                if p >= 0:
+                    # not first time
+                    slices.append((start_pos, end_pos))
+                start_pos = base + s * sec_size
+                end_pos = start_pos + sec_size
+            p = s
+            s = sat[s]
+        assert s == EOCSID
+        # print >> self.logfile, len(slices) + 1, "slices"
+        if not slices:
+            # The stream is contiguous ... just what we like!
+            return (mem, start_pos, size)
+        slices.append((start_pos, end_pos))
+        return (b''.join([mem[start_pos:end_pos] for start_pos, end_pos in slices]), 0, size)

tablib/packages/xlrd3/xfcell.py

@@ -0,0 +1,276 @@
+# Author:  mozman <mozman@gmx.at>
+# Purpose: xfcell -- cell with convenient xf function
+# Created: 04.12.2010
+# Copyright (C) 2010, Manfred Moitzi
+# License: BSD-style licence
+
+"""
+The XFCell() object contains the data for one cell.
+
+WARNING: You don't call this class yourself. You access Cell objects
+via methods of the Sheet object(s) that you found in the Book object that
+was returned when you called xlrd.open_workbook("myfile.xls").
+
+Cell objects have four attributes: `ctype` is an int, `value` (which depends
+on `ctype`), `xf_index` and `sheet`, a reference to the containing sheet. If
+**formatting_info** is not enabled when the workbook is opened, xf_index will
+be **None**.
+
+The following table describes the types of cells and how their values
+are represented in Python.
+
+=============== ===== ============ ==========================================
+Type symbol     Const Python value Note
+=============== ===== ============ ==========================================
+XL_CELL_EMPTY   0     ""
+XL_CELL_TEXT    1     str
+XL_CELL_NUMBER  2     float
+XL_CELL_DATE    3     float
+XL_CELL_BOOLEAN 4     int          1 means TRUE, 0 means FALSE
+XL_CELL_ERROR   5     int          representing internal Excel codes; for a
+                                   text representation, refer to the supplied
+                                   dictionary error_text_from_code
+XL_CELL_BLANK   6     ""           this type will appear only when
+                                   open_workbook(..., formatting_info=True)
+                                   is used.
+=============== ===== ============ ==========================================
+"""
+
+import datetime
+
+from .xldate import xldate_as_tuple
+from .biffh import XL_CELL_DATE, BaseObject
+
+class XFCell(BaseObject):
+    """ Extended Cell() class with convenient methods for easy access of cell
+    properties.
+    """
+    __slots__ = ['sheet', 'ctype', 'value', 'xf']
+
+    def __init__(self, ctype, value, xf_index=None, sheet=None):
+        self.sheet = sheet
+        self.ctype = ctype
+        self.value = value
+
+        if xf_index is not None:
+            self.xf = self.book.xf_list[xf_index]
+        else:
+            self.xf = None
+
+    @property
+    def book(self):
+        return self.sheet.book
+
+    @property
+    def has_xf(self):
+        return (self.xf is not None)
+
+    @property
+    def xf_index(self):
+        if self.has_xf:
+            return self.xf.xf_index
+        else:
+            return None
+
+    @property
+    def parent_style(self):
+        return self.book.xf_list[self.xf.parent_style_index]
+
+    @property
+    def is_datetime(self):
+        return self.ctype == XL_CELL_DATE
+
+    @property
+    def has_date(self):
+        if self.is_datetime:
+            return self.value > 1.
+        return False
+
+    def get_color(self, index):
+        return self.book.colour_map[index]
+
+    def datetime(self):
+        """ Returns a datetime.datetime object if cell type is XL_CELL_DATE
+        else raises a TypeError, and raises ValueError if the the cell has
+        not date value (only time value is present).
+        """
+        if self.is_datetime:
+            if self.has_date:
+                date = xldate_as_tuple(self.value, self.book.datemode)
+                return datetime.datetime(*date)
+            else:
+                raise ValueError("Cell has no date value.")
+        else:
+            raise TypeError("Cell is not a XL_CELL_DATE.")
+
+    def date(self):
+        """ Returns a datetime.date object if cell type is XL_CELL_DATE
+        else raises a **TypeError**. Raises **ValueError** if the cell
+        doesn't have a date value (only time value is present).
+        """
+        dt = self.datetime()
+        return dt.date()
+
+    def time(self):
+        """ Returns a datetime.time object if cell type is XL_CELL_DATE else
+        raises a TypeError.
+        """
+        if self.is_datetime:
+            date = xldate_as_tuple(self.value, self.book.datemode)
+            return datetime.time(date[3], date[4], date[5])
+        else:
+            raise TypeError("Cell is not a XL_CELL_DATE.")
+
+    #
+    # access the XFBackground() class
+    #
+
+    @property
+    def background(self):
+        if self.xf.is_style and \
+           self.xf._background_flag == 0:
+            return self.xf.background
+        elif self.xf._background_flag:
+            return self.xf.background
+        else:
+            return self.parent_style.background
+
+    def background_color(self):
+        """ Get cell background-color as 3-tuple. """
+        color_index = self.xf.background.background_colour_index
+        return self.get_color(color_index)
+
+    def fill_pattern(self):
+        return self.xf.background.fill_pattern
+
+    def pattern_color(self):
+        color_index = self.xf.background.pattern_colour_index
+        return self.get_color(color_index)
+
+    #
+    # access the Font() class
+    #
+
+    @property
+    def font_index(self):
+        if self.xf.is_style and \
+           self.xf._font_flag == 0:
+            return self.xf.font_index
+        elif self.xf._font_flag:
+            return self.xf.font_index
+        else:
+            return self.parent_style.font_index
+
+    @property
+    def font(self):
+        """ Get the Font() class. """
+        return self.book.font_list[self.xf.font_index]
+
+    def font_color(self):
+        """ Get cell foreground-color as 3-tuple. """
+        return self.get_color(self.font.colour_index)
+
+    #
+    # access the Format() class
+    #
+
+    @property
+    def format_key(self):
+        if self.xf.is_style and \
+           self.xf._format_flag == 0:
+            return self.xf.format_key
+        elif self.xf._format_flag:
+            return self.xf.format_key
+        else:
+            return self.parent_style.format_key
+
+    @property
+    def format(self):
+        """ Get the Format() class. """
+        return self.book.format_map[self.format_key]
+
+    def format_str(self):
+        """ Get the associated 'format_str'. """
+        return self.format.format_str
+
+    #
+    # access the XFAligment() class
+    #
+
+    @property
+    def alignment(self):
+        if self.xf.is_style and \
+           self.xf._alignment_flag == 0:
+            return self.xf.alignment
+        elif self.xf._alignment_flag:
+            return self.xf.alignment
+        else:
+            return self.parent_style.alignment
+
+    #
+    # access the XFBorder() class
+    #
+
+    @property
+    def border(self):
+        if self.xf.is_style and \
+           self.xf._border_flag == 0:
+            return self.xf.border
+        elif self.xf._border_flag:
+            return self.xf.border
+        else:
+            return self.parent_style.border
+
+    def bordercolors(self):
+        """ Get border color as dict of rgb-color-tuples. """
+        border = self.border
+        return {
+            'top': self.get_color(border.top_colour_index),
+            'bottom': self.get_color(border.bottom_colour_index),
+            'left': self.get_color(border.left_colour_index),
+            'right': self.get_color(border.right_colour_index),
+            'diag': self.get_color(border.diag_colour_index),
+        }
+
+    def borderstyles(self):
+        """ Get border styles as dict of ints. """
+        border = self.border
+        return {
+            'top': border.top_line_style,
+            'bottom': border.bottom_line_style,
+            'left': border.left_line_style,
+            'right': border.right_line_style,
+            'diag': border.diag_line_style,
+        }
+
+    @property
+    def has_up_diag(self):
+        """ Draw a line across the cell from bottom left to top right. """
+        return bool(self.border.diag_up)
+
+    @property
+    def has_down_diag(self):
+        """ Draw a line across the cell from top left to bottom right. """
+        return bool(self.border.diag_down)
+
+    #
+    # access the XFProtection() class
+    #
+
+    @property
+    def protection(self):
+        if self.xf.is_style and \
+           self.xf._protection_flag == 0:
+            return self.xf.protection
+        elif self.xf._protection_flag:
+            return self.xf.protection
+        else:
+            return self.parent_style.protection
+
+    @property
+    def is_cell_locked(self):
+        return bool(self.protection.cell_locked)
+
+    @property
+    def is_formula_hidden(self):
+        return bool(self.protection.cell_locked)

tablib/packages/xlrd3/xfconst.py

@@ -0,0 +1,84 @@
+# Author:  mozman <mozman@gmx.at>
+# Purpose: xfconst -- xf constants
+# Created: 05.12.2010
+# Copyright (C) 2010, Manfred Moitzi
+# License: BSD-style licence
+
+# excelfileformat section 5.115.1 pg. 219
+HOR_ALIGN_GENERAL = 0
+HOR_ALIGN_LEFT = 1
+HOR_ALIGN_CENTRED = 2
+HOR_ALIGN_RIGHT = 3
+HOR_ALIGN_FILLED = 4
+HOR_ALIGN_JUSTIFIED = 5
+HOR_ALIGN_CENTRED_ACROSS_SELECTION = 6
+HOR_ALIGN_DISTRIBUTED = 7
+
+# excelfileformat section 5.115.1 pg. 220
+VERT_ALIGN_TOP = 0
+VERT_ALIGN_CENTRED = 1
+VERT_ALIGN_BOTTOM = 2
+VERT_ALIGN_JUSTIFIED = 3
+VERT_ALIGN_DISTRIBUTED = 4
+
+# excelfileformat section 5.115.1 pg. 220
+ORIENTATION_NONE = 0
+ORIENTATION_STACKED = 1
+ORIENTATION_90_COUNTERCLOCKWISE = 2
+ORIENTATION_90_CLOCKWISE = 3
+
+# excelfileformat section 5.115.1 pg. 220
+ROTATION_NONE = 0
+ROTATION_STACKED = 255
+# other values:
+# Value    Description
+# =======  =================================
+# 1-90     1 to 90 degrees counterclockwise
+# 91-180   1 to 90 degrees clockwise
+
+# excelfileformat section 2.5.11  Line Styles for Cell Borders pg. 24
+LS_NOLINE = 0
+
+# solid line
+LS_THIN = 1
+
+# solid line
+LS_MEDIUM = 2
+
+# - - - - - - - - -
+LS_DASHED = 3
+
+# .................
+LS_DOTTED = 4
+
+# solid line
+LS_THICK = 5
+
+# =================
+LS_DOUBLE = 6
+
+# very thin dotted
+LS_HAIR = 7
+
+# - - - - - - - - -
+LS_MEDIUM_DASHED = 8
+
+# - . - . - . - . -
+LS_THIN_DASH_DOTTED = 9
+
+# - . - . - . - . -
+LS_MEDIUM_DASH_DOTTED = 10
+
+# - .. - .. - .. - .. -
+LS_THIN_DASH_DOT_DOTTED = 11
+
+# - .. - .. - .. - .. -
+LS_MEDIUM_DASH_DOT_DOTTED = 12
+
+# \\\ . \\\ . \\\ . \\\ . \\\
+LS_SLANTED_MEDIUM_DASH_DOTTED = 13
+
+
+
+
+

tablib/packages/xlrd3/xldate.py

@@ -0,0 +1,167 @@
+# No part of the content of this file was derived from the works of David Giffin.
+#
+# Copyright © 2005-2008 Stephen John Machin, Lingfo Pty Ltd
+# This module is part of the xlrd3 package, which is released under a
+# BSD-style licence.
+#
+# Provides function(s) for dealing with Microsoft Excel ™ dates.
+#
+# 2008-10-18 SJM Fix bug in xldate_from_date_tuple (affected some years after 2099)
+#
+# The conversion from days to (year, month, day) starts with
+# an integral "julian day number" aka JDN.
+# FWIW, JDN 0 corresponds to noon on Monday November 24 in Gregorian year -4713.
+# More importantly:
+#    Noon on Gregorian 1900-03-01 (day 61 in the 1900-based system) is JDN 2415080.0
+#    Noon on Gregorian 1904-01-02 (day  1 in the 1904-based system) is JDN 2416482.0
+
+def ifd(x, y):
+    return divmod(x, y)[0]
+
+_JDN_delta = (2415080 - 61, 2416482 - 1)
+assert _JDN_delta[1] - _JDN_delta[0] == 1462
+
+class XLDateError(ValueError): pass
+
+class XLDateNegative(XLDateError): pass
+class XLDateAmbiguous(XLDateError): pass
+class XLDateTooLarge(XLDateError): pass
+class XLDateBadDatemode(XLDateError): pass
+class XLDateBadTuple(XLDateError): pass
+
+_XLDAYS_TOO_LARGE = (2958466, 2958466 - 1462) # This is equivalent to 10000-01-01
+
+# Convert an Excel number (presumed to represent a date, a datetime or a time) into
+# a tuple suitable for feeding to datetime or mx.DateTime constructors.
+# @param xldate The Excel number
+# @param datemode 0: 1900-based, 1: 1904-based.
+# <br>WARNING: when using this function to
+# interpret the contents of a workbook, you should pass in the Book.datemode
+# attribute of that workbook. Whether
+# the workbook has ever been anywhere near a Macintosh is irrelevant.
+# @return Gregorian (year, month, day, hour, minute, nearest_second).
+# <br>Special case: if 0.0 <= xldate < 1.0, it is assumed to represent a time;
+# (0, 0, 0, hour, minute, second) will be returned.
+# <br>Note: 1904-01-01 is not regarded as a valid date in the datemode 1 system; its "serial number"
+# is zero.
+# @throws XLDateNegative xldate < 0.00
+# @throws XLDateAmbiguous The 1900 leap-year problem (datemode == 0 and 1.0 <= xldate < 61.0)
+# @throws XLDateTooLarge Gregorian year 10000 or later
+# @throws XLDateBadDatemode datemode arg is neither 0 nor 1
+# @throws XLDateError Covers the 4 specific errors
+
+def xldate_as_tuple(xldate, datemode):
+    if datemode not in (0, 1):
+        raise XLDateBadDatemode(datemode)
+    if xldate == 0.00:
+        return (0, 0, 0, 0, 0, 0)
+    if xldate < 0.00:
+        raise XLDateNegative(xldate)
+    xldays = int(xldate)
+    frac = xldate - xldays
+    seconds = int(round(frac * 86400.0))
+    assert 0 <= seconds <= 86400
+    if seconds == 86400:
+        hour = minute = second = 0
+        xldays += 1
+    else:
+        # second = seconds % 60; minutes = seconds // 60
+        minutes, second = divmod(seconds, 60)
+        # minute = minutes % 60; hour    = minutes // 60
+        hour, minute = divmod(minutes, 60)
+    if xldays >= _XLDAYS_TOO_LARGE[datemode]:
+        raise XLDateTooLarge(xldate)
+
+    if xldays == 0:
+        return (0, 0, 0, hour, minute, second)
+
+    if xldays < 61 and datemode == 0:
+        raise XLDateAmbiguous(xldate)
+
+    jdn = xldays + _JDN_delta[datemode]
+    yreg = (ifd(ifd(jdn * 4 + 274277, 146097) * 3, 4) + jdn + 1363) * 4 + 3
+    mp = ifd(yreg % 1461, 4) * 535 + 333
+    d = ifd(mp % 16384, 535) + 1
+    # mp /= 16384
+    mp >>= 14
+    if mp >= 10:
+        return (ifd(yreg, 1461) - 4715, mp - 9, d, hour, minute, second)
+    else:
+        return (ifd(yreg, 1461) - 4716, mp + 3, d, hour, minute, second)
+
+# === conversions from date/time to xl numbers
+
+def _leap(y):
+    if y % 4: return 0
+    if y % 100: return 1
+    if y % 400: return 0
+    return 1
+
+_days_in_month = (None, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31)
+
+# Convert a date tuple (year, month, day) to an Excel date.
+# @param year Gregorian year.
+# @param month 1 <= month <= 12
+# @param day 1 <= day <= last day of that (year, month)
+# @param datemode 0: 1900-based, 1: 1904-based.
+# @throws XLDateAmbiguous The 1900 leap-year problem (datemode == 0 and 1.0 <= xldate < 61.0)
+# @throws XLDateBadDatemode datemode arg is neither 0 nor 1
+# @throws XLDateBadTuple (year, month, day) is too early/late or has invalid component(s)
+# @throws XLDateError Covers the specific errors
+
+def xldate_from_date_tuple(datetuple, datemode):
+
+    (year, month, day) = datetuple
+    if datemode not in (0, 1):
+        raise XLDateBadDatemode(datemode)
+
+    if year == 0 and month == 0 and day == 0:
+        return 0.00
+
+    if not (1900 <= year <= 9999):
+        raise XLDateBadTuple("Invalid year: %r" % ((year, month, day),))
+    if not (1 <= month <= 12):
+        raise XLDateBadTuple("Invalid month: %r" % ((year, month, day),))
+    if  day < 1 \
+    or (day > _days_in_month[month] and not(day == 29 and month == 2 and _leap(year))):
+        raise XLDateBadTuple("Invalid day: %r" % ((year, month, day),))
+
+    Yp = year + 4716
+    M = month
+    if M <= 2:
+        Yp = Yp - 1
+        Mp = M + 9
+    else:
+        Mp = M - 3
+    jdn = ifd(1461 * Yp, 4) + ifd(979 * Mp + 16, 32) + \
+        day - 1364 - ifd(ifd(Yp + 184, 100) * 3, 4)
+    xldays = jdn - _JDN_delta[datemode]
+    if xldays <= 0:
+        raise XLDateBadTuple("Invalid (year, month, day): %r" % ((year, month, day),))
+    if xldays < 61 and datemode == 0:
+        raise XLDateAmbiguous("Before 1900-03-01: %r" % ((year, month, day),))
+    return float(xldays)
+
+# Convert a time tuple (hour, minute, second) to an Excel "date" value (fraction of a day).
+# @param hour 0 <= hour < 24
+# @param minute 0 <= minute < 60
+# @param second 0 <= second < 60
+# @throws XLDateBadTuple Out-of-range hour, minute, or second
+
+def xldate_from_time_tuple(timetuple):
+    (hour, minute, second) = timetuple
+    if 0 <= hour < 24 and 0 <= minute < 60 and 0 <= second < 60:
+        return ((second / 60.0 + minute) / 60.0 + hour) / 24.0
+    raise XLDateBadTuple("Invalid (hour, minute, second): %r" % ((hour, minute, second),))
+
+# Convert a datetime tuple (year, month, day, hour, minute, second) to an Excel date value.
+# For more details, refer to other xldate_from_*_tuple functions.
+# @param datetime_tuple (year, month, day, hour, minute, second)
+# @param datemode 0: 1900-based, 1: 1904-based.
+
+def xldate_from_datetime_tuple(datetime_tuple, datemode):
+    return (
+        xldate_from_date_tuple(datetime_tuple[:3], datemode)
+        +
+        xldate_from_time_tuple(datetime_tuple[3:])
+        )

tablib/packages/xlwt/ExcelFormulaLexer.py

@@ -4,7 +4,7 @@ import sys
 from antlr import EOF, CommonToken as Tok, TokenStream, TokenStreamException
 import struct
 import ExcelFormulaParser
-from re import compile as recompile, match, LOCALE, UNICODE, IGNORECASE, VERBOSE
+from re import compile as recompile, match, UNICODE, IGNORECASE, VERBOSE
 
 
 int_const_pattern = r"\d+\b"
@@ -51,7 +51,7 @@ pattern_type_tuples = (
 
 _re = recompile(
     '(' + ')|('.join([i[0] for i in pattern_type_tuples]) + ')',
-    VERBOSE+LOCALE+IGNORECASE)
+    VERBOSE+IGNORECASE)
 
 _toktype = [None] + [i[1] for i in pattern_type_tuples]
 # need dummy at start because re.MatchObject.lastindex counts from 1

tablib/packages/xlwt3/ExcelFormula.py

@@ -4,7 +4,7 @@ from .antlr import ANTLRException
 
 
 class Formula(object):
-    __slots__ = ["__init__",  "__s", "__parser", "__sheet_refs", "__xcall_refs"]
+    __slots__ = ["__s", "__parser", "__sheet_refs", "__xcall_refs"]
 
 
     def __init__(self, s):

tablib/packages/xlwt3/ExcelFormulaLexer.py

@@ -2,7 +2,7 @@ import sys
 from .antlr import EOF, CommonToken as Tok, TokenStream, TokenStreamException
 import struct
 from . import ExcelFormulaParser
-from re import compile as recompile, match, LOCALE, UNICODE, IGNORECASE, VERBOSE
+from re import compile as recompile, match, UNICODE, IGNORECASE, VERBOSE
 
 
 int_const_pattern = r"\d+\b"
@@ -49,7 +49,7 @@ pattern_type_tuples = (
 
 _re = recompile(
     '(' + ')|('.join([i[0] for i in pattern_type_tuples]) + ')',
-    VERBOSE+LOCALE+IGNORECASE)
+    VERBOSE+IGNORECASE)
 
 _toktype = [None] + [i[1] for i in pattern_type_tuples]
 # need dummy at start because re.MatchObject.lastindex counts from 1

tablib/packages/yaml/composer.py

@@ -1,15 +1,16 @@
-
 __all__ = ['Composer', 'ComposerError']
 
 from error import MarkedYAMLError
-from events import *
-from nodes import *
+from events import StreamEndEvent, StreamStartEvent, AliasEvent, SequenceEndEvent, SequenceStartEvent, MappingEndEvent,\
+    MappingStartEvent, ScalarEvent
+from nodes import MappingNode, ScalarNode, SequenceNode
+
 
 class ComposerError(MarkedYAMLError):
     pass
 
-class Composer(object):
 
+class Composer(object):
     def __init__(self):
         self.anchors = {}
 
@@ -39,8 +40,8 @@ class Composer(object):
         if not self.check_event(StreamEndEvent):
             event = self.get_event()
             raise ComposerError("expected a single document in the stream",
-                    document.start_mark, "but found another document",
-                    event.start_mark)
+                                document.start_mark, "but found another document",
+                                event.start_mark)
 
         # Drop the STREAM-END event.
         self.get_event()
@@ -66,15 +67,14 @@ class Composer(object):
             anchor = event.anchor
             if anchor not in self.anchors:
                 raise ComposerError(None, None, "found undefined alias %r"
-                        % anchor.encode('utf-8'), event.start_mark)
+                                    % anchor.encode('utf-8'), event.start_mark)
             return self.anchors[anchor]
         event = self.peek_event()
         anchor = event.anchor
-        if anchor is not None:
-            if anchor in self.anchors:
+        if anchor is not None and anchor in self.anchors:
                 raise ComposerError("found duplicate anchor %r; first occurence"
-                        % anchor.encode('utf-8'), self.anchors[anchor].start_mark,
-                        "second occurence", event.start_mark)
+                                    % anchor.encode('utf-8'), self.anchors[anchor].start_mark,
+                                    "second occurence", event.start_mark)
         self.descend_resolver(parent, index)
         if self.check_event(ScalarEvent):
             node = self.compose_scalar_node(anchor)
@@ -91,7 +91,7 @@ class Composer(object):
         if tag is None or tag == u'!':
             tag = self.resolve(ScalarNode, event.value, event.implicit)
         node = ScalarNode(tag, event.value,
-                event.start_mark, event.end_mark, style=event.style)
+                          event.start_mark, event.end_mark, style=event.style)
         if anchor is not None:
             self.anchors[anchor] = node
         return node
@@ -102,8 +102,8 @@ class Composer(object):
         if tag is None or tag == u'!':
             tag = self.resolve(SequenceNode, None, start_event.implicit)
         node = SequenceNode(tag, [],
-                start_event.start_mark, None,
-                flow_style=start_event.flow_style)
+                            start_event.start_mark, None,
+                            flow_style=start_event.flow_style)
         if anchor is not None:
             self.anchors[anchor] = node
         index = 0
@@ -120,20 +120,19 @@ class Composer(object):
         if tag is None or tag == u'!':
             tag = self.resolve(MappingNode, None, start_event.implicit)
         node = MappingNode(tag, [],
-                start_event.start_mark, None,
-                flow_style=start_event.flow_style)
+                           start_event.start_mark, None,
+                           flow_style=start_event.flow_style)
         if anchor is not None:
             self.anchors[anchor] = node
         while not self.check_event(MappingEndEvent):
-            #key_event = self.peek_event()
+            # key_event = self.peek_event()
             item_key = self.compose_node(node, None)
-            #if item_key in node.value:
+            # if item_key in node.value:
             #    raise ComposerError("while composing a mapping", start_event.start_mark,
             #            "found duplicate key", key_event.start_mark)
             item_value = self.compose_node(node, item_key)
-            #node.value[item_key] = item_value
+            # node.value[item_key] = item_value
             node.value.append((item_key, item_value))
         end_event = self.get_event()
         node.end_mark = end_event.end_mark
         return node
-

tablib/packages/yaml/constructor.py

@@ -1,6 +1,5 @@
-
 __all__ = ['BaseConstructor', 'SafeConstructor', 'Constructor',
-    'ConstructorError']
+           'ConstructorError']
 
 from error import *
 from nodes import *
@@ -12,13 +11,17 @@ try:
 except NameError:
     from sets import Set as set
 
-import binascii, re, sys, types
+import binascii
+import re
+import sys
+import types
+
 
 class ConstructorError(MarkedYAMLError):
     pass
 
-class BaseConstructor(object):
 
+class BaseConstructor(object):
     yaml_constructors = {}
     yaml_multi_constructors = {}
 
@@ -65,7 +68,7 @@ class BaseConstructor(object):
             return self.constructed_objects[node]
         if node in self.recursive_objects:
             raise ConstructorError(None, None,
-                    "found unconstructable recursive node", node.start_mark)
+                                   "found unconstructable recursive node", node.start_mark)
         self.recursive_objects[node] = None
         constructor = None
         tag_suffix = None
@@ -110,23 +113,23 @@ class BaseConstructor(object):
     def construct_scalar(self, node):
         if not isinstance(node, ScalarNode):
             raise ConstructorError(None, None,
-                    "expected a scalar node, but found %s" % node.id,
-                    node.start_mark)
+                                   "expected a scalar node, but found %s" % node.id,
+                                   node.start_mark)
         return node.value
 
     def construct_sequence(self, node, deep=False):
         if not isinstance(node, SequenceNode):
             raise ConstructorError(None, None,
-                    "expected a sequence node, but found %s" % node.id,
-                    node.start_mark)
+                                   "expected a sequence node, but found %s" % node.id,
+                                   node.start_mark)
         return [self.construct_object(child, deep=deep)
                 for child in node.value]
 
     def construct_mapping(self, node, deep=False):
         if not isinstance(node, MappingNode):
             raise ConstructorError(None, None,
-                    "expected a mapping node, but found %s" % node.id,
-                    node.start_mark)
+                                   "expected a mapping node, but found %s" % node.id,
+                                   node.start_mark)
         mapping = {}
         for key_node, value_node in node.value:
             key = self.construct_object(key_node, deep=deep)
@@ -134,7 +137,7 @@ class BaseConstructor(object):
                 hash(key)
             except TypeError, exc:
                 raise ConstructorError("while constructing a mapping", node.start_mark,
-                        "found unacceptable key (%s)" % exc, key_node.start_mark)
+                                       "found unacceptable key (%s)" % exc, key_node.start_mark)
             value = self.construct_object(value_node, deep=deep)
             mapping[key] = value
         return mapping
@@ -142,8 +145,8 @@ class BaseConstructor(object):
     def construct_pairs(self, node, deep=False):
         if not isinstance(node, MappingNode):
             raise ConstructorError(None, None,
-                    "expected a mapping node, but found %s" % node.id,
-                    node.start_mark)
+                                   "expected a mapping node, but found %s" % node.id,
+                                   node.start_mark)
         pairs = []
         for key_node, value_node in node.value:
             key = self.construct_object(key_node, deep=deep)
@@ -155,16 +158,18 @@ class BaseConstructor(object):
         if not 'yaml_constructors' in cls.__dict__:
             cls.yaml_constructors = cls.yaml_constructors.copy()
         cls.yaml_constructors[tag] = constructor
+
     add_constructor = classmethod(add_constructor)
 
     def add_multi_constructor(cls, tag_prefix, multi_constructor):
         if not 'yaml_multi_constructors' in cls.__dict__:
             cls.yaml_multi_constructors = cls.yaml_multi_constructors.copy()
         cls.yaml_multi_constructors[tag_prefix] = multi_constructor
+
     add_multi_constructor = classmethod(add_multi_constructor)
 
-class SafeConstructor(BaseConstructor):
 
+class SafeConstructor(BaseConstructor):
     def construct_scalar(self, node):
         if isinstance(node, MappingNode):
             for key_node, value_node in node.value:
@@ -187,9 +192,9 @@ class SafeConstructor(BaseConstructor):
                     for subnode in value_node.value:
                         if not isinstance(subnode, MappingNode):
                             raise ConstructorError("while constructing a mapping",
-                                    node.start_mark,
-                                    "expected a mapping for merging, but found %s"
-                                    % subnode.id, subnode.start_mark)
+                                                   node.start_mark,
+                                                   "expected a mapping for merging, but found %s"
+                                                   % subnode.id, subnode.start_mark)
                         self.flatten_mapping(subnode)
                         submerge.append(subnode.value)
                     submerge.reverse()
@@ -197,8 +202,8 @@ class SafeConstructor(BaseConstructor):
                         merge.extend(value)
                 else:
                     raise ConstructorError("while constructing a mapping", node.start_mark,
-                            "expected a mapping or list of mappings for merging, but found %s"
-                            % value_node.id, value_node.start_mark)
+                                           "expected a mapping or list of mappings for merging, but found %s"
+                                           % value_node.id, value_node.start_mark)
             elif key_node.tag == u'tag:yaml.org,2002:value':
                 key_node.tag = u'tag:yaml.org,2002:str'
                 index += 1
@@ -217,12 +222,12 @@ class SafeConstructor(BaseConstructor):
         return None
 
     bool_values = {
-        u'yes':     True,
-        u'no':      False,
-        u'true':    True,
-        u'false':   False,
-        u'on':      True,
-        u'off':     False,
+        u'yes': True,
+        u'no': False,
+        u'true': True,
+        u'false': False,
+        u'on': True,
+        u'off': False,
     }
 
     def construct_yaml_bool(self, node):
@@ -240,27 +245,27 @@ class SafeConstructor(BaseConstructor):
         if value == '0':
             return 0
         elif value.startswith('0b'):
-            return sign*int(value[2:], 2)
+            return sign * int(value[2:], 2)
         elif value.startswith('0x'):
-            return sign*int(value[2:], 16)
+            return sign * int(value[2:], 16)
         elif value[0] == '0':
-            return sign*int(value, 8)
+            return sign * int(value, 8)
         elif ':' in value:
             digits = [int(part) for part in value.split(':')]
             digits.reverse()
             base = 1
             value = 0
             for digit in digits:
-                value += digit*base
+                value += digit * base
                 base *= 60
-            return sign*value
+            return sign * value
         else:
-            return sign*int(value)
+            return sign * int(value)
 
     inf_value = 1e300
-    while inf_value != inf_value*inf_value:
+    while inf_value != inf_value * inf_value:
         inf_value *= inf_value
-    nan_value = -inf_value/inf_value   # Trying to make a quiet NaN (like C99).
+    nan_value = -inf_value / inf_value  # Trying to make a quiet NaN (like C99).
 
     def construct_yaml_float(self, node):
         value = str(self.construct_scalar(node))
@@ -271,7 +276,7 @@ class SafeConstructor(BaseConstructor):
         if value[0] in '+-':
             value = value[1:]
         if value == '.inf':
-            return sign*self.inf_value
+            return sign * self.inf_value
         elif value == '.nan':
             return self.nan_value
         elif ':' in value:
@@ -280,11 +285,11 @@ class SafeConstructor(BaseConstructor):
             base = 1
             value = 0.0
             for digit in digits:
-                value += digit*base
+                value += digit * base
                 base *= 60
-            return sign*value
+            return sign * value
         else:
-            return sign*float(value)
+            return sign * float(value)
 
     def construct_yaml_binary(self, node):
         value = self.construct_scalar(node)
@@ -292,10 +297,10 @@ class SafeConstructor(BaseConstructor):
             return str(value).decode('base64')
         except (binascii.Error, UnicodeEncodeError), exc:
             raise ConstructorError(None, None,
-                    "failed to decode base64 data: %s" % exc, node.start_mark) 
+                                   "failed to decode base64 data: %s" % exc, node.start_mark)
 
     timestamp_regexp = re.compile(
-            ur'''^(?P<year>[0-9][0-9][0-9][0-9])
+        ur'''^(?P<year>[0-9][0-9][0-9][0-9])
                 -(?P<month>[0-9][0-9]?)
                 -(?P<day>[0-9][0-9]?)
                 (?:(?:[Tt]|[ \t]+)
@@ -343,16 +348,16 @@ class SafeConstructor(BaseConstructor):
         yield omap
         if not isinstance(node, SequenceNode):
             raise ConstructorError("while constructing an ordered map", node.start_mark,
-                    "expected a sequence, but found %s" % node.id, node.start_mark)
+                                   "expected a sequence, but found %s" % node.id, node.start_mark)
         for subnode in node.value:
             if not isinstance(subnode, MappingNode):
                 raise ConstructorError("while constructing an ordered map", node.start_mark,
-                        "expected a mapping of length 1, but found %s" % subnode.id,
-                        subnode.start_mark)
+                                       "expected a mapping of length 1, but found %s" % subnode.id,
+                                       subnode.start_mark)
             if len(subnode.value) != 1:
                 raise ConstructorError("while constructing an ordered map", node.start_mark,
-                        "expected a single mapping item, but found %d items" % len(subnode.value),
-                        subnode.start_mark)
+                                       "expected a single mapping item, but found %d items" % len(subnode.value),
+                                       subnode.start_mark)
             key_node, value_node = subnode.value[0]
             key = self.construct_object(key_node)
             value = self.construct_object(value_node)
@@ -364,16 +369,16 @@ class SafeConstructor(BaseConstructor):
         yield pairs
         if not isinstance(node, SequenceNode):
             raise ConstructorError("while constructing pairs", node.start_mark,
-                    "expected a sequence, but found %s" % node.id, node.start_mark)
+                                   "expected a sequence, but found %s" % node.id, node.start_mark)
         for subnode in node.value:
             if not isinstance(subnode, MappingNode):
                 raise ConstructorError("while constructing pairs", node.start_mark,
-                        "expected a mapping of length 1, but found %s" % subnode.id,
-                        subnode.start_mark)
+                                       "expected a mapping of length 1, but found %s" % subnode.id,
+                                       subnode.start_mark)
             if len(subnode.value) != 1:
                 raise ConstructorError("while constructing pairs", node.start_mark,
-                        "expected a single mapping item, but found %d items" % len(subnode.value),
-                        subnode.start_mark)
+                                       "expected a single mapping item, but found %d items" % len(subnode.value),
+                                       subnode.start_mark)
             key_node, value_node = subnode.value[0]
             key = self.construct_object(key_node)
             value = self.construct_object(value_node)
@@ -415,62 +420,63 @@ class SafeConstructor(BaseConstructor):
 
     def construct_undefined(self, node):
         raise ConstructorError(None, None,
-                "could not determine a constructor for the tag %r" % node.tag.encode('utf-8'),
-                node.start_mark)
+                               "could not determine a constructor for the tag %r" % node.tag.encode('utf-8'),
+                               node.start_mark)
+
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:null',
-        SafeConstructor.construct_yaml_null)
+    u'tag:yaml.org,2002:null',
+    SafeConstructor.construct_yaml_null)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:bool',
-        SafeConstructor.construct_yaml_bool)
+    u'tag:yaml.org,2002:bool',
+    SafeConstructor.construct_yaml_bool)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:int',
-        SafeConstructor.construct_yaml_int)
+    u'tag:yaml.org,2002:int',
+    SafeConstructor.construct_yaml_int)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:float',
-        SafeConstructor.construct_yaml_float)
+    u'tag:yaml.org,2002:float',
+    SafeConstructor.construct_yaml_float)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:binary',
-        SafeConstructor.construct_yaml_binary)
+    u'tag:yaml.org,2002:binary',
+    SafeConstructor.construct_yaml_binary)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:timestamp',
-        SafeConstructor.construct_yaml_timestamp)
+    u'tag:yaml.org,2002:timestamp',
+    SafeConstructor.construct_yaml_timestamp)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:omap',
-        SafeConstructor.construct_yaml_omap)
+    u'tag:yaml.org,2002:omap',
+    SafeConstructor.construct_yaml_omap)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:pairs',
-        SafeConstructor.construct_yaml_pairs)
+    u'tag:yaml.org,2002:pairs',
+    SafeConstructor.construct_yaml_pairs)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:set',
-        SafeConstructor.construct_yaml_set)
+    u'tag:yaml.org,2002:set',
+    SafeConstructor.construct_yaml_set)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:str',
-        SafeConstructor.construct_yaml_str)
+    u'tag:yaml.org,2002:str',
+    SafeConstructor.construct_yaml_str)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:seq',
-        SafeConstructor.construct_yaml_seq)
+    u'tag:yaml.org,2002:seq',
+    SafeConstructor.construct_yaml_seq)
 
 SafeConstructor.add_constructor(
-        u'tag:yaml.org,2002:map',
-        SafeConstructor.construct_yaml_map)
+    u'tag:yaml.org,2002:map',
+    SafeConstructor.construct_yaml_map)
 
 SafeConstructor.add_constructor(None,
-        SafeConstructor.construct_undefined)
+                                SafeConstructor.construct_undefined)
+
 
 class Constructor(SafeConstructor):
-
     def construct_python_str(self, node):
         return self.construct_scalar(node).encode('utf-8')
 
@@ -481,7 +487,7 @@ class Constructor(SafeConstructor):
         return long(self.construct_yaml_int(node))
 
     def construct_python_complex(self, node):
-       return complex(self.construct_scalar(node))
+        return complex(self.construct_scalar(node))
 
     def construct_python_tuple(self, node):
         return tuple(self.construct_sequence(node))
@@ -489,21 +495,21 @@ class Constructor(SafeConstructor):
     def find_python_module(self, name, mark):
         if not name:
             raise ConstructorError("while constructing a Python module", mark,
-                    "expected non-empty name appended to the tag", mark)
+                                   "expected non-empty name appended to the tag", mark)
         try:
             __import__(name)
         except ImportError, exc:
             raise ConstructorError("while constructing a Python module", mark,
-                    "cannot find module %r (%s)" % (name.encode('utf-8'), exc), mark)
+                                   "cannot find module %r (%s)" % (name.encode('utf-8'), exc), mark)
         return sys.modules[name]
 
     def find_python_name(self, name, mark):
         if not name:
             raise ConstructorError("while constructing a Python object", mark,
-                    "expected non-empty name appended to the tag", mark)
+                                   "expected non-empty name appended to the tag", mark)
         if u'.' in name:
             # Python 2.4 only
-            #module_name, object_name = name.rsplit('.', 1)
+            # module_name, object_name = name.rsplit('.', 1)
             items = name.split('.')
             object_name = items.pop()
             module_name = '.'.join(items)
@@ -514,40 +520,41 @@ class Constructor(SafeConstructor):
             __import__(module_name)
         except ImportError, exc:
             raise ConstructorError("while constructing a Python object", mark,
-                    "cannot find module %r (%s)" % (module_name.encode('utf-8'), exc), mark)
+                                   "cannot find module %r (%s)" % (module_name.encode('utf-8'), exc), mark)
         module = sys.modules[module_name]
         if not hasattr(module, object_name):
             raise ConstructorError("while constructing a Python object", mark,
-                    "cannot find %r in the module %r" % (object_name.encode('utf-8'),
-                        module.__name__), mark)
+                                   "cannot find %r in the module %r" % (object_name.encode('utf-8'),
+                                                                        module.__name__), mark)
         return getattr(module, object_name)
 
     def construct_python_name(self, suffix, node):
         value = self.construct_scalar(node)
         if value:
             raise ConstructorError("while constructing a Python name", node.start_mark,
-                    "expected the empty value, but found %r" % value.encode('utf-8'),
-                    node.start_mark)
+                                   "expected the empty value, but found %r" % value.encode('utf-8'),
+                                   node.start_mark)
         return self.find_python_name(suffix, node.start_mark)
 
     def construct_python_module(self, suffix, node):
         value = self.construct_scalar(node)
         if value:
             raise ConstructorError("while constructing a Python module", node.start_mark,
-                    "expected the empty value, but found %r" % value.encode('utf-8'),
-                    node.start_mark)
+                                   "expected the empty value, but found %r" % value.encode('utf-8'),
+                                   node.start_mark)
         return self.find_python_module(suffix, node.start_mark)
 
-    class classobj: pass
+    class classobj:
+        pass
 
     def make_python_instance(self, suffix, node,
-            args=None, kwds=None, newobj=False):
+                             args=None, kwds=None, newobj=False):
         if not args:
             args = []
         if not kwds:
             kwds = {}
         cls = self.find_python_name(suffix, node.start_mark)
-        if newobj and isinstance(cls, type(self.classobj))  \
+        if newobj and isinstance(cls, type(self.classobj)) \
                 and not args and not kwds:
             instance = self.classobj()
             instance.__class__ = cls
@@ -618,6 +625,7 @@ class Constructor(SafeConstructor):
     def construct_python_object_new(self, suffix, node):
         return self.construct_python_object_apply(suffix, node, newobj=True)
 
+
 Constructor.add_constructor(
     u'tag:yaml.org,2002:python/none',
     Constructor.construct_yaml_null)
@@ -681,4 +689,3 @@ Constructor.add_multi_constructor(
 Constructor.add_multi_constructor(
     u'tag:yaml.org,2002:python/object/new:',
     Constructor.construct_python_object_new)
-

tablib/packages/yaml3/composer.py

@@ -1,15 +1,15 @@
-
 __all__ = ['Composer', 'ComposerError']
 
 from .error import MarkedYAMLError
 from .events import *
 from .nodes import *
 
+
 class ComposerError(MarkedYAMLError):
     pass
 
-class Composer:
 
+class Composer:
     def __init__(self):
         self.anchors = {}
 
@@ -39,8 +39,8 @@ class Composer:
         if not self.check_event(StreamEndEvent):
             event = self.get_event()
             raise ComposerError("expected a single document in the stream",
-                    document.start_mark, "but found another document",
-                    event.start_mark)
+                                document.start_mark, "but found another document",
+                                event.start_mark)
 
         # Drop the STREAM-END event.
         self.get_event()
@@ -66,15 +66,15 @@ class Composer:
             anchor = event.anchor
             if anchor not in self.anchors:
                 raise ComposerError(None, None, "found undefined alias %r"
-                        % anchor, event.start_mark)
+                                    % anchor, event.start_mark)
             return self.anchors[anchor]
         event = self.peek_event()
         anchor = event.anchor
         if anchor is not None:
             if anchor in self.anchors:
                 raise ComposerError("found duplicate anchor %r; first occurence"
-                        % anchor, self.anchors[anchor].start_mark,
-                        "second occurence", event.start_mark)
+                                    % anchor, self.anchors[anchor].start_mark,
+                                    "second occurence", event.start_mark)
         self.descend_resolver(parent, index)
         if self.check_event(ScalarEvent):
             node = self.compose_scalar_node(anchor)
@@ -91,7 +91,7 @@ class Composer:
         if tag is None or tag == '!':
             tag = self.resolve(ScalarNode, event.value, event.implicit)
         node = ScalarNode(tag, event.value,
-                event.start_mark, event.end_mark, style=event.style)
+                          event.start_mark, event.end_mark, style=event.style)
         if anchor is not None:
             self.anchors[anchor] = node
         return node
@@ -102,8 +102,8 @@ class Composer:
         if tag is None or tag == '!':
             tag = self.resolve(SequenceNode, None, start_event.implicit)
         node = SequenceNode(tag, [],
-                start_event.start_mark, None,
-                flow_style=start_event.flow_style)
+                            start_event.start_mark, None,
+                            flow_style=start_event.flow_style)
         if anchor is not None:
             self.anchors[anchor] = node
         index = 0
@@ -120,20 +120,19 @@ class Composer:
         if tag is None or tag == '!':
             tag = self.resolve(MappingNode, None, start_event.implicit)
         node = MappingNode(tag, [],
-                start_event.start_mark, None,
-                flow_style=start_event.flow_style)
+                           start_event.start_mark, None,
+                           flow_style=start_event.flow_style)
         if anchor is not None:
             self.anchors[anchor] = node
         while not self.check_event(MappingEndEvent):
-            #key_event = self.peek_event()
+            # key_event = self.peek_event()
             item_key = self.compose_node(node, None)
-            #if item_key in node.value:
+            # if item_key in node.value:
             #    raise ComposerError("while composing a mapping", start_event.start_mark,
             #            "found duplicate key", key_event.start_mark)
             item_value = self.compose_node(node, item_key)
-            #node.value[item_key] = item_value
+            # node.value[item_key] = item_value
             node.value.append((item_key, item_value))
         end_event = self.get_event()
         node.end_mark = end_event.end_mark
         return node
-

tablib/packages/yaml3/constructor.py

@@ -683,4 +683,3 @@ Constructor.add_multi_constructor(
 Constructor.add_multi_constructor(
     'tag:yaml.org,2002:python/object/new:',
     Constructor.construct_python_object_new)
-

test_tablib.py

@@ -1,15 +1,17 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-
 """Tests for Tablib."""
 
+import json
 import unittest
 import sys
+import os
+
+import datetime
 
 import tablib
-from tablib.compat import markup
-
-
+from tablib.compat import markup, unicode, is_py3
+from tablib.core import Row
 
 
 class TablibTestCase(unittest.TestCase):
@@ -28,17 +30,15 @@ class TablibTestCase(unittest.TestCase):
         self.george = ('George', 'Washington', 67)
         self.tom = ('Thomas', 'Jefferson', 50)
 
-        self.founders = tablib.Dataset(headers=self.headers)
+        self.founders = tablib.Dataset(headers=self.headers, title='Founders')
         self.founders.append(self.john)
         self.founders.append(self.george)
         self.founders.append(self.tom)
 
-
     def tearDown(self):
         """Teardown."""
         pass
 
-
     def test_empty_append(self):
         """Verify append() correctly adds tuple with no headers."""
         new_row = (1, 2, 3)
@@ -48,7 +48,6 @@ class TablibTestCase(unittest.TestCase):
         self.assertTrue(data.width == len(new_row))
         self.assertTrue(data[0] == new_row)
 
-
     def test_empty_append_with_headers(self):
         """Verify append() correctly detects mismatch of number of
         headers and data.
@@ -58,6 +57,17 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertRaises(tablib.InvalidDimensions, data.append, new_row)
 
+    def test_set_headers_with_incorrect_dimension(self):
+        """Verify headers correctly detects mismatch of number of
+        headers and data.
+        """
+
+        data.append(self.john)
+
+        def set_header_callable():
+            data.headers = ['first_name']
+
+        self.assertRaises(tablib.InvalidDimensions, set_header_callable)
 
     def test_add_column(self):
         """Verify adding column works with/without headers."""
@@ -79,7 +89,6 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(data['age'], new_col)
 
-
     def test_add_column_no_data_no_headers(self):
         """Verify adding new column with no headers."""
 
@@ -91,27 +100,83 @@ class TablibTestCase(unittest.TestCase):
         self.assertEqual(data.width, 1)
         self.assertEqual(data.height, len(new_col))
 
+    def test_add_column_with_header_ignored(self):
+        """Verify append_col() ignores the header if data.headers has
+        not previously been set
+        """
+
+        new_col = ('reitz', 'monke')
+
+        data.append_col(new_col, header='first_name')
+
+        self.assertEqual(data[0], tuple([new_col[0]]))
+        self.assertEqual(data.width, 1)
+        self.assertEqual(data.height, len(new_col))
+        self.assertEqual(data.headers, None)
+
+    def test_add_column_with_header_and_headers_only_exist(self):
+        """Verify append_col() with header correctly detects mismatch when
+        headers exist but there is no existing row data
+        """
+
+        data.headers = ['first_name']
+        # no data
+
+        new_col = ('allen')
+
+        def append_col_callable():
+            data.append_col(new_col, header='middle_name')
+
+        self.assertRaises(tablib.InvalidDimensions, append_col_callable)
+
+    def test_add_column_with_header_and_data_exists(self):
+        """Verify append_col() works when headers and rows exists"""
+
+        data.headers = self.headers
+        data.append(self.john)
+
+        new_col = [10];
+
+        data.append_col(new_col, header='age')
+
+        self.assertEqual(data.height, 1)
+        self.assertEqual(data.width, len(self.john) + 1)
+        self.assertEqual(data['age'], new_col)
+        self.assertEqual(len(data.headers), len(self.headers) + 1)
 
     def test_add_callable_column(self):
         """Verify adding column with values specified as callable."""
 
-        new_col = [lambda x: x[0]]
+        new_col = lambda x: x[0]
 
         self.founders.append_col(new_col, header='first_again')
 
-
     def test_header_slicing(self):
         """Verify slicing by headers."""
 
         self.assertEqual(self.founders['first_name'],
-            [self.john[0], self.george[0], self.tom[0]])
+                         [self.john[0], self.george[0], self.tom[0]])
 
         self.assertEqual(self.founders['last_name'],
-            [self.john[1], self.george[1], self.tom[1]])
+                         [self.john[1], self.george[1], self.tom[1]])
 
         self.assertEqual(self.founders['gpa'],
-            [self.john[2], self.george[2], self.tom[2]])
+                         [self.john[2], self.george[2], self.tom[2]])
 
+    def test_get_col(self):
+        """Verify getting columns by index"""
+
+        self.assertEqual(
+            self.founders.get_col(list(self.headers).index('first_name')),
+            [self.john[0], self.george[0], self.tom[0]])
+
+        self.assertEqual(
+            self.founders.get_col(list(self.headers).index('last_name')),
+            [self.john[1], self.george[1], self.tom[1]])
+
+        self.assertEqual(
+            self.founders.get_col(list(self.headers).index('gpa')),
+            [self.john[2], self.george[2], self.tom[2]])
 
     def test_data_slicing(self):
         """Verify slicing by data."""
@@ -129,6 +194,16 @@ class TablibTestCase(unittest.TestCase):
         self.assertEqual(self.founders[1:3], [self.george, self.tom])
         self.assertEqual(self.founders[2:], [self.tom])
 
+    def test_row_slicing(self):
+        """Verify Row's __getslice__ method. Issue #184."""
+
+        john = Row(self.john)
+
+        self.assertEqual(john[:], list(self.john[:]))
+        self.assertEqual(john[0:], list(self.john[0:]))
+        self.assertEqual(john[:2], list(self.john[:2]))
+        self.assertEqual(john[0:2], list(self.john[0:2]))
+        self.assertEqual(john[0:-1], list(self.john[0:-1]))
 
     def test_delete(self):
         """Verify deleting from dataset works."""
@@ -152,7 +227,6 @@ class TablibTestCase(unittest.TestCase):
         # Delete from invalid index
         self.assertRaises(IndexError, self.founders.__delitem__, 3)
 
-
     def test_csv_export(self):
         """Verify exporting dataset object as CSV."""
 
@@ -170,11 +244,10 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(csv, self.founders.csv)
 
-
     def test_tsv_export(self):
-        """Verify exporting dataset object as CSV."""
+        """Verify exporting dataset object as TSV."""
 
-        # Build up the csv string with headers first, followed by each row
+        # Build up the tsv string with headers first, followed by each row
         tsv = ''
         for col in self.headers:
             tsv += col + '\t'
@@ -188,7 +261,6 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(tsv, self.founders.tsv)
 
-
     def test_html_export(self):
         """HTML export"""
 
@@ -200,7 +272,6 @@ class TablibTestCase(unittest.TestCase):
         html.thead.close()
 
         for founder in self.founders:
-
             html.tr(markup.oneliner.td(founder))
 
         html.table.close()
@@ -208,11 +279,99 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(html, self.founders.html)
 
+    def test_html_export_none_value(self):
+        """HTML export"""
+
+        html = markup.page()
+        html.table.open()
+        html.thead.open()
+
+        html.tr(markup.oneliner.th(['foo', '', 'bar']))
+        html.thead.close()
+
+        html.tr(markup.oneliner.td(['foo', '', 'bar']))
+
+        html.table.close()
+        html = str(html)
+
+        headers = ['foo', None, 'bar'];
+        d = tablib.Dataset(['foo', None, 'bar'], headers=headers)
+
+        self.assertEqual(html, d.html)
+
+    def test_latex_export(self):
+        """LaTeX export"""
+
+        expected = """\
+% Note: add \\usepackage{booktabs} to your preamble
+%
+\\begin{table}[!htbp]
+  \\centering
+  \\caption{Founders}
+  \\begin{tabular}{lrr}
+    \\toprule
+      first\\_name & last\\_name & gpa \\\\
+    \\cmidrule(r){1-1} \\cmidrule(lr){2-2} \\cmidrule(l){3-3}
+      John & Adams & 90 \\\\
+      George & Washington & 67 \\\\
+      Thomas & Jefferson & 50 \\\\
+    \\bottomrule
+  \\end{tabular}
+\\end{table}
+"""
+        output = self.founders.latex
+        self.assertEqual(output, expected)
+
+    def test_latex_export_empty_dataset(self):
+        self.assertTrue(tablib.Dataset().latex is not None)
+
+    def test_latex_export_no_headers(self):
+        d = tablib.Dataset()
+        d.append(('one', 'two', 'three'))
+        self.assertTrue('one' in d.latex)
+
+    def test_latex_export_caption(self):
+        d = tablib.Dataset()
+        d.append(('one', 'two', 'three'))
+        self.assertFalse('caption' in d.latex)
+
+        d.title = 'Title'
+        self.assertTrue('\\caption{Title}' in d.latex)
+
+    def test_latex_export_none_values(self):
+        headers = ['foo', None, 'bar']
+        d = tablib.Dataset(['foo', None, 'bar'], headers=headers)
+        output = d.latex
+        self.assertTrue('foo' in output)
+        self.assertFalse('None' in output)
+
+    def test_latex_escaping(self):
+        d = tablib.Dataset(['~', '^'])
+        output = d.latex
+
+        self.assertFalse('~' in output)
+        self.assertTrue('textasciitilde' in output)
+        self.assertFalse('^' in output)
+        self.assertTrue('textasciicircum' in output)
+
+    def test_str_no_columns(self):
+        d = tablib.Dataset(['a', 1], ['b', 2], ['c', 3])
+        output = '%s' % d
+
+        self.assertEqual(output.splitlines(), [
+            'a|1',
+            'b|2',
+            'c|3'
+        ])
 
     def test_unicode_append(self):
-        """Passes in a single unicode charecter and exports."""
+        """Passes in a single unicode character and exports."""
+
+        if is_py3:
+            new_row = ('å', 'é')
+        else:
+            exec ("new_row = (u'å', u'é')")
 
-        new_row = ('å', 'é')
         data.append(new_row)
 
         data.json
@@ -223,10 +382,30 @@ class TablibTestCase(unittest.TestCase):
         data.xlsx
         data.ods
         data.html
+        data.latex
 
+    def test_datetime_append(self):
+        """Passes in a single datetime and a single date and exports."""
+
+        new_row = (
+            datetime.datetime.now(),
+            datetime.datetime.today(),
+        )
+
+        data.append(new_row)
+
+        data.json
+        data.yaml
+        data.csv
+        data.tsv
+        data.xls
+        data.xlsx
+        data.ods
+        data.html
+        data.latex
 
     def test_book_export_no_exceptions(self):
-        """Test that varoius exports don't error out."""
+        """Test that various exports don't error out."""
 
         book = tablib.Databook()
         book.add_sheet(data)
@@ -236,7 +415,7 @@ class TablibTestCase(unittest.TestCase):
         book.xls
         book.xlsx
         book.ods
-
+        book.html
 
     def test_json_import_set(self):
         """Generate and import JSON set serialization."""
@@ -248,8 +427,7 @@ class TablibTestCase(unittest.TestCase):
 
         data.json = _json
 
-        self.assertEqual(_json, data.json)
-
+        self.assertEqual(json.loads(_json), json.loads(data.json))
 
     def test_json_import_book(self):
         """Generate and import JSON book serialization."""
@@ -262,8 +440,7 @@ class TablibTestCase(unittest.TestCase):
 
         book.json = _json
 
-        self.assertEqual(_json, book.json)
-
+        self.assertEqual(json.loads(_json), json.loads(book.json))
 
     def test_yaml_import_set(self):
         """Generate and import YAML set serialization."""
@@ -277,7 +454,6 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(_yaml, data.yaml)
 
-
     def test_yaml_import_book(self):
         """Generate and import YAML book serialization."""
         data.append(self.john)
@@ -291,7 +467,6 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(_yaml, book.yaml)
 
-
     def test_csv_import_set(self):
         """Generate and import CSV set serialization."""
         data.append(self.john)
@@ -304,6 +479,17 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(_csv, data.csv)
 
+    def test_csv_import_set_semicolons(self):
+        """Test for proper output with semicolon separated CSV."""
+        data.append(self.john)
+        data.append(self.george)
+        data.headers = self.headers
+
+        _csv = data.get_csv(delimiter=';')
+
+        data.set_csv(_csv, delimiter=';')
+
+        self.assertEqual(_csv, data.get_csv(delimiter=';'))
 
     def test_csv_import_set_with_spaces(self):
         """Generate and import CSV set serialization when row values have
@@ -318,6 +504,32 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(_csv, data.csv)
 
+    def test_csv_import_set_semicolon_with_spaces(self):
+        """Generate and import semicolon separated CSV set serialization when row values have
+        spaces."""
+        data.append(('Bill Gates', 'Microsoft'))
+        data.append(('Steve Jobs', 'Apple'))
+        data.headers = ('Name', 'Company')
+
+        _csv = data.get_csv(delimiter=';')
+
+        data.set_csv(_csv, delimiter=';')
+
+        self.assertEqual(_csv, data.get_csv(delimiter=';'))
+
+    def test_csv_import_set_with_newlines(self):
+        """Generate and import CSV set serialization when row values have
+        newlines."""
+        data.append(('Markdown\n=======',
+                     'A cool language\n\nwith paragraphs'))
+        data.append(('reStructedText\n==============',
+                     'Another cool language\n\nwith paragraphs'))
+        data.headers = ('title', 'body')
+
+        _csv = data.csv
+        data.csv = _csv
+
+        self.assertEqual(_csv, data.csv)
 
     def test_tsv_import_set(self):
         """Generate and import TSV set serialization."""
@@ -331,6 +543,107 @@ class TablibTestCase(unittest.TestCase):
 
         self.assertEqual(_tsv, data.tsv)
 
+    def test_dbf_import_set(self):
+        data.append(self.john)
+        data.append(self.george)
+        data.headers = self.headers
+
+        _dbf = data.dbf
+        data.dbf = _dbf
+
+        # self.assertEqual(_dbf, data.dbf)
+        try:
+            self.assertEqual(_dbf, data.dbf)
+        except AssertionError:
+            index = 0
+            so_far = ''
+            for reg_char, data_char in zip(_dbf, data.dbf):
+                so_far += chr(data_char)
+                if reg_char != data_char and index not in [1, 2, 3]:
+                    raise AssertionError('Failing at char %s: %s vs %s %s' % (
+                        index, reg_char, data_char, so_far))
+                index += 1
+
+    def test_dbf_export_set(self):
+        """Test DBF import."""
+        data.append(self.john)
+        data.append(self.george)
+        data.append(self.tom)
+        data.headers = self.headers
+
+        _regression_dbf = (b'\x03r\x06\x06\x03\x00\x00\x00\x81\x00\xab\x00\x00'
+                           b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+                           b'\x00\x00\x00FIRST_NAME\x00C\x00\x00\x00\x00P\x00\x00\x00\x00\x00'
+                           b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00LAST_NAME\x00\x00C\x00'
+                           b'\x00\x00\x00P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+                           b'\x00\x00GPA\x00\x00\x00\x00\x00\x00\x00\x00N\x00\x00\x00\x00\n'
+                           b'\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\r'
+                           )
+        _regression_dbf += b' John' + (b' ' * 75)
+        _regression_dbf += b' Adams' + (b' ' * 74)
+        _regression_dbf += b' 90.0000000'
+        _regression_dbf += b' George' + (b' ' * 73)
+        _regression_dbf += b' Washington' + (b' ' * 69)
+        _regression_dbf += b' 67.0000000'
+        _regression_dbf += b' Thomas' + (b' ' * 73)
+        _regression_dbf += b' Jefferson' + (b' ' * 70)
+        _regression_dbf += b' 50.0000000'
+        _regression_dbf += b'\x1a'
+
+        if is_py3:
+            # If in python3, decode regression string to binary.
+            # _regression_dbf = bytes(_regression_dbf, 'utf-8')
+            # _regression_dbf = _regression_dbf.replace(b'\n', b'\r')
+            pass
+
+        try:
+            self.assertEqual(_regression_dbf, data.dbf)
+        except AssertionError:
+            index = 0
+            found_so_far = ''
+            for reg_char, data_char in zip(_regression_dbf, data.dbf):
+                # found_so_far += chr(data_char)
+                if reg_char != data_char and index not in [1, 2, 3]:
+                    raise AssertionError(
+                        'Failing at char %s: %s vs %s (found %s)' % (
+                            index, reg_char, data_char, found_so_far))
+                index += 1
+
+    def test_dbf_format_detect(self):
+        """Test the DBF format detection."""
+        _dbf = (b'\x03r\x06\x03\x03\x00\x00\x00\x81\x00\xab\x00\x00'
+                b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+                b'\x00\x00\x00FIRST_NAME\x00C\x00\x00\x00\x00P\x00\x00\x00\x00\x00'
+                b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00LAST_NAME\x00\x00C\x00'
+                b'\x00\x00\x00P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+                b'\x00\x00GPA\x00\x00\x00\x00\x00\x00\x00\x00N\x00\x00\x00\x00\n'
+                b'\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\r'
+                )
+        _dbf += b' John' + (b' ' * 75)
+        _dbf += b' Adams' + (b' ' * 74)
+        _dbf += b' 90.0000000'
+        _dbf += b' George' + (b' ' * 73)
+        _dbf += b' Washington' + (b' ' * 69)
+        _dbf += b' 67.0000000'
+        _dbf += b' Thomas' + (b' ' * 73)
+        _dbf += b' Jefferson' + (b' ' * 70)
+        _dbf += b' 50.0000000'
+        _dbf += b'\x1a'
+
+        _yaml = '- {age: 90, first_name: John, last_name: Adams}'
+        _tsv = 'foo\tbar'
+        _csv = '1,2,3\n4,5,6\n7,8,9\n'
+        _json = '[{"last_name": "Adams","age": 90,"first_name": "John"}]'
+
+        _bunk = (
+            '¡¡¡¡¡¡¡¡£™∞¢£§∞§¶•¶ª∞¶•ªº••ª–º§•†•§º¶•†¥ª–º•§ƒø¥¨©πƒø†ˆ¥ç©¨√øˆ¥≈†ƒ¥ç©ø¨çˆ¥ƒçø¶'
+        )
+        self.assertTrue(tablib.formats.dbf.detect(_dbf))
+        self.assertFalse(tablib.formats.dbf.detect(_yaml))
+        self.assertFalse(tablib.formats.dbf.detect(_tsv))
+        self.assertFalse(tablib.formats.dbf.detect(_csv))
+        self.assertFalse(tablib.formats.dbf.detect(_json))
+        self.assertFalse(tablib.formats.dbf.detect(_bunk))
 
     def test_csv_format_detect(self):
         """Test CSV format detection."""
@@ -347,7 +660,6 @@ class TablibTestCase(unittest.TestCase):
         self.assertTrue(tablib.formats.csv.detect(_csv))
         self.assertFalse(tablib.formats.csv.detect(_bunk))
 
-
     def test_tsv_format_detect(self):
         """Test TSV format detection."""
 
@@ -363,7 +675,6 @@ class TablibTestCase(unittest.TestCase):
         self.assertTrue(tablib.formats.tsv.detect(_tsv))
         self.assertFalse(tablib.formats.tsv.detect(_bunk))
 
-
     def test_json_format_detect(self):
         """Test JSON format detection."""
 
@@ -375,18 +686,18 @@ class TablibTestCase(unittest.TestCase):
         self.assertTrue(tablib.formats.json.detect(_json))
         self.assertFalse(tablib.formats.json.detect(_bunk))
 
-
     def test_yaml_format_detect(self):
         """Test YAML format detection."""
 
         _yaml = '- {age: 90, first_name: John, last_name: Adams}'
+        _tsv = 'foo\tbar'
         _bunk = (
             '¡¡¡¡¡¡---///\n\n\n¡¡£™∞¢£§∞§¶•¶ª∞¶•ªº••ª–º§•†•§º¶•†¥ª–º•§ƒø¥¨©πƒø†ˆ¥ç©¨√øˆ¥≈†ƒ¥ç©ø¨çˆ¥ƒçø¶'
         )
 
         self.assertTrue(tablib.formats.yaml.detect(_yaml))
         self.assertFalse(tablib.formats.yaml.detect(_bunk))
-
+        self.assertFalse(tablib.formats.yaml.detect(_tsv))
 
     def test_auto_format_detect(self):
         """Test auto format detection."""
@@ -394,13 +705,14 @@ class TablibTestCase(unittest.TestCase):
         _yaml = '- {age: 90, first_name: John, last_name: Adams}'
         _json = '[{"last_name": "Adams","age": 90,"first_name": "John"}]'
         _csv = '1,2,3\n4,5,6\n7,8,9\n'
+        _tsv = '1\t2\t3\n4\t5\t6\n7\t8\t9\n'
         _bunk = '¡¡¡¡¡¡---///\n\n\n¡¡£™∞¢£§∞§¶•¶ª∞¶•ªº••ª–º§•†•§º¶•†¥ª–º•§ƒø¥¨©πƒø†ˆ¥ç©¨√øˆ¥≈†ƒ¥ç©ø¨çˆ¥ƒçø¶'
 
-        self.assertEqual(tablib.detect(_yaml)[0], tablib.formats.yaml)
-        self.assertEqual(tablib.detect(_csv)[0], tablib.formats.csv)
-        self.assertEqual(tablib.detect(_json)[0], tablib.formats.json)
-        self.assertEqual(tablib.detect(_bunk)[0], None)
-
+        self.assertEqual(tablib.detect_format(_yaml), 'yaml')
+        self.assertEqual(tablib.detect_format(_csv), 'csv')
+        self.assertEqual(tablib.detect_format(_tsv), 'tsv')
+        self.assertEqual(tablib.detect_format(_json), 'json')
+        self.assertEqual(tablib.detect_format(_bunk), None)
 
     def test_transpose(self):
         """Transpose a dataset."""
@@ -410,12 +722,20 @@ class TablibTestCase(unittest.TestCase):
         second_row = transposed_founders[1]
 
         self.assertEqual(transposed_founders.headers,
-                  ["first_name","John", "George", "Thomas"])
+                         ["first_name", "John", "George", "Thomas"])
         self.assertEqual(first_row,
-                   ("last_name","Adams", "Washington", "Jefferson"))
+                         ("last_name", "Adams", "Washington", "Jefferson"))
         self.assertEqual(second_row,
-                   ("gpa",90, 67, 50))
+                         ("gpa", 90, 67, 50))
 
+    def test_transpose_multiple_headers(self):
+
+        data = tablib.Dataset()
+        data.headers = ("first_name", "last_name", "age")
+        data.append(('John', 'Adams', 90))
+        data.append(('George', 'Washington', 67))
+        data.append(('John', 'Tyler', 71))
+        self.assertEqual(data.transpose().transpose().dict, data.dict)
 
     def test_row_stacking(self):
         """Row stacking."""
@@ -428,12 +748,10 @@ class TablibTestCase(unittest.TestCase):
         row_stacked = self.founders.stack(to_join)
 
         for column in row_stacked.headers:
-
             original_data = self.founders[column]
             expected_data = original_data + original_data
             self.assertEqual(row_stacked[column], expected_data)
 
-
     def test_column_stacking(self):
         """Column stacking"""
 
@@ -445,19 +763,18 @@ class TablibTestCase(unittest.TestCase):
         column_stacked = self.founders.stack_cols(to_join)
 
         for index, row in enumerate(column_stacked):
-
             original_data = self.founders[index]
             expected_data = original_data + original_data
             self.assertEqual(row, expected_data)
 
         self.assertEqual(column_stacked[0],
-                   ("John", "Adams", 90, "John", "Adams", 90))
-
+                         ("John", "Adams", 90, "John", "Adams", 90))
 
     def test_sorting(self):
         """Sort columns."""
 
         sorted_data = self.founders.sort(col="first_name")
+        self.assertEqual(sorted_data.title, 'Founders')
 
         first_row = sorted_data[0]
         second_row = sorted_data[2]
@@ -470,6 +787,23 @@ class TablibTestCase(unittest.TestCase):
         self.assertEqual(second_row, expected_second)
         self.assertEqual(third_row, expected_third)
 
+    def test_remove_duplicates(self):
+        """Unique Rows."""
+
+        self.founders.append(self.john)
+        self.founders.append(self.george)
+        self.founders.append(self.tom)
+        self.assertEqual(self.founders[0], self.founders[3])
+        self.assertEqual(self.founders[1], self.founders[4])
+        self.assertEqual(self.founders[2], self.founders[5])
+        self.assertEqual(self.founders.height, 6)
+
+        self.founders.remove_duplicates()
+
+        self.assertEqual(self.founders[0], self.john)
+        self.assertEqual(self.founders[1], self.george)
+        self.assertEqual(self.founders[2], self.tom)
+        self.assertEqual(self.founders.height, 3)
 
     def test_wipe(self):
         """Purge a dataset."""
@@ -487,6 +821,24 @@ class TablibTestCase(unittest.TestCase):
         self.assertTrue(data.width == len(new_row))
         self.assertTrue(data[0] == new_row)
 
+    def test_subset(self):
+        """Create a subset of a dataset"""
+
+        rows = (0, 2)
+        columns = ('first_name', 'gpa')
+
+        data.headers = self.headers
+
+        data.append(self.john)
+        data.append(self.george)
+        data.append(self.tom)
+
+        # Verify data is truncated
+        subset = data.subset(rows=rows, cols=columns)
+        self.assertEqual(type(subset), tablib.Dataset)
+        self.assertEqual(subset.headers, list(columns))
+        self.assertEqual(subset._data[0].list, ['John', 90])
+        self.assertEqual(subset._data[1].list, ['Thomas', 50])
 
     def test_formatters(self):
         """Confirm formatters are being triggered."""
@@ -507,10 +859,103 @@ class TablibTestCase(unittest.TestCase):
         if sys.version_info[0] > 2:
             data.append(['\xfc', '\xfd'])
         else:
-            exec("data.append([u'\xfc', u'\xfd'])")
-
+            exec ("data.append([u'\xfc', u'\xfd'])")
 
         data.csv
 
+    def test_csv_column_select(self):
+        """Build up a CSV and test selecting a column"""
+
+        data = tablib.Dataset()
+        data.csv = self.founders.csv
+
+        headers = data.headers
+        self.assertTrue(isinstance(headers[0], unicode))
+
+        orig_first_name = self.founders[self.headers[0]]
+        csv_first_name = data[headers[0]]
+        self.assertEqual(orig_first_name, csv_first_name)
+
+    def test_csv_column_delete(self):
+        """Build up a CSV and test deleting a column"""
+
+        data = tablib.Dataset()
+        data.csv = self.founders.csv
+
+        target_header = data.headers[0]
+        self.assertTrue(isinstance(target_header, unicode))
+
+        del data[target_header]
+
+        self.assertTrue(target_header not in data.headers)
+
+    def test_csv_column_sort(self):
+        """Build up a CSV and test sorting a column by name"""
+
+        data = tablib.Dataset()
+        data.csv = self.founders.csv
+
+        orig_target_header = self.founders.headers[1]
+        target_header = data.headers[1]
+
+        self.founders.sort(orig_target_header)
+        data.sort(target_header)
+
+        self.assertEqual(self.founders[orig_target_header], data[target_header])
+
+    def test_unicode_renders_markdown_table(self):
+        # add another entry to test right field width for
+        # integer
+        self.founders.append(('Old', 'Man', 100500))
+        self.assertEqual('first_name|last_name |gpa   ', unicode(self.founders).split('\n')[0])
+
+    def test_databook_add_sheet_accepts_only_dataset_instances(self):
+        class NotDataset(object):
+            def append(self, item):
+                pass
+
+        dataset = NotDataset()
+        dataset.append(self.john)
+
+        self.assertRaises(tablib.InvalidDatasetType, book.add_sheet, dataset)
+
+    def test_databook_add_sheet_accepts_dataset_subclasses(self):
+        class DatasetSubclass(tablib.Dataset):
+            pass
+
+        # just checking if subclass of tablib.Dataset can be added to Databook
+        dataset = DatasetSubclass()
+        dataset.append(self.john)
+        dataset.append(self.tom)
+
+        try:
+            book.add_sheet(dataset)
+        except tablib.InvalidDatasetType:
+            self.fail("Subclass of tablib.Dataset should be accepted by Databook.add_sheet")
+
+    def test_csv_formatter_support_kwargs(self):
+        """Test CSV import and export with formatter configuration."""
+        data.append(self.john)
+        data.append(self.george)
+        data.headers = self.headers
+
+        expected = 'first_name;last_name;gpa\nJohn;Adams;90\nGeorge;Washington;67\n'
+
+        kwargs = dict(delimiter=';', lineterminator='\n')
+        _csv = data.export('csv', **kwargs)
+        self.assertEqual(expected, _csv)
+
+        # the import works but consider default delimiter=','
+        d1 = tablib.import_set(_csv, format="csv")
+        self.assertEqual(1, len(d1.headers))
+
+        d2 = tablib.import_set(_csv, format="csv", **kwargs)
+        self.assertEqual(3, len(d2.headers))
+
+    def test_databook_formatter_support_kwargs(self):
+        """Test XLSX export with formatter configuration."""
+        self.founders.export('xlsx', freeze_panes=False)
+
+
 if __name__ == '__main__':
     unittest.main()

tox.ini

@@ -1,11 +1,11 @@
+# Tox (http://tox.testrun.org/) is a tool for running tests
+# in multiple virtualenvs. This configuration file will run the
+# test suite on all supported python versions. To use it, "pip install tox"
+# and then run "tox" from this directory.
+
 [tox]
-envlist = py25,py26,py27,py3
+envlist = py26, py27, py32, py33, py34, py35, py36, pypy
 
 [testenv]
-commands=py.test --junitxml=junit-{envname}.xml
+commands = python setup.py test
 deps = pytest
-
-
-[testenv:py3]
-basepython=/usr/bin/python3
-simplejson = pytest