Compare commits

..

1 Commits

Author SHA1 Message Date
Kenneth Reitz 551fc90461 Merge branch 'release/0.6.4' 2010-09-20 09:21:22 -04:00
69 changed files with 756 additions and 8996 deletions
-10
View File
@@ -1,10 +0,0 @@
# .coveragerc to control coverage.py
[report]
# Regexes for lines to exclude from consideration
exclude_lines =
# Have to re-enable the standard pragma:
pragma: no cover
# Don't complain if non-runnable code isn't run:
if __name__ == .__main__.:
-10
View File
@@ -1,10 +0,0 @@
[![Jazzband](https://jazzband.co/static/img/jazzband.svg)](https://jazzband.co/)
This is a [Jazzband](https://jazzband.co/) project. By contributing you agree to abide
by the [Contributor Code of Conduct](https://jazzband.co/about/conduct) and follow the
[guidelines](https://jazzband.co/about/guidelines).
If you'd like to contribute, simply fork
[the repository](https://github.com/jazzband/tablib), commit your changes to a feature
branch, and send a pull request to `master`. Make sure you add yourself to
[AUTHORS](https://github.com/jazzband/tablib/blob/master/AUTHORS).
-21
View File
@@ -17,24 +17,3 @@ profile
# vi noise
*.swp
docs/_build/*
coverage.xml
nosetests.xml
junit-py25.xml
junit-py26.xml
junit-py27.xml
# tox noise
.tox
# pyenv noise
.python-version
tablib.egg-info/*
# Coverage
.coverage
htmlcov
# setuptools noise
.eggs
*.egg-info
-6
View File
@@ -1,6 +0,0 @@
[settings]
multi_line_output=3
include_trailing_comma=True
force_grid_wrap=0
use_parentheses=True
line_length=88
-22
View File
@@ -1,22 +0,0 @@
language: python
python:
- 2.7
- 3.6
- 3.7
- 3.8
cache: pip
dist: xenial
install: travis_retry pip install tox-travis
script: tox
after_success: bash <(curl -s https://codecov.io/bash)
deploy:
provider: pypi
user: jazzband
server: https://jazzband.co/projects/tablib/upload
distributions: sdist bdist_wheel
password:
secure: svV4fYtodwW+iTyFOm5ISEfhVwcA+6vTskD3x6peznc40TdMV9Ek8nT3Q/NB4lCbXoUw2qR4H6uhLCjesnv/VvVk/qbitCyD8ySlgwOV5n7NzJs8lC8EYaHSjGQjatTwJAokfGVYkPawkI7HXDqtDggLUQBK+Ag8HDW+XBSbQIU=
on:
tags: true
repo: jazzband/tablib
python: 3.7
+11 -28
View File
@@ -1,30 +1,13 @@
Tablib was originally written by Kenneth Reitz and is now maintained
by the Jazzband GitHub team.
Tablib is written and maintained by Kenneth Reitz and
various contributors:
Here is a list of passed and present much-appreciated contributors:
Development Lead
````````````````
Alex Gaynor
Andrii Soldatenko
Benjamin Wohlwend
Bruno Soares
Claude Paroz
Erik Youngren
Hugo van Kemenade
Iuri de Silvio
Jakub Janoszek
James Douglass
Joel Friedly
Josh Ourisman
Kenneth Reitz
Luca Beltrame
Luke Lee
Marc Abramowitz
Marco Dallagiacoma
Mark Rogers
Mark Walling
Mathias Loesch
Mike Waldner
Rabin Nankhwa
Tommy Anthony
Tsuyoshi Hombashi
Tushar Makkar
- Kenneth Reitz <me@kennethreitz.com>
Patches and Suggestions
```````````````````````
- A Lucky Someone
-273
View File
@@ -1,273 +0,0 @@
# History
## 0.14.0 (2019-10-19)
### Deprecations
- The 0.14.x series will be the last to support Python 2
### Breaking changes
- Dropped Python 3.4 support
### Improvements
- Added Python 3.7 and 3.8 support
- The project is now maintained by the Jazzband team, https://jazzband.co
- Improved format autodetection and added autodetection for the odf format.
- Added search to all documentation pages
- Open xlsx workbooks in read-only mode (#316)
- Unpin requirements
- Only install backports.csv on Python 2
### Bugfixes
- Fixed `DataBook().load` parameter ordering (first stream, then format).
- Fixed a regression for xlsx exports where non-string values were forced to
strings (#314)
- Fixed xlsx format detection (which was often detected as `xls` format)
## 0.13.0 (2019-03-08)
- Added reStructuredText output capability (#336)
- Added Jira output capability
- Stopped calling openpyxl deprecated methods (accessing cells, removing sheets)
(openpyxl minimal version is now 2.4.0)
- Fixed a circular dependency issue in JSON output (#332)
- Fixed Unicode error for the CSV export on Python 2 (#215)
- Removed usage of optional `ujson` (#311)
- Dropped Python 3.3 support
## 0.12.1 (2017-09-01)
- Favor `Dataset.export(<format>)` over `Dataset.<format>` syntax in docs
- Make Panda dependency optional
## 0.12.0 (2017-08-27)
- Add initial Panda DataFrame support
- Dropped Python 2.6 support
## 0.11.5 (2017-06-13)
- Use `yaml.safe_load` for importing yaml.
## 0.11.4 (2017-01-23)
- Use built-in `json` package if available
- Support Python 3.5+ in classifiers
### Bugfixes
- Fixed textual representation for Dataset with no headers
- Handle decimal types
## 0.11.3 (2016-02-16)
- Release fix.
## 0.11.2 (2016-02-16)
### Bugfixes
- Fix export only formats.
- Fix for xlsx output.
## 0.11.1 (2016-02-07)
### Bugfixes
- Fixed packaging error on Python 3.
## 0.11.0 (2016-02-07)
### New Formats!
- Added LaTeX table export format (`Dataset.latex`).
- Support for dBase (DBF) files (`Dataset.dbf`).
### Improvements
- New import/export interface (`Dataset.export()`, `Dataset.load()`).
- CSV custom delimiter support (`Dataset.export('csv', delimiter='$')`).
- Adding ability to remove duplicates to all rows in a dataset (`Dataset.remove_duplicates()`).
- Added a mechanism to avoid `datetime.datetime` issues when serializing data.
- New `detect_format()` function (mostly for internal use).
- Update the vendored unicodecsv to fix `None` handling.
- Only freeze the headers row, not the headers columns (xls).
### Breaking Changes
- `detect()` function removed.
### Bugfixes
- Fix XLSX import.
- Bugfix for `Dataset.transpose().transpose()`.
## 0.10.0 (2014-05-27)
* Unicode Column Headers
* ALL the bugfixes!
## 0.9.11 (2011-06-30)
* Bugfixes
## 0.9.10 (2011-06-22)
* Bugfixes
## 0.9.9 (2011-06-21)
* Dataset API Changes
* `stack_rows` => `stack`, `stack_columns` => `stack_cols`
* column operations have their own methods now (`append_col`, `insert_col`)
* List-style `pop()`
* Redis-style `rpush`, `lpush`, `rpop`, `lpop`, `rpush_col`, and `lpush_col`
## 0.9.8 (2011-05-22)
* OpenDocument Spreadsheet support (.ods)
* Full Unicode TSV support
## 0.9.7 (2011-05-12)
* Full XLSX Support!
* Pickling Bugfix
* Compat Module
## 0.9.6 (2011-05-12)
* `seperators` renamed to `separators`
* Full unicode CSV support
## 0.9.5 (2011-03-24)
* Python 3.1, Python 3.2 Support (same code base!)
* Formatter callback support
* Various bug fixes
## 0.9.4 (2011-02-18)
* Python 2.5 Support!
* Tox Testing for 2.5, 2.6, 2.7
* AnyJSON Integrated
* OrderedDict support
* Caved to community pressure (spaces)
## 0.9.3 (2011-01-31)
* Databook duplication leak fix.
* HTML Table output.
* Added column sorting.
## 0.9.2 (2010-11-17)
* Transpose method added to Datasets.
* New frozen top row in Excel output.
* Pickling support for Datasets and Rows.
* Support for row/column stacking.
## 0.9.1 (2010-11-04)
* Minor reference shadowing bugfix.
## 0.9.0 (2010-11-04)
* Massive documentation update!
* Tablib.org!
* Row tagging and Dataset filtering!
* Column insert/delete support
* Column append API change (header required)
* Internal Changes (Row object and use thereof)
## 0.8.5 (2010-10-06)
* New import system. All dependencies attempt to load from site-packages,
then fallback on tenderized modules.
## 0.8.4 (2010-10-04)
* Updated XLS output: Only wrap if '\\n' in cell.
## 0.8.3 (2010-10-04)
* Ability to append new column passing a callable
as the value that will be applied to every row.
## 0.8.2 (2010-10-04)
* Added alignment wrapping to written cells.
* Added separator support to XLS.
## 0.8.1 (2010-09-28)
* Packaging Fix
## 0.8.0 (2010-09-25)
* New format plugin system!
* Imports! ELEGANT Imports!
* Tests. Lots of tests.
## 0.7.1 (2010-09-20)
* Reverting methods back to properties.
* Windows bug compensated in documentation.
## 0.7.0 (2010-09-20)
* Renamed DataBook Databook for consistency.
* Export properties changed to methods (XLS filename / StringIO bug).
* Optional Dataset.xls(path='filename') support (for writing on windows).
* Added utf-8 on the worksheet level.
## 0.6.4 (2010-09-19)
* Updated unicode export for XLS.
* More exhaustive unit tests.
## 0.6.3 (2010-09-14)
* Added Dataset.append() support for columns.
## 0.6.2 (2010-09-13)
* Fixed Dataset.append() error on empty dataset.
* Updated Dataset.headers property w/ validation.
* Added Testing Fixtures.
## 0.6.1 (2010-09-12)
* Packaging hotfixes.
## 0.6.0 (2010-09-11)
* Public Release.
* Export Support for XLS, JSON, YAML, and CSV.
* DataBook Export for XLS, JSON, and YAML.
* Python Dict Property Support.
+35
View File
@@ -0,0 +1,35 @@
History
=======
0.6.4 (2010-09-13)
------------------
* Updated unicode export for XLS
* More exhaustive unit tests
0.6.3 (2010-09-14)
------------------
* Added Dataset.append() support for columns.
0.6.2 (2010-09-13)
------------------
* Fixed Dataset.append() error on empty dataset.
* Updated Dataset.headers property w/ validation.
* Added Testing Fixtures.
0.6.1 (2010-09-12)
------------------
* Packaging hotfixes.
0.6.0 (2010-09-11)
------------------
* Public Release.
* Export Support for XLS, JSON, YAML, and CSV.
* DataBook Export for XLS, JSON, and YAML.
* Python Dict Property Support.
+2 -3
View File
@@ -1,5 +1,4 @@
Copyright 2016 Kenneth Reitz
Copyright 2019 Jazzband
Copyright (c) 2010 Kenneth Reitz.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
@@ -17,4 +16,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
THE SOFTWARE.
+1 -6
View File
@@ -1,6 +1 @@
recursive-include docs *
recursive-include tests *
include pytest.ini tox.ini .isort.cfg .coveragerc HISTORY.md README.md LICENSE AUTHORS
prune docs/_build
prune *.pyc
prune __pycache__
include HISTORY.rst README.rst LICENSE AUTHORS
-177
View File
@@ -1,177 +0,0 @@
# Tablib: format-agnostic tabular dataset library
[![Jazzband](https://jazzband.co/static/img/badge.svg)](https://jazzband.co/)
[![Build Status](https://travis-ci.org/jazzband/tablib.svg?branch=master)](https://travis-ci.org/jazzband/tablib)
[![codecov](https://codecov.io/gh/jazzband/tablib/branch/master/graph/badge.svg)](https://codecov.io/gh/jazzband/tablib)
_____ ______ ___________ ______
__ /_______ ____ /_ ___ /___(_)___ /_
_ __/_ __ `/__ __ \__ / __ / __ __ \
/ /_ / /_/ / _ /_/ /_ / _ / _ /_/ /
\__/ \__,_/ /_.___/ /_/ /_/ /_.___/
Tablib is a format-agnostic tabular dataset library, written in Python.
Output formats supported:
- Excel (Sets + Books)
- JSON (Sets + Books)
- YAML (Sets + Books)
- Pandas DataFrames (Sets)
- HTML (Sets)
- Jira (Sets)
- TSV (Sets)
- ODS (Sets)
- CSV (Sets)
- DBF (Sets)
Note that tablib *purposefully* excludes XML support. It always will. (Note: This is a
joke. Pull requests are welcome.)
## Overview
`tablib.Dataset()`
A Dataset is a table of tabular data.
It may or may not have a header row.
They can be build and manipulated as raw Python datatypes (Lists of tuples|dictionaries).
Datasets can be imported from JSON, YAML, DBF, and CSV;
they can be exported to XLSX, XLS, ODS, JSON, YAML, DBF, CSV, TSV, and HTML.
`tablib.Databook()`
A Databook is a set of Datasets.
The most common form of a Databook is an Excel file with multiple spreadsheets.
Databooks can be imported from JSON and YAML;
they can be exported to XLSX, XLS, ODS, JSON, and YAML.
## Usage
Populate fresh data files:
```python
headers = ('first_name', 'last_name')
data = [
('John', 'Adams'),
('George', 'Washington')
]
data = tablib.Dataset(*data, headers=headers)
```
Intelligently add new rows:
```python
>>> data.append(('Henry', 'Ford'))
```
Intelligently add new columns:
```python
>>> data.append_col((90, 67, 83), header='age')
```
Slice rows:
```python
>>> print(data[:2])
[('John', 'Adams', 90), ('George', 'Washington', 67)]
```
Slice columns by header:
```python
>>> print(data['first_name'])
['John', 'George', 'Henry']
```
Easily delete rows:
```python
>>> del data[1]
```
## Exports
Drumroll please...........
### JSON!
```python
>>> print(data.export('json'))
[
{
"last_name": "Adams",
"age": 90,
"first_name": "John"
},
{
"last_name": "Ford",
"age": 83,
"first_name": "Henry"
}
]
```
### YAML!
```python
>>> print(data.export('yaml'))
- {age: 90, first_name: John, last_name: Adams}
- {age: 83, first_name: Henry, last_name: Ford}
```
### CSV...
```python
>>> print(data.export('csv'))
first_name,last_name,age
John,Adams,90
Henry,Ford,83
```
### EXCEL!
```python
>>> with open('people.xls', 'wb') as f:
... f.write(data.export('xls'))
```
### DBF!
```python
>>> with open('people.dbf', 'wb') as f:
... f.write(data.export('dbf'))
```
### Pandas DataFrame!
```python
>>> print(data.export('df')):
first_name last_name age
0 John Adams 90
1 Henry Ford 83
```
It's that easy.
## Installation
To install tablib, simply:
```console
$ pip install tablib[pandas]
```
Make sure to check out [Tablib on PyPI](https://pypi.org/project/tablib/)!
## Contribute
Please see the [contributing guide](https://github.com/jazzband/tablib/blob/master/.github/CONTRIBUTING.md).
+144
View File
@@ -0,0 +1,144 @@
Tablib: format-agnostic tabular dataset library
===============================================
::
_____ ______ ___________ ______
__ /_______ ____ /_ ___ /___(_)___ /_
_ __/_ __ `/__ __ \__ / __ / __ __ \
/ /_ / /_/ / _ /_/ /_ / _ / _ /_/ /
\__/ \__,_/ /_.___/ /_/ /_/ /_.___/
Tablib is a format-agnostic tabular dataset library, written in Python.
Output formats supported:
- Excel
- JSON
- YAML
- CSV
At this time, Tablib supports the **export** of it's powerful Dataset object instances into any of the above formats. Import is underway.
Note that tablib *purposefully* excludes XML support. It always will.
Usage
-----
Populate fresh data files: ::
headers = ('first_name', 'last_name')
data = [
('John', 'Adams'),
('George', 'Washington')
]
data = tablib.Dataset(*data, headers=headers)
Intelligently add new rows: ::
>>> data.append(('Henry', 'Ford'))
Intelligently add new columns: ::
>>> data.append(col=('age', 90, 67, 83))
Slice rows: ::
>>> print data[:2]
[('John', 'Adams', 90), ('George', 'Washington', 67)]
Slice columns by header: ::
>>> print data['first_name']
['John', 'George', 'Henry']
Easily delete rows: ::
>>> del data[1]
Drumroll please...........
JSON!
+++++
::
>>> print data.json
[
{
"last_name": "Adams",
"age": 90,
"first_name": "John"
},
{
"last_name": "Ford",
"age": 83,
"first_name": "Henry"
}
]
YAML!
+++++
::
>>> print data.yaml
- {age: 90, first_name: John, last_name: Adams}
- {age: 83, first_name: Henry, last_name: Ford}
CSV...
++++++
::
>>> print data.csv
first_name,last_name,age
John,Adams,90
Henry,Ford,83
EXCEL!
++++++
::
>>> open('people.xls').write(data.xls)
It's that easy.
Installation
------------
To install tablib, simply: ::
$ pip install tablib
Or, if you absolutely must: ::
$ easy_install tablib
Contribute
----------
If you'd like to contribute, simply fork `the repository`_, commit your changes to the **develop** branch (or branch off of it), and send a pull request. Make sure you add yourself to AUTHORS_.
Roadmap
-------
- Add ability to add/remove full columns
- Import datasets from CSV, JSON, YAML
- Release CLI Interface
- Auto-detect import format
- Add possible other exports (SQL?)
- Possibly plugin-ify format architecture
- Ability to assign types to rows (set, regex=, &c.)
- Plugin support
.. _`the repository`: http://github.com/kennethreitz/tablib
.. _AUTHORS: http://github.com/kennethreitz/tablib/blob/master/AUTHORS
-130
View File
@@ -1,130 +0,0 @@
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
clean:
-rm -rf $(BUILDDIR)/*
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Tablib.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Tablib.qhc"
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/Tablib"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Tablib"
@echo "# devhelp"
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
make -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
-18
View File
@@ -1,18 +0,0 @@
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
"backports.csv" = "*"
odfpy = "*"
openpyxl = ">=2.4.0"
pandas = "*"
xlrd = "*"
xlwt = "*"
PyYAML = "*"
[requires]
python_version = "3.6"
View File
-11
View File
@@ -1,11 +0,0 @@
<h3><a href="https://tablib.readthedocs.io">About Tablib</a></h3>
<p>
Tablib is an MIT Licensed format-agnostic tabular dataset library, written in Python. It allows you to import, export, and manipulate tabular data sets. Advanced features include, segregation, dynamic columns, tags & filtering, and seamless format import & export.
</p>
<h3>Useful Links</h3>
<ul>
<li><a href="https://tablib.readthedocs.io">The Tablib Website</a></li>
<li><a href="https://pypi.org/project/tablib">Tablib @ PyPI</a></li>
<li><a href="https://github.com/jazzband/tablib">Tablib @ GitHub</a></li>
<li><a href="https://github.com/jazzband/tablib/issues">Issue Tracker</a></li>
</ul>
-4
View File
@@ -1,4 +0,0 @@
<h3><a href="https://tablib.readthedocs.io">About Tablib</a></h3>
<p>
Tablib is an MIT Licensed format-agnostic tabular dataset library, written in Python. It allows you to import, export, and manipulate tabular data sets. Advanced features include, segregation, dynamic columns, tags & filtering, and seamless format import & export.
</p>
-64
View File
@@ -1,64 +0,0 @@
.. _api:
===
API
===
.. module:: tablib
This part of the documentation covers all the interfaces of Tablib. For
parts where Tablib depends on external libraries, we document the most
important right here and provide links to the canonical documentation.
--------------
Dataset Object
--------------
.. autoclass:: Dataset
:inherited-members:
---------------
Databook Object
---------------
.. autoclass:: Databook
:inherited-members:
---------
Functions
---------
.. autofunction:: detect_format
.. autofunction:: import_set
----------
Exceptions
----------
.. class:: InvalidDatasetType
You're trying to add something that doesn't quite look right.
.. class:: InvalidDimensions
You're trying to add something that doesn't quite fit right.
.. class:: UnsupportedFormat
You're trying to add something that doesn't quite taste right.
Now, go start some :ref:`Tablib Development <development>`.
-227
View File
@@ -1,227 +0,0 @@
# -*- coding: utf-8 -*-
#
# Tablib documentation build configuration file, created by
# sphinx-quickstart on Tue Oct 5 15:25:21 2010.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
from pkg_resources import get_distribution
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# sys.path.insert(0, os.path.abspath('..'))
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.viewcode']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'Tablib'
copyright = u'2019 Jazzband'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The full version, including alpha/beta/rc tags.
release = get_distribution('tablib').version
# The short X.Y version.
version = '.'.join(release.split('.')[:2])
# for example take major/minor
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
# pygments_style = ''
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'alabaster'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
html_sidebars = {
'index': ['sidebarintro.html', 'sourcelink.html', 'searchbox.html'],
'**': ['sidebarlogo.html', 'localtoc.html', 'relations.html',
'sourcelink.html', 'searchbox.html']
}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
html_show_sphinx = False
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'Tablibdoc'
# -- Options for LaTeX output --------------------------------------------------
# The paper size ('letter' or 'a4').
#latex_paper_size = 'letter'
# The font size ('10pt', '11pt' or '12pt').
#latex_font_size = '10pt'
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('index', 'Tablib.tex', u'Tablib Documentation',
u'Jazzband', 'manual'),
]
latex_use_modindex = False
latex_elements = {
'papersize': 'a4paper',
'pointsize': '12pt',
}
latex_use_parts = True
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Additional stuff for the LaTeX preamble.
#latex_preamble = ''
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for manual page output --------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'tablib', u'Tablib Documentation',
[u'Jazzband'], 1)
]
-207
View File
@@ -1,207 +0,0 @@
.. _development:
Development
===========
Tablib is under active development, and contributors are welcome.
If you have a feature request, suggestion, or bug report, please open a new
issue on GitHub_. To submit patches, please send a pull request on GitHub_.
.. _GitHub: https://github.com/jazzband/tablib/
.. _design:
---------------------
Design Considerations
---------------------
Tablib was developed with a few :pep:`20` idioms in mind.
#. Beautiful is better than ugly.
#. Explicit is better than implicit.
#. Simple is better than complex.
#. Complex is better than complicated.
#. Readability counts.
A few other things to keep in mind:
#. Keep your code DRY.
#. Strive to be as simple (to use) as possible.
.. _scm:
--------------
Source Control
--------------
Tablib source is controlled with Git_, the lean, mean, distributed source
control machine.
The repository is publicly accessible.
.. code-block:: console
git clone git://github.com/jazzband/tablib.git
The project is hosted on **GitHub**.
GitHub:
https://github.com/jazzband/tablib
Git Branch Structure
++++++++++++++++++++
Feature / Hotfix / Release branches follow a `Successful Git Branching Model`_ .
Git-flow_ is a great tool for managing the repository. I highly recommend it.
``master``
Current production release (|version|) on PyPi.
Each release is tagged.
When submitting patches, please place your feature/change in its own branch prior to opening a pull request on GitHub_.
.. _Git: https://git-scm.org
.. _`Successful Git Branching Model`: https://nvie.com/posts/a-successful-git-branching-model/
.. _git-flow: https://github.com/nvie/gitflow
.. _newformats:
------------------
Adding New Formats
------------------
Tablib welcomes new format additions! Format suggestions include:
* MySQL Dump
Coding by Convention
++++++++++++++++++++
Tablib features a micro-framework for adding format support.
The easiest way to understand it is to use it.
So, let's define our own format, named *xxx*.
1. Write a new format interface.
:class:`tablib.core` follows a simple pattern for automatically utilizing your format throughout Tablib.
Function names are crucial.
Example **tablib/formats/_xxx.py**: ::
title = 'xxx'
def export_set(dset):
....
# returns string representation of given dataset
def export_book(dbook):
....
# returns string representation of given databook
def import_set(dset, in_stream):
...
# populates given Dataset with given datastream
def import_book(dbook, in_stream):
...
# returns Databook instance
def detect(stream):
...
# returns True if given stream is parsable as xxx
.. admonition:: Excluding Support
If the format excludes support for an import/export mechanism (*e.g.*
:class:`csv <tablib.Dataset.csv>` excludes
:class:`Databook <tablib.Databook>` support),
simply don't define the respective functions.
Appropriate errors will be raised.
2. Add your new format module to the :class:`tablib.formats.available` tuple.
3. Add a mock property to the :class:`Dataset <tablib.Dataset>` class with verbose `reStructured Text`_ docstring.
This alleviates IDE confusion, and allows for pretty auto-generated Sphinx_ documentation.
4. Write respective :ref:`tests <testing>`.
.. _testing:
--------------
Testing Tablib
--------------
Testing is crucial to Tablib's stability.
This stable project is used in production by many companies and developers,
so it is important to be certain that every version released is fully operational.
When developing a new feature for Tablib, be sure to write proper tests for it as well.
When developing a feature for Tablib,
the easiest way to test your changes for potential issues is to simply run the test suite directly.
.. code-block:: console
$ tox
----------------------
Continuous Integration
----------------------
Every pull request is automatically tested and inspected upon receipt with `Travis CI`_.
If you broke the build, you will receive an email accordingly.
Anyone may view the build status and history at any time.
https://travis-ci.org/jazzband/tablib
Additional reports will also be included here in the future, including :pep:`8` checks and stress reports for extremely large datasets.
.. _`Travis CI`: https://travis-ci.org/
.. _docs:
-----------------
Building the Docs
-----------------
Documentation is written in the powerful, flexible,
and standard Python documentation format, `reStructured Text`_.
Documentation builds are powered by the powerful Pocoo project, Sphinx_.
The :ref:`API Documentation <api>` is mostly documented inline throughout the module.
The Docs live in ``tablib/docs``.
In order to build them, you will first need to install Sphinx.
.. code-block:: console
$ pip install sphinx
Then, to build an HTML version of the docs, simply run the following from the ``docs`` directory:
.. code-block:: console
$ make html
Your ``docs/_build/html`` directory will then contain an HTML representation of the documentation,
ready for publication on most web servers.
You can also generate the documentation in **epub**, **latex**, **json**, *&c* similarly.
.. _`reStructured Text`: http://docutils.sourceforge.net/rst.html
.. _Sphinx: http://sphinx.pocoo.org
.. _`GitHub Pages`: https://pages.github.com
----------
Make sure to check out the :ref:`API Documentation <api>`.
-116
View File
@@ -1,116 +0,0 @@
.. Tablib documentation master file, created by
sphinx-quickstart on Tue Oct 5 15:25:21 2010.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Tablib: Pythonic Tabular Datasets
=================================
Release v\ |version|. (:ref:`Installation <install>`)
.. Contents:
..
.. .. toctree::
.. :maxdepth: 2
..
.. Indices and tables
.. ==================
..
.. * :ref:`genindex`
.. * :ref:`modindex`
.. * :ref:`search`
Tablib is an `MIT Licensed <https://mit-license.org/>`_ format-agnostic tabular dataset library, written in Python.
It allows you to import, export, and manipulate tabular data sets.
Advanced features include segregation, dynamic columns, tags & filtering,
and seamless format import & export.
::
>>> data = tablib.Dataset(headers=['First Name', 'Last Name', 'Age'])
>>> for i in [('Kenneth', 'Reitz', 22), ('Bessie', 'Monke', 21)]:
... data.append(i)
>>> print(data.export('json'))
[{"Last Name": "Reitz", "First Name": "Kenneth", "Age": 22}, {"Last Name": "Monke", "First Name": "Bessie", "Age": 21}]
>>> print(data.export('yaml'))
- {Age: 22, First Name: Kenneth, Last Name: Reitz}
- {Age: 21, First Name: Bessie, Last Name: Monke}
>>> data.export('xlsx')
<redacted binary data>
>>> data.export('df')
First Name Last Name Age
0 Kenneth Reitz 22
1 Bessie Monke 21
Testimonials
------------
`National Geographic <https://www.nationalgeographic.com/>`_,
`Digg, Inc <https://digg.com/>`_,
`Northrop Grumman <https://www.northropgrumman.com/>`_,
`Discovery Channel <https://dsc.discovery.com/>`_,
and `The Sunlight Foundation <https://sunlightfoundation.com/>`_ use Tablib internally.
**Greg Thorton**
Tablib by @kennethreitz saved my life.
I had to consolidate like 5 huge poorly maintained lists of domains and data.
It was a breeze!
**Dave Coutts**
It's turning into one of my most used modules of 2010.
You really hit a sweet spot for managing tabular data with a minimal amount of code and effort.
**Joshua Ourisman**
Tablib has made it so much easier to deal with the inevitable 'I want an Excel file!' requests from clients...
**Brad Montgomery**
I think you nailed the "Python Zen" with tablib.
Thanks again for an awesome lib!
User's Guide
------------
This part of the documentation, which is mostly prose, begins with some background information about Tablib, then focuses on step-by-step instructions for getting the most out of your datasets.
.. toctree::
:maxdepth: 2
intro
.. toctree::
:maxdepth: 2
install
.. toctree::
:maxdepth: 2
tutorial
.. toctree::
:maxdepth: 2
development
API Reference
-------------
If you are looking for information on a specific function, class or
method, this part of the documentation is for you.
.. toctree::
:maxdepth: 2
api
-67
View File
@@ -1,67 +0,0 @@
.. _install:
Installation
============
This part of the documentation covers the installation of Tablib. The first step to using any software package is getting it properly installed.
.. _installing:
-----------------
Installing Tablib
-----------------
Distribute & Pip
----------------
Of course, the recommended way to install Tablib is with `pip <https://pip.pypa.io>`_:
.. code-block:: console
$ pip install tablib[pandas]
-------------------
Download the Source
-------------------
You can also install tablib from source.
The latest release (|version|) is available from GitHub.
* tarball_
* zipball_
.. _
Once you have a copy of the source,
you can embed it in your Python package,
or install it into your site-packages easily.
.. code-block:: console
$ python setup.py install
To download the full source history from Git, see :ref:`Source Control <scm>`.
.. _tarball: https://github.com/jazzband/tablib/tarball/master
.. _zipball: https://github.com/jazzband/tablib/zipball/master
.. _updates:
Staying Updated
---------------
The latest version of Tablib will always be available here:
* PyPI: https://pypi.org/project/tablib/
* GitHub: https://github.com/jazzband/tablib/
When a new version is available, upgrading is simple::
$ pip install tablib --upgrade
Now, go get a :ref:`Quick Start <quickstart>`.
-84
View File
@@ -1,84 +0,0 @@
.. _intro:
Introduction
============
This part of the documentation covers all the interfaces of Tablib.
Tablib is a format-agnostic tabular dataset library, written in Python.
It allows you to Pythonically import, export, and manipulate tabular data sets.
Advanced features include segregation, dynamic columns, tags / filtering, and
seamless format import/export.
Philosophy
----------
Tablib was developed with a few :pep:`20` idioms in mind.
#. Beautiful is better than ugly.
#. Explicit is better than implicit.
#. Simple is better than complex.
#. Complex is better than complicated.
#. Readability counts.
All contributions to Tablib should keep these important rules in mind.
.. mit:
MIT License
-----------
A large number of open source projects you find today are `GPL Licensed`_.
While the GPL has its time and place, it should most certainly not be your
go-to license for your next open source project.
A project that is released as GPL cannot be used in any commercial product
without the product itself also being offered as open source. The MIT, BSD, and
ISC licenses are great alternatives to the GPL that allow your open-source
software to be used in proprietary, closed-source software.
Tablib is released under terms of `The MIT License`_.
.. _`GPL Licensed`: https://opensource.org/licenses/gpl-license.php
.. _`The MIT License`: https://opensource.org/licenses/mit-license.php
.. _license:
Tablib License
--------------
Copyright 2017 Kenneth Reitz
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
.. _pythonsupport:
Pythons Supported
-----------------
At this time, the following Python versions are officially supported:
* CPython 2.7
* CPython 3.5
* CPython 3.6
* CPython 3.7
Now, go :ref:`Install Tablib <install>`.
-118
View File
@@ -1,118 +0,0 @@
\definecolor{TitleColor}{rgb}{0,0,0}
\definecolor{InnerLinkColor}{rgb}{0,0,0}
\renewcommand{\maketitle}{%
\begin{titlepage}%
\let\footnotesize\small
\let\footnoterule\relax
\ifsphinxpdfoutput
\begingroup
% This \def is required to deal with multi-line authors; it
% changes \\ to ', ' (comma-space), making it pass muster for
% generating document info in the PDF file.
\def\\{, }
\pdfinfo{
/Author (\@author)
/Title (\@title)
}
\endgroup
\fi
\begin{flushright}%
%\sphinxlogo%
{\center
\vspace*{3cm}
\includegraphics{logo.pdf}
\vspace{3cm}
\par
{\rm\Huge \@title \par}%
{\em\LARGE \py@release\releaseinfo \par}
{\large
\@date \par
\py@authoraddress \par
}}%
\end{flushright}%\par
\@thanks
\end{titlepage}%
\cleardoublepage%
\setcounter{footnote}{0}%
\let\thanks\relax\let\maketitle\relax
%\gdef\@thanks{}\gdef\@author{}\gdef\@title{}
}
\fancypagestyle{normal}{
\fancyhf{}
\fancyfoot[LE,RO]{{\thepage}}
\fancyfoot[LO]{{\nouppercase{\rightmark}}}
\fancyfoot[RE]{{\nouppercase{\leftmark}}}
\fancyhead[LE,RO]{{ \@title, \py@release}}
\renewcommand{\headrulewidth}{0.4pt}
\renewcommand{\footrulewidth}{0.4pt}
}
\fancypagestyle{plain}{
\fancyhf{}
\fancyfoot[LE,RO]{{\thepage}}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0.4pt}
}
\titleformat{\section}{\Large}%
{\py@TitleColor\thesection}{0.5em}{\py@TitleColor}{\py@NormalColor}
\titleformat{\subsection}{\large}%
{\py@TitleColor\thesubsection}{0.5em}{\py@TitleColor}{\py@NormalColor}
\titleformat{\subsubsection}{}%
{\py@TitleColor\thesubsubsection}{0.5em}{\py@TitleColor}{\py@NormalColor}
\titleformat{\paragraph}{\large}%
{\py@TitleColor}{0em}{\py@TitleColor}{\py@NormalColor}
\ChNameVar{\raggedleft\normalsize}
\ChNumVar{\raggedleft \bfseries\Large}
\ChTitleVar{\raggedleft \rm\Huge}
\renewcommand\thepart{\@Roman\c@part}
\renewcommand\part{%
\pagestyle{empty}
\if@noskipsec \leavevmode \fi
\cleardoublepage
\vspace*{6cm}%
\@afterindentfalse
\secdef\@part\@spart}
\def\@part[#1]#2{%
\ifnum \c@secnumdepth >\m@ne
\refstepcounter{part}%
\addcontentsline{toc}{part}{\thepart\hspace{1em}#1}%
\else
\addcontentsline{toc}{part}{#1}%
\fi
{\parindent \z@ %\center
\interlinepenalty \@M
\normalfont
\ifnum \c@secnumdepth >\m@ne
\rm\Large \partname~\thepart
\par\nobreak
\fi
\MakeUppercase{\rm\Huge #2}%
\markboth{}{}\par}%
\nobreak
\vskip 8ex
\@afterheading}
\def\@spart#1{%
{\parindent \z@ %\center
\interlinepenalty \@M
\normalfont
\huge \bfseries #1\par}%
\nobreak
\vskip 3ex
\@afterheading}
% use inconsolata font
\usepackage{inconsolata}
% fix single quotes, for inconsolata. (does not work)
%%\usepackage{textcomp}
%%\begingroup
%% \catcode`'=\active
%% \g@addto@macro\@noligs{\let'\textsinglequote}
%% \endgroup
%%\endinput
-395
View File
@@ -1,395 +0,0 @@
.. _quickstart:
==========
Quickstart
==========
Eager to get started?
This page gives a good introduction in how to get started with Tablib.
This assumes you already have Tablib installed.
If you do not, head over to the :ref:`Installation <install>` section.
First, make sure that:
* Tablib is :ref:`installed <install>`
* Tablib is :ref:`up-to-date <updates>`
Let's get started with some simple use cases and examples.
------------------
Creating a Dataset
------------------
A :class:`Dataset <tablib.Dataset>` is nothing more than what its name implies—a set of data.
Creating your own instance of the :class:`tablib.Dataset` object is simple. ::
data = tablib.Dataset()
You can now start filling this :class:`Dataset <tablib.Dataset>` object with data.
.. admonition:: Example Context
From here on out, if you see ``data``, assume that it's a fresh
:class:`Dataset <tablib.Dataset>` object.
-----------
Adding Rows
-----------
Let's say you want to collect a simple list of names. ::
# collection of names
names = ['Kenneth Reitz', 'Bessie Monke']
for name in names:
# split name appropriately
fname, lname = name.split()
# add names to Dataset
data.append([fname, lname])
You can get a nice, Pythonic view of the dataset at any time with :class:`Dataset.dict`::
>>> data.dict
[('Kenneth', 'Reitz'), ('Bessie', 'Monke')]
--------------
Adding Headers
--------------
It's time to enhance our :class:`Dataset` by giving our columns some titles.
To do so, set :class:`Dataset.headers`. ::
data.headers = ['First Name', 'Last Name']
Now our data looks a little different. ::
>>> data.dict
[{'Last Name': 'Reitz', 'First Name': 'Kenneth'},
{'Last Name': 'Monke', 'First Name': 'Bessie'}]
--------------
Adding Columns
--------------
Now that we have a basic :class:`Dataset` in place, let's add a column of **ages** to it. ::
data.append_col([22, 20], header='Age')
Let's view the data now. ::
>>> data.dict
[{'Last Name': 'Reitz', 'First Name': 'Kenneth', 'Age': 22},
{'Last Name': 'Monke', 'First Name': 'Bessie', 'Age': 20}]
It's that easy.
--------------
Importing Data
--------------
Creating a :class:`tablib.Dataset` object by importing a pre-existing file is simple. ::
imported_data = Dataset().load(open('data.csv').read())
This detects what sort of data is being passed in, and uses an appropriate formatter to do the import. So you can import from a variety of different file types.
--------------
Exporting Data
--------------
Tablib's killer feature is the ability to export your :class:`Dataset` objects into a number of formats.
**Comma-Separated Values** ::
>>> data.export('csv')
Last Name,First Name,Age
Reitz,Kenneth,22
Monke,Bessie,20
**JavaScript Object Notation** ::
>>> data.export('json')
[{"Last Name": "Reitz", "First Name": "Kenneth", "Age": 22}, {"Last Name": "Monke", "First Name": "Bessie", "Age": 20}]
**YAML Ain't Markup Language** ::
>>> data.export('yaml')
- {Age: 22, First Name: Kenneth, Last Name: Reitz}
- {Age: 20, First Name: Bessie, Last Name: Monke}
**Microsoft Excel** ::
>>> data.export('xls')
<redacted binary data>
**Pandas DataFrame** ::
>>> data.export('df')
First Name Last Name Age
0 Kenneth Reitz 22
1 Bessie Monke 21
------------------------
Selecting Rows & Columns
------------------------
You can slice and dice your data, just like a standard Python list. ::
>>> data[0]
('Kenneth', 'Reitz', 22)
If we had a set of data consisting of thousands of rows,
it could be useful to get a list of values in a column.
To do so, we access the :class:`Dataset` as if it were a standard Python dictionary. ::
>>> data['First Name']
['Kenneth', 'Bessie']
You can also access the column using its index. ::
>>> data.headers
['Last Name', 'First Name', 'Age']
>>> data.get_col(1)
['Kenneth', 'Bessie']
Let's find the average age. ::
>>> ages = data['Age']
>>> float(sum(ages)) / len(ages)
21.0
-----------------------
Removing Rows & Columns
-----------------------
It's easier than you could imagine. Delete a column::
>>> del data['Col Name']
Delete a range of rows::
>>> del data[0:12]
==============
Advanced Usage
==============
This part of the documentation services to give you an idea that are otherwise hard to extract from the :ref:`API Documentation <api>`
And now for something completely different.
.. _dyncols:
---------------
Dynamic Columns
---------------
.. versionadded:: 0.8.3
Thanks to Josh Ourisman, Tablib now supports adding dynamic columns.
A dynamic column is a single callable object (*e.g.* a function).
Let's add a dynamic column to our :class:`Dataset` object.
In this example, we have a function that generates a random grade for our students. ::
import random
def random_grade(row):
"""Returns a random integer for entry."""
return (random.randint(60,100)/100.0)
data.append_col(random_grade, header='Grade')
Let's have a look at our data. ::
>>> data.export('yaml')
- {Age: 22, First Name: Kenneth, Grade: 0.6, Last Name: Reitz}
- {Age: 20, First Name: Bessie, Grade: 0.75, Last Name: Monke}
Let's remove that column. ::
>>> del data['Grade']
When you add a dynamic column, the first argument that is passed in to the given callable is the current data row.
You can use this to perform calculations against your data row.
For example, we can use the data available in the row to guess the gender of a student. ::
def guess_gender(row):
"""Calculates gender of given student data row."""
m_names = ('Kenneth', 'Mike', 'Yuri')
f_names = ('Bessie', 'Samantha', 'Heather')
name = row[0]
if name in m_names:
return 'Male'
elif name in f_names:
return 'Female'
else:
return 'Unknown'
Adding this function to our dataset as a dynamic column would result in: ::
>>> data.export('yaml')
- {Age: 22, First Name: Kenneth, Gender: Male, Last Name: Reitz}
- {Age: 20, First Name: Bessie, Gender: Female, Last Name: Monke}
.. _tags:
----------------------------
Filtering Datasets with Tags
----------------------------
.. versionadded:: 0.9.0
When constructing a :class:`Dataset` object,
you can add tags to rows by specifying the ``tags`` parameter.
This allows you to filter your :class:`Dataset` later.
This can be useful to separate rows of data based on arbitrary criteria
(*e.g.* origin) that you don't want to include in your :class:`Dataset`.
Let's tag some students. ::
students = tablib.Dataset()
students.headers = ['first', 'last']
students.rpush(['Kenneth', 'Reitz'], tags=['male', 'technical'])
students.rpush(['Bessie', 'Monke'], tags=['female', 'creative'])
Now that we have extra meta-data on our rows, we can easily filter our :class:`Dataset`. Let's just see Male students. ::
>>> students.filter(['male']).yaml
- {first: Kenneth, Last: Reitz}
It's that simple. The original :class:`Dataset` is untouched.
Open an Excel Workbook and read first sheet
-------------------------------------------
To open an Excel 2007 and later workbook with a single sheet (or a workbook with multiple sheets but you just want the first sheet), use the following:
data = tablib.Dataset()
data.xlsx = open('my_excel_file.xlsx', 'rb').read()
print(data)
Excel Workbook With Multiple Sheets
------------------------------------
When dealing with a large number of :class:`Datasets <Dataset>` in spreadsheet format,
it's quite common to group multiple spreadsheets into a single Excel file, known as a Workbook.
Tablib makes it extremely easy to build workbooks with the handy :class:`Databook` class.
Let's say we have 3 different :class:`Datasets <Dataset>`.
All we have to do is add them to a :class:`Databook` object... ::
book = tablib.Databook((data1, data2, data3))
... and export to Excel just like :class:`Datasets <Dataset>`. ::
with open('students.xls', 'wb') as f:
f.write(book.xls)
The resulting ``students.xls`` file will contain a separate spreadsheet for each :class:`Dataset` object in the :class:`Databook`.
.. admonition:: Binary Warning
Make sure to open the output file in binary mode.
.. _separators:
----------
Separators
----------
.. versionadded:: 0.8.2
When constructing a spreadsheet,
it's often useful to create a blank row containing information on the upcoming data. So,
::
daniel_tests = [
('11/24/09', 'Math 101 Mid-term Exam', 56.),
('05/24/10', 'Math 101 Final Exam', 62.)
]
suzie_tests = [
('11/24/09', 'Math 101 Mid-term Exam', 56.),
('05/24/10', 'Math 101 Final Exam', 62.)
]
# Create new dataset
tests = tablib.Dataset()
tests.headers = ['Date', 'Test Name', 'Grade']
# Daniel's Tests
tests.append_separator('Daniel\'s Scores')
for test_row in daniel_tests:
tests.append(test_row)
# Susie's Tests
tests.append_separator('Susie\'s Scores')
for test_row in suzie_tests:
tests.append(test_row)
# Write spreadsheet to disk
with open('grades.xls', 'wb') as f:
f.write(tests.export('xls'))
The resulting **tests.xls** will have the following layout:
Daniel's Scores:
* '11/24/09', 'Math 101 Mid-term Exam', 56.
* '05/24/10', 'Math 101 Final Exam', 62.
Suzie's Scores:
* '11/24/09', 'Math 101 Mid-term Exam', 56.
* '05/24/10', 'Math 101 Final Exam', 62.
.. admonition:: Format Support
At this time, only :class:`Excel <Dataset.xls>` output supports separators.
----
Now, go check out the :ref:`API Documentation <api>` or begin :ref:`Tablib Development <development>`.
Vendored
+7
View File
@@ -0,0 +1,7 @@
from fabric.api import *
def scrub():
""" Death to the bytecode! """
local("rm -fr dist build")
local("find . -name \"*.pyc\" -exec rm '{}' ';'")
-4
View File
@@ -1,4 +0,0 @@
[pytest]
norecursedirs = .git .*
addopts = -rsxX --showlocals --tb=native --cov=tablib --cov-report xml --cov-report term --cov-report html
python_paths = .
+3
View File
@@ -0,0 +1,3 @@
xlwt
simplejson
PyYAML
Executable → Regular
+32 -45
View File
@@ -2,56 +2,43 @@
# -*- coding: utf-8 -*-
import os
import re
import sys
from setuptools import find_packages, setup
from distutils.core import setup
install = [
'odfpy',
'openpyxl>=2.4.0',
'backports.csv;python_version<"3.0"',
'markuppy',
'xlrd',
'xlwt',
'pyyaml',
]
def publish():
"""Publish to PyPi"""
os.system("python setup.py sdist upload")
if sys.argv[-1] == "publish":
publish()
sys.exit()
setup(
name='tablib',
use_scm_version=True,
setup_requires=['setuptools_scm'],
description='Format agnostic tabular data library (XLS, JSON, YAML, CSV)',
long_description=(open('README.md').read() + '\n\n' +
open('HISTORY.md').read()),
long_description_content_type="text/markdown",
author='Kenneth Reitz',
author_email='me@kennethreitz.org',
maintainer='Jazzband',
maintainer_email='roadies@jazzband.co',
url='https://tablib.readthedocs.io',
packages=find_packages(where="src"),
package_dir={"": "src"},
license='MIT',
classifiers=[
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Natural Language :: English',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
],
python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',
install_requires=install,
extras_require={
'pandas': ['pandas'],
},
name='tablib',
version='0.6.4',
description='Format agnostic tabular data library (XLS, JSON, YAML, CSV)',
long_description=open('README.rst').read() + '\n\n' +
open('HISTORY.rst').read(),
author='Kenneth Reitz',
author_email='me@kennethreitz.com',
url='http://github.com/kennethreitz/tablib',
packages=['tablib'],
install_requires=['xlwt', 'simplejson', 'PyYAML'],
license='MIT',
classifiers=(
'Development Status :: 4 - Beta',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python',
# 'Programming Language :: Python :: 2.5',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
),
# entry_points={
# 'console_scripts': [
# 'tabbed = tablib.cli:start',
# ],
# }
)
-13
View File
@@ -1,13 +0,0 @@
""" Tablib. """
from pkg_resources import get_distribution, DistributionNotFound
from tablib.core import (
Databook, Dataset, detect_format, import_set, import_book,
InvalidDatasetType, InvalidDimensions, UnsupportedFormat
)
try:
__version__ = get_distribution(__name__).version
except DistributionNotFound:
# package is not installed
__version__ = None
-36
View File
@@ -1,36 +0,0 @@
# -*- coding: utf-8 -*-
"""
tablib.compat
~~~~~~~~~~~~~
Tablib compatiblity module.
"""
import sys
is_py3 = (sys.version_info[0] > 2)
if is_py3:
from io import StringIO
from statistics import median
from itertools import zip_longest as izip_longest
import csv
import tablib.packages.dbfpy3 as dbfpy
unicode = str
xrange = range
else:
from StringIO import StringIO
from tablib.packages.statistics import median
from itertools import izip_longest
from backports import csv
import tablib.packages.dbfpy as dbfpy
unicode = unicode
xrange = xrange
from MarkupPy import markup # Kept temporarily to avoid breaking existing imports
-1160
View File
File diff suppressed because it is too large Load Diff
-21
View File
@@ -1,21 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - formats
"""
from . import _csv as csv
from . import _json as json
from . import _xls as xls
from . import _yaml as yaml
from . import _tsv as tsv
from . import _html as html
from . import _xlsx as xlsx
from . import _ods as ods
from . import _dbf as dbf
from . import _latex as latex
from . import _df as df
from . import _rst as rst
from . import _jira as jira
# xlsx before as xls (xlrd) can also read xlsx
available = (json, xlsx, xls, yaml, csv, dbf, tsv, html, jira, latex, ods, df, rst)
-59
View File
@@ -1,59 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - *SV Support.
"""
from tablib.compat import csv, StringIO, unicode
title = 'csv'
extensions = ('csv',)
DEFAULT_DELIMITER = unicode(',')
def export_stream_set(dataset, **kwargs):
"""Returns CSV representation of Dataset as file-like."""
stream = StringIO()
kwargs.setdefault('delimiter', DEFAULT_DELIMITER)
_csv = csv.writer(stream, **kwargs)
for row in dataset._package(dicts=False):
_csv.writerow(row)
stream.seek(0)
return stream
def export_set(dataset, **kwargs):
"""Returns CSV representation of Dataset."""
stream = export_stream_set(dataset, **kwargs)
return stream.getvalue()
def import_set(dset, in_stream, headers=True, **kwargs):
"""Returns dataset from CSV stream."""
dset.wipe()
kwargs.setdefault('delimiter', DEFAULT_DELIMITER)
rows = csv.reader(StringIO(in_stream), **kwargs)
for i, row in enumerate(rows):
if (i == 0) and (headers):
dset.headers = row
elif row:
dset.append(row)
def detect(stream, delimiter=DEFAULT_DELIMITER):
"""Returns True if given stream is valid CSV."""
try:
csv.Sniffer().sniff(stream, delimiters=delimiter)
return True
except Exception:
return False
-87
View File
@@ -1,87 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - DBF Support.
"""
import tempfile
import struct
import os
from tablib.compat import StringIO
from tablib.compat import dbfpy
from tablib.compat import is_py3
if is_py3:
from tablib.packages.dbfpy3 import dbf
from tablib.packages.dbfpy3 import dbfnew
from tablib.packages.dbfpy3 import record as dbfrecord
import io
else:
from tablib.packages.dbfpy import dbf
from tablib.packages.dbfpy import dbfnew
from tablib.packages.dbfpy import record as dbfrecord
title = 'dbf'
extensions = ('csv',)
DEFAULT_ENCODING = 'utf-8'
def export_set(dataset):
"""Returns DBF representation of a Dataset"""
new_dbf = dbfnew.dbf_new()
temp_file, temp_uri = tempfile.mkstemp()
# create the appropriate fields based on the contents of the first row
first_row = dataset[0]
for fieldname, field_value in zip(dataset.headers, first_row):
if type(field_value) in [int, float]:
new_dbf.add_field(fieldname, 'N', 10, 8)
else:
new_dbf.add_field(fieldname, 'C', 80)
new_dbf.write(temp_uri)
dbf_file = dbf.Dbf(temp_uri, readOnly=0)
for row in dataset:
record = dbfrecord.DbfRecord(dbf_file)
for fieldname, field_value in zip(dataset.headers, row):
record[fieldname] = field_value
record.store()
dbf_file.close()
dbf_stream = open(temp_uri, 'rb')
if is_py3:
stream = io.BytesIO(dbf_stream.read())
else:
stream = StringIO(dbf_stream.read())
dbf_stream.close()
os.close(temp_file)
os.remove(temp_uri)
return stream.getvalue()
def import_set(dset, in_stream, headers=True):
"""Returns a dataset from a DBF stream."""
dset.wipe()
if is_py3:
_dbf = dbf.Dbf(io.BytesIO(in_stream))
else:
_dbf = dbf.Dbf(StringIO(in_stream))
dset.headers = _dbf.fieldNames
for record in range(_dbf.recordCount):
row = [_dbf[record][f] for f in _dbf.fieldNames]
dset.append(row)
def detect(stream):
"""Returns True if the given stream is valid DBF"""
#_dbf = dbf.Table(StringIO(stream))
try:
if is_py3:
if type(stream) is not bytes:
stream = bytes(stream, 'utf-8')
_dbf = dbf.Dbf(io.BytesIO(stream), readOnly=True)
else:
_dbf = dbf.Dbf(StringIO(stream), readOnly=True)
return True
except Exception:
return False
-43
View File
@@ -1,43 +0,0 @@
""" Tablib - DataFrame Support.
"""
import sys
from io import BytesIO
try:
from pandas import DataFrame
except ImportError:
DataFrame = None
import tablib
from tablib.compat import unicode
title = 'df'
extensions = ('df', )
def detect(stream):
"""Returns True if given stream is a DataFrame."""
if DataFrame is None:
return False
try:
DataFrame(stream)
return True
except ValueError:
return False
def export_set(dset, index=None):
"""Returns DataFrame representation of DataBook."""
if DataFrame is None:
raise NotImplementedError(
'DataFrame Format requires `pandas` to be installed.'
' Try `pip install tablib[pandas]`.')
dataframe = DataFrame(dset.dict, columns=dset.headers)
return dataframe
def import_set(dset, in_stream):
"""Returns dataset from DataFrame."""
dset.wipe()
dset.dict = in_stream.to_dict(orient='records')
-65
View File
@@ -1,65 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - HTML export support.
"""
import codecs
import sys
from io import BytesIO
from MarkupPy import markup
import tablib
from tablib.compat import unicode
BOOK_ENDINGS = 'h3'
title = 'html'
extensions = ('html', )
def export_set(dataset):
"""HTML representation of a Dataset."""
stream = BytesIO()
page = markup.page()
page.table.open()
if dataset.headers is not None:
new_header = [item if item is not None else '' for item in dataset.headers]
page.thead.open()
headers = markup.oneliner.th(new_header)
page.tr(headers)
page.thead.close()
for row in dataset:
new_row = [item if item is not None else '' for item in row]
html_row = markup.oneliner.td(new_row)
page.tr(html_row)
page.table.close()
# Allow unicode characters in output
wrapper = codecs.getwriter("utf8")(stream)
wrapper.writelines(unicode(page))
return stream.getvalue().decode('utf-8')
def export_book(databook):
"""HTML representation of a Databook."""
stream = BytesIO()
# Allow unicode characters in output
wrapper = codecs.getwriter("utf8")(stream)
for i, dset in enumerate(databook._datasets):
title = (dset.title if dset.title else 'Set %s' % (i))
wrapper.write('<%s>%s</%s>\n' % (BOOK_ENDINGS, title, BOOK_ENDINGS))
wrapper.write(dset.html)
wrapper.write('\n')
return stream.getvalue().decode('utf-8')
-39
View File
@@ -1,39 +0,0 @@
# -*- coding: utf-8 -*-
"""Tablib - Jira table export support.
Generates a Jira table from the dataset.
"""
from tablib.compat import unicode
title = 'jira'
def export_set(dataset):
"""Formats the dataset according to the Jira table syntax:
||heading 1||heading 2||heading 3||
|col A1|col A2|col A3|
|col B1|col B2|col B3|
:param dataset: dataset to serialize
:type dataset: tablib.core.Dataset
"""
header = _get_header(dataset.headers) if dataset.headers else ''
body = _get_body(dataset)
return '%s\n%s' % (header, body) if header else body
def _get_body(dataset):
return '\n'.join([_serialize_row(row) for row in dataset])
def _get_header(headers):
return _serialize_row(headers, delimiter='||')
def _serialize_row(row, delimiter='|'):
return '%s%s%s' % (delimiter,
delimiter.join([unicode(item) if item else ' ' for item in row]),
delimiter)
-59
View File
@@ -1,59 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - JSON Support
"""
import decimal
import json
from uuid import UUID
import tablib
title = 'json'
extensions = ('json', 'jsn')
def serialize_objects_handler(obj):
if isinstance(obj, (decimal.Decimal, UUID)):
return str(obj)
elif hasattr(obj, 'isoformat'):
return obj.isoformat()
else:
return obj
def export_set(dataset):
"""Returns JSON representation of Dataset."""
return json.dumps(dataset.dict, default=serialize_objects_handler)
def export_book(databook):
"""Returns JSON representation of Databook."""
return json.dumps(databook._package(), default=serialize_objects_handler)
def import_set(dset, in_stream):
"""Returns dataset from JSON stream."""
dset.wipe()
dset.dict = json.loads(in_stream)
def import_book(dbook, in_stream):
"""Returns databook from JSON stream."""
dbook.wipe()
for sheet in json.loads(in_stream):
data = tablib.Dataset()
data.title = sheet['title']
data.dict = sheet['data']
dbook.add_sheet(data)
def detect(stream):
"""Returns True if given stream is valid JSON."""
try:
json.loads(stream)
return True
except (TypeError, ValueError):
return False
-134
View File
@@ -1,134 +0,0 @@
# -*- coding: utf-8 -*-
"""Tablib - LaTeX table export support.
Generates a LaTeX booktabs-style table from the dataset.
"""
import re
from tablib.compat import unicode
title = 'latex'
extensions = ('tex',)
TABLE_TEMPLATE = """\
%% Note: add \\usepackage{booktabs} to your preamble
%%
\\begin{table}[!htbp]
\\centering
%(CAPTION)s
\\begin{tabular}{%(COLSPEC)s}
\\toprule
%(HEADER)s
%(MIDRULE)s
%(BODY)s
\\bottomrule
\\end{tabular}
\\end{table}
"""
TEX_RESERVED_SYMBOLS_MAP = dict([
('\\', '\\textbackslash{}'),
('{', '\\{'),
('}', '\\}'),
('$', '\\$'),
('&', '\\&'),
('#', '\\#'),
('^', '\\textasciicircum{}'),
('_', '\\_'),
('~', '\\textasciitilde{}'),
('%', '\\%'),
])
TEX_RESERVED_SYMBOLS_RE = re.compile(
'(%s)' % '|'.join(map(re.escape, TEX_RESERVED_SYMBOLS_MAP.keys())))
def export_set(dataset):
"""Returns LaTeX representation of dataset
:param dataset: dataset to serialize
:type dataset: tablib.core.Dataset
"""
caption = '\\caption{%s}' % dataset.title if dataset.title else '%'
colspec = _colspec(dataset.width)
header = _serialize_row(dataset.headers) if dataset.headers else ''
midrule = _midrule(dataset.width)
body = '\n'.join([_serialize_row(row) for row in dataset])
return TABLE_TEMPLATE % dict(CAPTION=caption, COLSPEC=colspec,
HEADER=header, MIDRULE=midrule, BODY=body)
def _colspec(dataset_width):
"""Generates the column specification for the LaTeX `tabular` environment
based on the dataset width.
The first column is justified to the left, all further columns are aligned
to the right.
.. note:: This is only a heuristic and most probably has to be fine-tuned
post export. Column alignment should depend on the data type, e.g., textual
content should usually be aligned to the left while numeric content almost
always should be aligned to the right.
:param dataset_width: width of the dataset
"""
spec = 'l'
for _ in range(1, dataset_width):
spec += 'r'
return spec
def _midrule(dataset_width):
"""Generates the table `midrule`, which may be composed of several
`cmidrules`.
:param dataset_width: width of the dataset to serialize
"""
if not dataset_width or dataset_width == 1:
return '\\midrule'
return ' '.join([_cmidrule(colindex, dataset_width) for colindex in
range(1, dataset_width + 1)])
def _cmidrule(colindex, dataset_width):
"""Generates the `cmidrule` for a single column with appropriate trimming
based on the column position.
:param colindex: Column index
:param dataset_width: width of the dataset
"""
rule = '\\cmidrule(%s){%d-%d}'
if colindex == 1:
# Rule of first column is trimmed on the right
return rule % ('r', colindex, colindex)
if colindex == dataset_width:
# Rule of last column is trimmed on the left
return rule % ('l', colindex, colindex)
# Inner columns are trimmed on the left and right
return rule % ('lr', colindex, colindex)
def _serialize_row(row):
"""Returns string representation of a single row.
:param row: single dataset row
"""
new_row = [_escape_tex_reserved_symbols(unicode(item)) if item else '' for
item in row]
return 6 * ' ' + ' & '.join(new_row) + ' \\\\'
def _escape_tex_reserved_symbols(input):
"""Escapes all TeX reserved symbols ('_', '~', etc.) in a string.
:param input: String to escape
"""
def replace(match):
return TEX_RESERVED_SYMBOLS_MAP[match.group()]
return TEX_RESERVED_SYMBOLS_RE.sub(replace, input)
-104
View File
@@ -1,104 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - ODF Support.
"""
from io import BytesIO
from odf import opendocument, style, table, text
from tablib.compat import unicode
title = 'ods'
extensions = ('ods',)
bold = style.Style(name="bold", family="paragraph")
bold.addElement(style.TextProperties(fontweight="bold", fontweightasian="bold", fontweightcomplex="bold"))
def export_set(dataset):
"""Returns ODF representation of Dataset."""
wb = opendocument.OpenDocumentSpreadsheet()
wb.automaticstyles.addElement(bold)
ws = table.Table(name=dataset.title if dataset.title else 'Tablib Dataset')
wb.spreadsheet.addElement(ws)
dset_sheet(dataset, ws)
stream = BytesIO()
wb.save(stream)
return stream.getvalue()
def export_book(databook):
"""Returns ODF representation of DataBook."""
wb = opendocument.OpenDocumentSpreadsheet()
wb.automaticstyles.addElement(bold)
for i, dset in enumerate(databook._datasets):
ws = table.Table(name=dset.title if dset.title else 'Sheet%s' % (i))
wb.spreadsheet.addElement(ws)
dset_sheet(dset, ws)
stream = BytesIO()
wb.save(stream)
return stream.getvalue()
def dset_sheet(dataset, ws):
"""Completes given worksheet from given Dataset."""
_package = dataset._package(dicts=False)
for i, sep in enumerate(dataset._separators):
_offset = i
_package.insert((sep[0] + _offset), (sep[1],))
for i, row in enumerate(_package):
row_number = i + 1
odf_row = table.TableRow(stylename=bold, defaultcellstylename='bold')
for j, col in enumerate(row):
try:
col = unicode(col, errors='ignore')
except TypeError:
## col is already unicode
pass
ws.addElement(table.TableColumn())
# bold headers
if (row_number == 1) and dataset.headers:
odf_row.setAttribute('stylename', bold)
ws.addElement(odf_row)
cell = table.TableCell()
p = text.P()
p.addElement(text.Span(text=col, stylename=bold))
cell.addElement(p)
odf_row.addElement(cell)
# wrap the rest
else:
try:
if '\n' in col:
ws.addElement(odf_row)
cell = table.TableCell()
cell.addElement(text.P(text=col))
odf_row.addElement(cell)
else:
ws.addElement(odf_row)
cell = table.TableCell()
cell.addElement(text.P(text=col))
odf_row.addElement(cell)
except TypeError:
ws.addElement(odf_row)
cell = table.TableCell()
cell.addElement(text.P(text=col))
odf_row.addElement(cell)
def detect(stream):
if isinstance(stream, bytes):
# load expects a file-like object.
stream = BytesIO(stream)
try:
opendocument.load(stream)
return True
except Exception:
return False
-273
View File
@@ -1,273 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - reStructuredText Support
"""
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from textwrap import TextWrapper
from tablib.compat import (
median,
unicode,
izip_longest,
)
title = 'rst'
extensions = ('rst',)
MAX_TABLE_WIDTH = 80 # Roughly. It may be wider to avoid breaking words.
JUSTIFY_LEFT = 'left'
JUSTIFY_CENTER = 'center'
JUSTIFY_RIGHT = 'right'
JUSTIFY_VALUES = (JUSTIFY_LEFT, JUSTIFY_CENTER, JUSTIFY_RIGHT)
def to_unicode(value):
if isinstance(value, bytes):
return value.decode('utf-8')
return unicode(value)
def _max_word_len(text):
"""
Return the length of the longest word in `text`.
>>> _max_word_len('Python Module for Tabular Datasets')
8
"""
return max((len(word) for word in text.split()))
def _get_column_string_lengths(dataset):
"""
Returns a list of string lengths of each column, and a list of
maximum word lengths.
"""
if dataset.headers:
column_lengths = [[len(h)] for h in dataset.headers]
word_lens = [_max_word_len(h) for h in dataset.headers]
else:
column_lengths = [[] for _ in range(dataset.width)]
word_lens = [0 for _ in range(dataset.width)]
for row in dataset.dict:
values = iter(row.values() if hasattr(row, 'values') else row)
for i, val in enumerate(values):
text = to_unicode(val)
column_lengths[i].append(len(text))
word_lens[i] = max(word_lens[i], _max_word_len(text))
return column_lengths, word_lens
def _row_to_lines(values, widths, wrapper, sep='|', justify=JUSTIFY_LEFT):
"""
Returns a table row of wrapped values as a list of lines
"""
if justify not in JUSTIFY_VALUES:
raise ValueError('Value of "justify" must be one of "{}"'.format(
'", "'.join(JUSTIFY_VALUES)
))
if justify == JUSTIFY_LEFT:
just = lambda text, width: text.ljust(width)
elif justify == JUSTIFY_CENTER:
just = lambda text, width: text.center(width)
else:
just = lambda text, width: text.rjust(width)
lpad = sep + ' ' if sep else ''
rpad = ' ' + sep if sep else ''
pad = ' ' + sep + ' '
cells = []
for value, width in zip(values, widths):
wrapper.width = width
text = to_unicode(value)
cell = wrapper.wrap(text)
cells.append(cell)
lines = izip_longest(*cells, fillvalue='')
lines = (
(just(cell_line, widths[i]) for i, cell_line in enumerate(line))
for line in lines
)
lines = [''.join((lpad, pad.join(line), rpad)) for line in lines]
return lines
def _get_column_widths(dataset, max_table_width=MAX_TABLE_WIDTH, pad_len=3):
"""
Returns a list of column widths proportional to the median length
of the text in their cells.
"""
str_lens, word_lens = _get_column_string_lengths(dataset)
median_lens = [int(median(lens)) for lens in str_lens]
total = sum(median_lens)
if total > max_table_width - (pad_len * len(median_lens)):
column_widths = (max_table_width * l // total for l in median_lens)
else:
column_widths = (l for l in median_lens)
# Allow for separator and padding:
column_widths = (w - pad_len if w > pad_len else w for w in column_widths)
# Rather widen table than break words:
column_widths = [max(w, l) for w, l in zip(column_widths, word_lens)]
return column_widths
def export_set_as_simple_table(dataset, column_widths=None):
"""
Returns reStructuredText grid table representation of dataset.
"""
lines = []
wrapper = TextWrapper()
if column_widths is None:
column_widths = _get_column_widths(dataset, pad_len=2)
border = ' '.join(['=' * w for w in column_widths])
lines.append(border)
if dataset.headers:
lines.extend(_row_to_lines(
dataset.headers,
column_widths,
wrapper,
sep='',
justify=JUSTIFY_CENTER,
))
lines.append(border)
for row in dataset.dict:
values = iter(row.values() if hasattr(row, 'values') else row)
lines.extend(_row_to_lines(values, column_widths, wrapper, ''))
lines.append(border)
return '\n'.join(lines)
def export_set_as_grid_table(dataset, column_widths=None):
"""
Returns reStructuredText grid table representation of dataset.
>>> from tablib import Dataset
>>> from tablib.formats import rst
>>> bits = ((0, 0), (1, 0), (0, 1), (1, 1))
>>> data = Dataset()
>>> data.headers = ['A', 'B', 'A and B']
>>> for a, b in bits:
... data.append([bool(a), bool(b), bool(a * b)])
>>> print(rst.export_set(data, force_grid=True))
+-------+-------+-------+
| A | B | A and |
| | | B |
+=======+=======+=======+
| False | False | False |
+-------+-------+-------+
| True | False | False |
+-------+-------+-------+
| False | True | False |
+-------+-------+-------+
| True | True | True |
+-------+-------+-------+
"""
lines = []
wrapper = TextWrapper()
if column_widths is None:
column_widths = _get_column_widths(dataset)
header_sep = '+=' + '=+='.join(['=' * w for w in column_widths]) + '=+'
row_sep = '+-' + '-+-'.join(['-' * w for w in column_widths]) + '-+'
lines.append(row_sep)
if dataset.headers:
lines.extend(_row_to_lines(
dataset.headers,
column_widths,
wrapper,
justify=JUSTIFY_CENTER,
))
lines.append(header_sep)
for row in dataset.dict:
values = iter(row.values() if hasattr(row, 'values') else row)
lines.extend(_row_to_lines(values, column_widths, wrapper))
lines.append(row_sep)
return '\n'.join(lines)
def _use_simple_table(head0, col0, width0):
"""
Use a simple table if the text in the first column is never wrapped
>>> _use_simple_table('menu', ['egg', 'bacon'], 10)
True
>>> _use_simple_table(None, ['lobster thermidor', 'spam'], 10)
False
"""
if head0 is not None:
head0 = to_unicode(head0)
if len(head0) > width0:
return False
for cell in col0:
cell = to_unicode(cell)
if len(cell) > width0:
return False
return True
def export_set(dataset, **kwargs):
"""
Returns reStructuredText table representation of dataset.
Returns a simple table if the text in the first column is never
wrapped, otherwise returns a grid table.
>>> from tablib import Dataset
>>> bits = ((0, 0), (1, 0), (0, 1), (1, 1))
>>> data = Dataset()
>>> data.headers = ['A', 'B', 'A and B']
>>> for a, b in bits:
... data.append([bool(a), bool(b), bool(a * b)])
>>> table = data.rst
>>> table.split('\\n') == [
... '===== ===== =====',
... ' A B A and',
... ' B ',
... '===== ===== =====',
... 'False False False',
... 'True False False',
... 'False True False',
... 'True True True ',
... '===== ===== =====',
... ]
True
"""
if not dataset.dict:
return ''
force_grid = kwargs.get('force_grid', False)
max_table_width = kwargs.get('max_table_width', MAX_TABLE_WIDTH)
column_widths = _get_column_widths(dataset, max_table_width)
use_simple_table = _use_simple_table(
dataset.headers[0] if dataset.headers else None,
dataset.get_col(0),
column_widths[0],
)
if use_simple_table and not force_grid:
return export_set_as_simple_table(dataset, column_widths)
else:
return export_set_as_grid_table(dataset, column_widths)
def export_book(databook):
"""
reStructuredText representation of a Databook.
Tables are separated by a blank line. All tables use the grid
format.
"""
return '\n\n'.join(export_set(dataset, force_grid=True)
for dataset in databook._datasets)
-30
View File
@@ -1,30 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - TSV (Tab Separated Values) Support.
"""
from tablib.compat import unicode
from tablib.formats._csv import (
export_set as export_set_wrapper,
import_set as import_set_wrapper,
detect as detect_wrapper,
)
title = 'tsv'
extensions = ('tsv',)
DELIMITER = unicode('\t')
def export_set(dataset):
"""Returns TSV representation of Dataset."""
return export_set_wrapper(dataset, delimiter=DELIMITER)
def import_set(dset, in_stream, headers=True):
"""Returns dataset from TSV stream."""
return import_set_wrapper(dset, in_stream, headers=headers, delimiter=DELIMITER)
def detect(stream):
"""Returns True if given stream is valid TSV."""
return detect_wrapper(stream, delimiter=DELIMITER)
-137
View File
@@ -1,137 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - XLS Support.
"""
import sys
from io import BytesIO
from tablib.compat import xrange
import tablib
import xlrd
import xlwt
from xlrd.biffh import XLRDError
title = 'xls'
extensions = ('xls',)
# special styles
wrap = xlwt.easyxf("alignment: wrap on")
bold = xlwt.easyxf("font: bold on")
def detect(stream):
"""Returns True if given stream is a readable excel file."""
try:
xlrd.open_workbook(file_contents=stream)
return True
except Exception:
pass
try:
xlrd.open_workbook(file_contents=stream.read())
return True
except Exception:
pass
try:
xlrd.open_workbook(filename=stream)
return True
except Exception:
return False
def export_set(dataset):
"""Returns XLS representation of Dataset."""
wb = xlwt.Workbook(encoding='utf8')
ws = wb.add_sheet(dataset.title if dataset.title else 'Tablib Dataset')
dset_sheet(dataset, ws)
stream = BytesIO()
wb.save(stream)
return stream.getvalue()
def export_book(databook):
"""Returns XLS representation of DataBook."""
wb = xlwt.Workbook(encoding='utf8')
for i, dset in enumerate(databook._datasets):
ws = wb.add_sheet(dset.title if dset.title else 'Sheet%s' % (i))
dset_sheet(dset, ws)
stream = BytesIO()
wb.save(stream)
return stream.getvalue()
def import_set(dset, in_stream, headers=True):
"""Returns databook from XLS stream."""
dset.wipe()
xls_book = xlrd.open_workbook(file_contents=in_stream)
sheet = xls_book.sheet_by_index(0)
dset.title = sheet.name
for i in xrange(sheet.nrows):
if (i == 0) and (headers):
dset.headers = sheet.row_values(0)
else:
dset.append(sheet.row_values(i))
def import_book(dbook, in_stream, headers=True):
"""Returns databook from XLS stream."""
dbook.wipe()
xls_book = xlrd.open_workbook(file_contents=in_stream)
for sheet in xls_book.sheets():
data = tablib.Dataset()
data.title = sheet.name
for i in xrange(sheet.nrows):
if (i == 0) and (headers):
data.headers = sheet.row_values(0)
else:
data.append(sheet.row_values(i))
dbook.add_sheet(data)
def dset_sheet(dataset, ws):
"""Completes given worksheet from given Dataset."""
_package = dataset._package(dicts=False)
for i, sep in enumerate(dataset._separators):
_offset = i
_package.insert((sep[0] + _offset), (sep[1],))
for i, row in enumerate(_package):
for j, col in enumerate(row):
# bold headers
if (i == 0) and dataset.headers:
ws.write(i, j, col, bold)
# frozen header row
ws.panes_frozen = True
ws.horz_split_pos = 1
# bold separators
elif len(row) < dataset.width:
ws.write(i, j, col, bold)
# wrap the rest
else:
try:
if '\n' in col:
ws.write(i, j, col, wrap)
else:
ws.write(i, j, col)
except TypeError:
ws.write(i, j, col)
-144
View File
@@ -1,144 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - XLSX Support.
"""
import sys
from io import BytesIO
import openpyxl
import tablib
Workbook = openpyxl.workbook.Workbook
ExcelWriter = openpyxl.writer.excel.ExcelWriter
get_column_letter = openpyxl.utils.get_column_letter
from tablib.compat import unicode
title = 'xlsx'
extensions = ('xlsx',)
def detect(stream):
"""Returns True if given stream is a readable excel file."""
if isinstance(stream, bytes):
# load_workbook expects a file-like object.
stream = BytesIO(stream)
try:
openpyxl.reader.excel.load_workbook(stream, read_only=True)
return True
except Exception:
return False
def export_set(dataset, freeze_panes=True):
"""Returns XLSX representation of Dataset."""
wb = Workbook()
ws = wb.worksheets[0]
ws.title = dataset.title if dataset.title else 'Tablib Dataset'
dset_sheet(dataset, ws, freeze_panes=freeze_panes)
stream = BytesIO()
wb.save(stream)
return stream.getvalue()
def export_book(databook, freeze_panes=True):
"""Returns XLSX representation of DataBook."""
wb = Workbook()
for sheet in wb.worksheets:
wb.remove(sheet)
for i, dset in enumerate(databook._datasets):
ws = wb.create_sheet()
ws.title = dset.title if dset.title else 'Sheet%s' % (i)
dset_sheet(dset, ws, freeze_panes=freeze_panes)
stream = BytesIO()
wb.save(stream)
return stream.getvalue()
def import_set(dset, in_stream, headers=True):
"""Returns databook from XLS stream."""
dset.wipe()
xls_book = openpyxl.reader.excel.load_workbook(BytesIO(in_stream), read_only=True)
sheet = xls_book.active
dset.title = sheet.title
for i, row in enumerate(sheet.rows):
row_vals = [c.value for c in row]
if (i == 0) and (headers):
dset.headers = row_vals
else:
dset.append(row_vals)
def import_book(dbook, in_stream, headers=True):
"""Returns databook from XLS stream."""
dbook.wipe()
xls_book = openpyxl.reader.excel.load_workbook(BytesIO(in_stream), read_only=True)
for sheet in xls_book.worksheets:
data = tablib.Dataset()
data.title = sheet.title
for i, row in enumerate(sheet.rows):
row_vals = [c.value for c in row]
if (i == 0) and (headers):
data.headers = row_vals
else:
data.append(row_vals)
dbook.add_sheet(data)
def dset_sheet(dataset, ws, freeze_panes=True):
"""Completes given worksheet from given Dataset."""
_package = dataset._package(dicts=False)
for i, sep in enumerate(dataset._separators):
_offset = i
_package.insert((sep[0] + _offset), (sep[1],))
bold = openpyxl.styles.Font(bold=True)
wrap_text = openpyxl.styles.Alignment(wrap_text=True)
for i, row in enumerate(_package):
row_number = i + 1
for j, col in enumerate(row):
col_idx = get_column_letter(j + 1)
cell = ws['%s%s' % (col_idx, row_number)]
# bold headers
if (row_number == 1) and dataset.headers:
cell.font = bold
if freeze_panes:
# Export Freeze only after first Line
ws.freeze_panes = 'A2'
# bold separators
elif len(row) < dataset.width:
cell.font = bold
# wrap the rest
else:
try:
str_col_value = unicode(col)
except TypeError:
str_col_value = ''
if '\n' in str_col_value:
cell.alignment = wrap_text
try:
cell.value = col
except (ValueError, TypeError):
cell.value = unicode(col)
-53
View File
@@ -1,53 +0,0 @@
# -*- coding: utf-8 -*-
""" Tablib - YAML Support.
"""
import tablib
import yaml
title = 'yaml'
extensions = ('yaml', 'yml')
def export_set(dataset):
"""Returns YAML representation of Dataset."""
return yaml.safe_dump(dataset._package(ordered=False))
def export_book(databook):
"""Returns YAML representation of Databook."""
return yaml.safe_dump(databook._package(ordered=False))
def import_set(dset, in_stream):
"""Returns dataset from YAML stream."""
dset.wipe()
dset.dict = yaml.safe_load(in_stream)
def import_book(dbook, in_stream):
"""Returns databook from YAML stream."""
dbook.wipe()
for sheet in yaml.safe_load(in_stream):
data = tablib.Dataset()
data.title = sheet['title']
data.dict = sheet['data']
dbook.add_sheet(data)
def detect(stream):
"""Returns True if given stream is valid YAML."""
try:
_yaml = yaml.safe_load(stream)
if isinstance(_yaml, (list, tuple, dict)):
return True
else:
return False
except (yaml.parser.ParserError, yaml.reader.ReaderError,
yaml.scanner.ScannerError):
return False
View File
-297
View File
@@ -1,297 +0,0 @@
#! /usr/bin/env python
"""DBF accessing helpers.
FIXME: more documentation needed
Examples:
Create new table, setup structure, add records:
dbf = Dbf(filename, new=True)
dbf.addField(
("NAME", "C", 15),
("SURNAME", "C", 25),
("INITIALS", "C", 10),
("BIRTHDATE", "D"),
)
for (n, s, i, b) in (
("John", "Miller", "YC", (1980, 10, 11)),
("Andy", "Larkin", "", (1980, 4, 11)),
):
rec = dbf.newRecord()
rec["NAME"] = n
rec["SURNAME"] = s
rec["INITIALS"] = i
rec["BIRTHDATE"] = b
rec.store()
dbf.close()
Open existed dbf, read some data:
dbf = Dbf(filename, True)
for rec in dbf:
for fldName in dbf.fieldNames:
print('%s:\t %s (%s)' % (fldName, rec[fldName],
type(rec[fldName])))
print()
dbf.close()
"""
"""History (most recent first):
11-feb-2007 [als] export INVALID_VALUE;
Dbf: added .ignoreErrors, .INVALID_VALUE
04-jul-2006 [als] added export declaration
20-dec-2005 [yc] removed fromStream and newDbf methods:
use argument of __init__ call must be used instead;
added class fields pointing to the header and
record classes.
17-dec-2005 [yc] split to several modules; reimplemented
13-dec-2005 [yc] adapted to the changes of the `strutil` module.
13-sep-2002 [als] support FoxPro Timestamp datatype
15-nov-1999 [jjk] documentation updates, add demo
24-aug-1998 [jjk] add some encodeValue methods (not tested), other tweaks
08-jun-1998 [jjk] fix problems, add more features
20-feb-1998 [jjk] fix problems, add more features
19-feb-1998 [jjk] add create/write capabilities
18-feb-1998 [jjk] from dbfload.py
"""
__version__ = "$Revision: 1.7 $"[11:-2]
__date__ = "$Date: 2007/02/11 09:23:13 $"[7:-2]
__author__ = "Jeff Kunce <kuncej@mail.conservation.state.mo.us>"
__all__ = ["Dbf"]
from . import header
from . import record
from utils import INVALID_VALUE
class Dbf(object):
"""DBF accessor.
FIXME:
docs and examples needed (dont' forget to tell
about problems adding new fields on the fly)
Implementation notes:
``_new`` field is used to indicate whether this is
a new data table. `addField` could be used only for
the new tables! If at least one record was appended
to the table it's structure couldn't be changed.
"""
__slots__ = ("name", "header", "stream",
"_changed", "_new", "_ignore_errors")
HeaderClass = header.DbfHeader
RecordClass = record.DbfRecord
INVALID_VALUE = INVALID_VALUE
# initialization and creation helpers
def __init__(self, f, readOnly=False, new=False, ignoreErrors=False):
"""Initialize instance.
Arguments:
f:
Filename or file-like object.
new:
True if new data table must be created. Assume
data table exists if this argument is False.
readOnly:
if ``f`` argument is a string file will
be opend in read-only mode; in other cases
this argument is ignored. This argument is ignored
even if ``new`` argument is True.
headerObj:
`header.DbfHeader` instance or None. If this argument
is None, new empty header will be used with the
all fields set by default.
ignoreErrors:
if set, failing field value conversion will return
``INVALID_VALUE`` instead of raising conversion error.
"""
if isinstance(f, basestring):
# a filename
self.name = f
if new:
# new table (table file must be
# created or opened and truncated)
self.stream = file(f, "w+b")
else:
# tabe file must exist
self.stream = file(f, ("r+b", "rb")[bool(readOnly)])
else:
# a stream
self.name = getattr(f, "name", "")
self.stream = f
if new:
# if this is a new table, header will be empty
self.header = self.HeaderClass()
else:
# or instantiated using stream
self.header = self.HeaderClass.fromStream(self.stream)
self.ignoreErrors = ignoreErrors
self._new = bool(new)
self._changed = False
# properties
closed = property(lambda self: self.stream.closed)
recordCount = property(lambda self: self.header.recordCount)
fieldNames = property(
lambda self: [_fld.name for _fld in self.header.fields])
fieldDefs = property(lambda self: self.header.fields)
changed = property(lambda self: self._changed or self.header.changed)
def ignoreErrors(self, value):
"""Update `ignoreErrors` flag on the header object and self"""
self.header.ignoreErrors = self._ignore_errors = bool(value)
ignoreErrors = property(
lambda self: self._ignore_errors,
ignoreErrors,
doc="""Error processing mode for DBF field value conversion
if set, failing field value conversion will return
``INVALID_VALUE`` instead of raising conversion error.
""")
# protected methods
def _fixIndex(self, index):
"""Return fixed index.
This method fails if index isn't a numeric object
(long or int). Or index isn't in a valid range
(less or equal to the number of records in the db).
If ``index`` is a negative number, it will be
treated as a negative indexes for list objects.
Return:
Return value is numeric object maning valid index.
"""
if not isinstance(index, (int, long)):
raise TypeError("Index must be a numeric object")
if index < 0:
# index from the right side
# fix it to the left-side index
index += len(self) + 1
if index >= len(self):
raise IndexError("Record index out of range")
return index
# iterface methods
def close(self):
self.flush()
self.stream.close()
def flush(self):
"""Flush data to the associated stream."""
if self.changed:
self.header.setCurrentDate()
self.header.write(self.stream)
self.stream.flush()
self._changed = False
def indexOfFieldName(self, name):
"""Index of field named ``name``."""
# FIXME: move this to header class
return self.header.fields.index(name)
def newRecord(self):
"""Return new record, which belong to this table."""
return self.RecordClass(self)
def append(self, record):
"""Append ``record`` to the database."""
record.index = self.header.recordCount
record._write()
self.header.recordCount += 1
self._changed = True
self._new = False
def addField(self, *defs):
"""Add field definitions.
For more information see `header.DbfHeader.addField`.
"""
if self._new:
self.header.addField(*defs)
else:
raise TypeError("At least one record was added, "
"structure can't be changed")
# 'magic' methods (representation and sequence interface)
def __repr__(self):
return "Dbf stream '%s'\n" % self.stream + repr(self.header)
def __len__(self):
"""Return number of records."""
return self.recordCount
def __getitem__(self, index):
"""Return `DbfRecord` instance."""
return self.RecordClass.fromStream(self, self._fixIndex(index))
def __setitem__(self, index, record):
"""Write `DbfRecord` instance to the stream."""
record.index = self._fixIndex(index)
record._write()
self._changed = True
self._new = False
# def __del__(self):
# """Flush stream upon deletion of the object."""
# self.flush()
def demo_read(filename):
_dbf = Dbf(filename, True)
for _rec in _dbf:
print()
print(repr(_rec))
_dbf.close()
def demo_create(filename):
_dbf = Dbf(filename, new=True)
_dbf.addField(
("NAME", "C", 15),
("SURNAME", "C", 25),
("INITIALS", "C", 10),
("BIRTHDATE", "D"),
)
for (_n, _s, _i, _b) in (
("John", "Miller", "YC", (1981, 1, 2)),
("Andy", "Larkin", "AL", (1982, 3, 4)),
("Bill", "Clinth", "", (1983, 5, 6)),
("Bobb", "McNail", "", (1984, 7, 8)),
):
_rec = _dbf.newRecord()
_rec["NAME"] = _n
_rec["SURNAME"] = _s
_rec["INITIALS"] = _i
_rec["BIRTHDATE"] = _b
_rec.store()
print(repr(_dbf))
_dbf.close()
if __name__ == '__main__':
import sys
_name = len(sys.argv) > 1 and sys.argv[1] or "county.dbf"
demo_create(_name)
demo_read(_name)
# vim: set et sw=4 sts=4 :
-189
View File
@@ -1,189 +0,0 @@
#!/usr/bin/python
""".DBF creation helpers.
Note: this is a legacy interface. New code should use Dbf class
for table creation (see examples in dbf.py)
TODO:
- handle Memo fields.
- check length of the fields accoring to the
`http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
"""
"""History (most recent first)
04-jul-2006 [als] added export declaration;
updated for dbfpy 2.0
15-dec-2005 [yc] define dbf_new.__slots__
14-dec-2005 [yc] added vim modeline; retab'd; added doc-strings;
dbf_new now is a new class (inherited from object)
??-jun-2000 [--] added by Hans Fiby
"""
__version__ = "$Revision: 1.4 $"[11:-2]
__date__ = "$Date: 2006/07/04 08:18:18 $"[7:-2]
__all__ = ["dbf_new"]
from dbf import *
from fields import *
from header import *
from record import *
class _FieldDefinition(object):
"""Field definition.
This is a simple structure, which contains ``name``, ``type``,
``len``, ``dec`` and ``cls`` fields.
Objects also implement get/setitem magic functions, so fields
could be accessed via sequence iterface, where 'name' has
index 0, 'type' index 1, 'len' index 2, 'dec' index 3 and
'cls' could be located at index 4.
"""
__slots__ = "name", "type", "len", "dec", "cls"
# WARNING: be attentive - dictionaries are mutable!
FLD_TYPES = {
# type: (cls, len)
"C": (DbfCharacterFieldDef, None),
"N": (DbfNumericFieldDef, None),
"L": (DbfLogicalFieldDef, 1),
# FIXME: support memos
# "M": (DbfMemoFieldDef),
"D": (DbfDateFieldDef, 8),
# FIXME: I'm not sure length should be 14 characters!
# but temporary I use it, cuz date is 8 characters
# and time 6 (hhmmss)
"T": (DbfDateTimeFieldDef, 14),
}
def __init__(self, name, type, len=None, dec=0):
_cls, _len = self.FLD_TYPES[type]
if _len is None:
if len is None:
raise ValueError("Field length must be defined")
_len = len
self.name = name
self.type = type
self.len = _len
self.dec = dec
self.cls = _cls
def getDbfField(self):
"Return `DbfFieldDef` instance from the current definition."
return self.cls(self.name, self.len, self.dec)
def appendToHeader(self, dbfh):
"""Create a `DbfFieldDef` instance and append it to the dbf header.
Arguments:
dbfh: `DbfHeader` instance.
"""
_dbff = self.getDbfField()
dbfh.addField(_dbff)
class dbf_new(object):
"""New .DBF creation helper.
Example Usage:
dbfn = dbf_new()
dbfn.add_field("name",'C',80)
dbfn.add_field("price",'N',10,2)
dbfn.add_field("date",'D',8)
dbfn.write("tst.dbf")
Note:
This module cannot handle Memo-fields,
they are special.
"""
__slots__ = ("fields",)
FieldDefinitionClass = _FieldDefinition
def __init__(self):
self.fields = []
def add_field(self, name, typ, len, dec=0):
"""Add field definition.
Arguments:
name:
field name (str object). field name must not
contain ASCII NULs and it's length shouldn't
exceed 10 characters.
typ:
type of the field. this must be a single character
from the "CNLMDT" set meaning character, numeric,
logical, memo, date and date/time respectively.
len:
length of the field. this argument is used only for
the character and numeric fields. all other fields
have fixed length.
FIXME: use None as a default for this argument?
dec:
decimal precision. used only for the numric fields.
"""
self.fields.append(self.FieldDefinitionClass(name, typ, len, dec))
def write(self, filename):
"""Create empty .DBF file using current structure."""
_dbfh = DbfHeader()
_dbfh.setCurrentDate()
for _fldDef in self.fields:
_fldDef.appendToHeader(_dbfh)
_dbfStream = file(filename, "wb")
_dbfh.write(_dbfStream)
_dbfStream.close()
def write_stream(self, stream):
_dbfh = DbfHeader()
_dbfh.setCurrentDate()
for _fldDef in self.fields:
_fldDef.appendToHeader(_dbfh)
_dbfh.write(stream)
if __name__ == '__main__':
# create a new DBF-File
dbfn = dbf_new()
dbfn.add_field("name", 'C', 80)
dbfn.add_field("price", 'N', 10, 2)
dbfn.add_field("date", 'D', 8)
dbfn.write("tst.dbf")
# test new dbf
print("*** created tst.dbf: ***")
dbft = Dbf('tst.dbf', readOnly=0)
print(repr(dbft))
# add a record
rec = DbfRecord(dbft)
rec['name'] = 'something'
rec['price'] = 10.5
rec['date'] = (2000, 1, 12)
rec.store()
# add another record
rec = DbfRecord(dbft)
rec['name'] = 'foo and bar'
rec['price'] = 12234
rec['date'] = (1992, 7, 15)
rec.store()
# show the records
print("*** inserted 2 records into tst.dbf: ***")
print(repr(dbft))
for i1 in range(len(dbft)):
rec = dbft[i1]
for fldName in dbft.fieldNames:
print('%s:\t %s' % (fldName, rec[fldName]))
print()
dbft.close()
# vim: set et sts=4 sw=4 :
-466
View File
@@ -1,466 +0,0 @@
"""DBF fields definitions.
TODO:
- make memos work
"""
"""History (most recent first):
26-may-2009 [als] DbfNumericFieldDef.decodeValue: strip zero bytes
05-feb-2009 [als] DbfDateFieldDef.encodeValue: empty arg produces empty date
16-sep-2008 [als] DbfNumericFieldDef decoding looks for decimal point
in the value to select float or integer return type
13-mar-2008 [als] check field name length in constructor
11-feb-2007 [als] handle value conversion errors
10-feb-2007 [als] DbfFieldDef: added .rawFromRecord()
01-dec-2006 [als] Timestamp columns use None for empty values
31-oct-2006 [als] support field types 'F' (float), 'I' (integer)
and 'Y' (currency);
automate export and registration of field classes
04-jul-2006 [als] added export declaration
10-mar-2006 [als] decode empty values for Date and Logical fields;
show field name in errors
10-mar-2006 [als] fix Numeric value decoding: according to spec,
value always is string representation of the number;
ensure that encoded Numeric value fits into the field
20-dec-2005 [yc] use field names in upper case
15-dec-2005 [yc] field definitions moved from `dbf`.
"""
__version__ = "$Revision: 1.14 $"[11:-2]
__date__ = "$Date: 2009/05/26 05:16:51 $"[7:-2]
__all__ = ["lookupFor",] # field classes added at the end of the module
import datetime
import struct
import sys
from . import utils
## abstract definitions
class DbfFieldDef(object):
"""Abstract field definition.
Child classes must override ``type`` class attribute to provide datatype
infromation of the field definition. For more info about types visit
`http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
Also child classes must override ``defaultValue`` field to provide
default value for the field value.
If child class has fixed length ``length`` class attribute must be
overriden and set to the valid value. None value means, that field
isn't of fixed length.
Note: ``name`` field must not be changed after instantiation.
"""
__slots__ = ("name", "length", "decimalCount",
"start", "end", "ignoreErrors")
# length of the field, None in case of variable-length field,
# or a number if this field is a fixed-length field
length = None
# field type. for more information about fields types visit
# `http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
# must be overriden in child classes
typeCode = None
# default value for the field. this field must be
# overriden in child classes
defaultValue = None
def __init__(self, name, length=None, decimalCount=None,
start=None, stop=None, ignoreErrors=False,
):
"""Initialize instance."""
assert self.typeCode is not None, "Type code must be overriden"
assert self.defaultValue is not None, "Default value must be overriden"
## fix arguments
if len(name) >10:
raise ValueError("Field name \"%s\" is too long" % name)
name = str(name).upper()
if self.__class__.length is None:
if length is None:
raise ValueError("[%s] Length isn't specified" % name)
length = int(length)
if length <= 0:
raise ValueError("[%s] Length must be a positive integer"
% name)
else:
length = self.length
if decimalCount is None:
decimalCount = 0
## set fields
self.name = name
# FIXME: validate length according to the specification at
# http://www.clicketyclick.dk/databases/xbase/format/data_types.html
self.length = length
self.decimalCount = decimalCount
self.ignoreErrors = ignoreErrors
self.start = start
self.end = stop
def __cmp__(self, other):
return cmp(self.name, str(other).upper())
def __hash__(self):
return hash(self.name)
def fromString(cls, string, start, ignoreErrors=False):
"""Decode dbf field definition from the string data.
Arguments:
string:
a string, dbf definition is decoded from. length of
the string must be 32 bytes.
start:
position in the database file.
ignoreErrors:
initial error processing mode for the new field (boolean)
"""
assert len(string) == 32
_length = ord(string[16])
return cls(utils.unzfill(string)[:11], _length, ord(string[17]),
start, start + _length, ignoreErrors=ignoreErrors)
fromString = classmethod(fromString)
def toString(self):
"""Return encoded field definition.
Return:
Return value is a string object containing encoded
definition of this field.
"""
if sys.version_info < (2, 4):
# earlier versions did not support padding character
_name = self.name[:11] + "\0" * (11 - len(self.name))
else:
_name = self.name.ljust(11, '\0')
return (
_name +
self.typeCode +
#data address
chr(0) * 4 +
chr(self.length) +
chr(self.decimalCount) +
chr(0) * 14
)
def __repr__(self):
return "%-10s %1s %3d %3d" % self.fieldInfo()
def fieldInfo(self):
"""Return field information.
Return:
Return value is a (name, type, length, decimals) tuple.
"""
return (self.name, self.typeCode, self.length, self.decimalCount)
def rawFromRecord(self, record):
"""Return a "raw" field value from the record string."""
return record[self.start:self.end]
def decodeFromRecord(self, record):
"""Return decoded field value from the record string."""
try:
return self.decodeValue(self.rawFromRecord(record))
except:
if self.ignoreErrors:
return utils.INVALID_VALUE
else:
raise
def decodeValue(self, value):
"""Return decoded value from string value.
This method shouldn't be used publicly. It's called from the
`decodeFromRecord` method.
This is an abstract method and it must be overridden in child classes.
"""
raise NotImplementedError
def encodeValue(self, value):
"""Return str object containing encoded field value.
This is an abstract method and it must be overriden in child classes.
"""
raise NotImplementedError
## real classes
class DbfCharacterFieldDef(DbfFieldDef):
"""Definition of the character field."""
typeCode = "C"
defaultValue = ""
def decodeValue(self, value):
"""Return string object.
Return value is a ``value`` argument with stripped right spaces.
"""
return value.rstrip(" ")
def encodeValue(self, value):
"""Return raw data string encoded from a ``value``."""
return str(value)[:self.length].ljust(self.length)
class DbfNumericFieldDef(DbfFieldDef):
"""Definition of the numeric field."""
typeCode = "N"
# XXX: now I'm not sure it was a good idea to make a class field
# `defaultValue` instead of a generic method as it was implemented
# previously -- it's ok with all types except number, cuz
# if self.decimalCount is 0, we should return 0 and 0.0 otherwise.
defaultValue = 0
def decodeValue(self, value):
"""Return a number decoded from ``value``.
If decimals is zero, value will be decoded as an integer;
or as a float otherwise.
Return:
Return value is a int (long) or float instance.
"""
value = value.strip(" \0")
if "." in value:
# a float (has decimal separator)
return float(value)
elif value:
# must be an integer
return int(value)
else:
return 0
def encodeValue(self, value):
"""Return string containing encoded ``value``."""
_rv = ("%*.*f" % (self.length, self.decimalCount, value))
if len(_rv) > self.length:
_ppos = _rv.find(".")
if 0 <= _ppos <= self.length:
_rv = _rv[:self.length]
else:
raise ValueError("[%s] Numeric overflow: %s (field width: %i)"
% (self.name, _rv, self.length))
return _rv
class DbfFloatFieldDef(DbfNumericFieldDef):
"""Definition of the float field - same as numeric."""
typeCode = "F"
class DbfIntegerFieldDef(DbfFieldDef):
"""Definition of the integer field."""
typeCode = "I"
length = 4
defaultValue = 0
def decodeValue(self, value):
"""Return an integer number decoded from ``value``."""
return struct.unpack("<i", value)[0]
def encodeValue(self, value):
"""Return string containing encoded ``value``."""
return struct.pack("<i", int(value))
class DbfCurrencyFieldDef(DbfFieldDef):
"""Definition of the currency field."""
typeCode = "Y"
length = 8
defaultValue = 0.0
def decodeValue(self, value):
"""Return float number decoded from ``value``."""
return struct.unpack("<q", value)[0] / 10000.
def encodeValue(self, value):
"""Return string containing encoded ``value``."""
return struct.pack("<q", round(value * 10000))
class DbfLogicalFieldDef(DbfFieldDef):
"""Definition of the logical field."""
typeCode = "L"
defaultValue = -1
length = 1
def decodeValue(self, value):
"""Return True, False or -1 decoded from ``value``."""
# Note: value always is 1-char string
if value == "?":
return -1
if value in "NnFf ":
return False
if value in "YyTt":
return True
raise ValueError("[%s] Invalid logical value %r" % (self.name, value))
def encodeValue(self, value):
"""Return a character from the "TF?" set.
Return:
Return value is "T" if ``value`` is True
"?" if value is -1 or False otherwise.
"""
if value is True:
return "T"
if value == -1:
return "?"
return "F"
class DbfMemoFieldDef(DbfFieldDef):
"""Definition of the memo field.
Note: memos aren't currenly completely supported.
"""
typeCode = "M"
defaultValue = " " * 10
length = 10
def decodeValue(self, value):
"""Return int .dbt block number decoded from the string object."""
#return int(value)
raise NotImplementedError
def encodeValue(self, value):
"""Return raw data string encoded from a ``value``.
Note: this is an internal method.
"""
#return str(value)[:self.length].ljust(self.length)
raise NotImplementedError
class DbfDateFieldDef(DbfFieldDef):
"""Definition of the date field."""
typeCode = "D"
defaultValue = utils.classproperty(lambda cls: datetime.date.today())
# "yyyymmdd" gives us 8 characters
length = 8
def decodeValue(self, value):
"""Return a ``datetime.date`` instance decoded from ``value``."""
if value.strip():
return utils.getDate(value)
else:
return None
def encodeValue(self, value):
"""Return a string-encoded value.
``value`` argument should be a value suitable for the
`utils.getDate` call.
Return:
Return value is a string in format "yyyymmdd".
"""
if value:
return utils.getDate(value).strftime("%Y%m%d")
else:
return " " * self.length
class DbfDateTimeFieldDef(DbfFieldDef):
"""Definition of the timestamp field."""
# a difference between JDN (Julian Day Number)
# and GDN (Gregorian Day Number). note, that GDN < JDN
JDN_GDN_DIFF = 1721425
typeCode = "T"
defaultValue = utils.classproperty(lambda cls: datetime.datetime.now())
# two 32-bits integers representing JDN and amount of
# milliseconds respectively gives us 8 bytes.
# note, that values must be encoded in LE byteorder.
length = 8
def decodeValue(self, value):
"""Return a `datetime.datetime` instance."""
assert len(value) == self.length
# LE byteorder
_jdn, _msecs = struct.unpack("<2I", value)
if _jdn >= 1:
_rv = datetime.datetime.fromordinal(_jdn - self.JDN_GDN_DIFF)
_rv += datetime.timedelta(0, _msecs / 1000.0)
else:
# empty date
_rv = None
return _rv
def encodeValue(self, value):
"""Return a string-encoded ``value``."""
if value:
value = utils.getDateTime(value)
# LE byteorder
_rv = struct.pack("<2I", value.toordinal() + self.JDN_GDN_DIFF,
(value.hour * 3600 + value.minute * 60 + value.second) * 1000)
else:
_rv = "\0" * self.length
assert len(_rv) == self.length
return _rv
_fieldsRegistry = {}
def registerField(fieldCls):
"""Register field definition class.
``fieldCls`` should be subclass of the `DbfFieldDef`.
Use `lookupFor` to retrieve field definition class
by the type code.
"""
assert fieldCls.typeCode is not None, "Type code isn't defined"
# XXX: use fieldCls.typeCode.upper()? in case of any decign
# don't forget to look to the same comment in ``lookupFor`` method
_fieldsRegistry[fieldCls.typeCode] = fieldCls
def lookupFor(typeCode):
"""Return field definition class for the given type code.
``typeCode`` must be a single character. That type should be
previously registered.
Use `registerField` to register new field class.
Return:
Return value is a subclass of the `DbfFieldDef`.
"""
# XXX: use typeCode.upper()? in case of any decign don't
# forget to look to the same comment in ``registerField``
return _fieldsRegistry[typeCode]
## register generic types
for (_name, _val) in globals().items():
if isinstance(_val, type) and issubclass(_val, DbfFieldDef) \
and (_name != "DbfFieldDef"):
__all__.append(_name)
registerField(_val)
del _name, _val
# vim: et sts=4 sw=4 :
-275
View File
@@ -1,275 +0,0 @@
"""DBF header definition.
TODO:
- handle encoding of the character fields
(encoding information stored in the DBF header)
"""
"""History (most recent first):
16-sep-2010 [als] fromStream: fix century of the last update field
11-feb-2007 [als] added .ignoreErrors
10-feb-2007 [als] added __getitem__: return field definitions
by field name or field number (zero-based)
04-jul-2006 [als] added export declaration
15-dec-2005 [yc] created
"""
__version__ = "$Revision: 1.6 $"[11:-2]
__date__ = "$Date: 2010/09/16 05:06:39 $"[7:-2]
__all__ = ["DbfHeader"]
try:
import cStringIO
except ImportError:
# when we're in python3, we cStringIO has been replaced by io.StringIO
import io as cStringIO
import datetime
import struct
import time
from . import fields
from . import utils
class DbfHeader(object):
"""Dbf header definition.
For more information about dbf header format visit
`http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_STRUCT`
Examples:
Create an empty dbf header and add some field definitions:
dbfh = DbfHeader()
dbfh.addField(("name", "C", 10))
dbfh.addField(("date", "D"))
dbfh.addField(DbfNumericFieldDef("price", 5, 2))
Create a dbf header with field definitions:
dbfh = DbfHeader([
("name", "C", 10),
("date", "D"),
DbfNumericFieldDef("price", 5, 2),
])
"""
__slots__ = ("signature", "fields", "lastUpdate", "recordLength",
"recordCount", "headerLength", "changed", "_ignore_errors")
## instance construction and initialization methods
def __init__(self, fields=None, headerLength=0, recordLength=0,
recordCount=0, signature=0x03, lastUpdate=None, ignoreErrors=False,
):
"""Initialize instance.
Arguments:
fields:
a list of field definitions;
recordLength:
size of the records;
headerLength:
size of the header;
recordCount:
number of records stored in DBF;
signature:
version number (aka signature). using 0x03 as a default meaning
"File without DBT". for more information about this field visit
``http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_NOTE_1_TARGET``
lastUpdate:
date of the DBF's update. this could be a string ('yymmdd' or
'yyyymmdd'), timestamp (int or float), datetime/date value,
a sequence (assuming (yyyy, mm, dd, ...)) or an object having
callable ``ticks`` field.
ignoreErrors:
error processing mode for DBF fields (boolean)
"""
self.signature = signature
if fields is None:
self.fields = []
else:
self.fields = list(fields)
self.lastUpdate = utils.getDate(lastUpdate)
self.recordLength = recordLength
self.headerLength = headerLength
self.recordCount = recordCount
self.ignoreErrors = ignoreErrors
# XXX: I'm not sure this is safe to
# initialize `self.changed` in this way
self.changed = bool(self.fields)
# @classmethod
def fromString(cls, string):
"""Return header instance from the string object."""
return cls.fromStream(cStringIO.StringIO(str(string)))
fromString = classmethod(fromString)
# @classmethod
def fromStream(cls, stream):
"""Return header object from the stream."""
stream.seek(0)
_data = stream.read(32)
(_cnt, _hdrLen, _recLen) = struct.unpack("<I2H", _data[4:12])
#reserved = _data[12:32]
_year = ord(_data[1])
if _year < 80:
# dBase II started at 1980. It is quite unlikely
# that actual last update date is before that year.
_year += 2000
else:
_year += 1900
## create header object
_obj = cls(None, _hdrLen, _recLen, _cnt, ord(_data[0]),
(_year, ord(_data[2]), ord(_data[3])))
## append field definitions
# position 0 is for the deletion flag
_pos = 1
_data = stream.read(1)
# The field definitions are ended either by \x0D OR a newline
# character, so we need to handle both when reading from a stream.
# When writing, dbfpy appears to write newlines instead of \x0D.
while _data[0] not in ["\x0D", "\n"]:
_data += stream.read(31)
_fld = fields.lookupFor(_data[11]).fromString(_data, _pos)
_obj._addField(_fld)
_pos = _fld.end
_data = stream.read(1)
return _obj
fromStream = classmethod(fromStream)
## properties
year = property(lambda self: self.lastUpdate.year)
month = property(lambda self: self.lastUpdate.month)
day = property(lambda self: self.lastUpdate.day)
def ignoreErrors(self, value):
"""Update `ignoreErrors` flag on self and all fields"""
self._ignore_errors = value = bool(value)
for _field in self.fields:
_field.ignoreErrors = value
ignoreErrors = property(
lambda self: self._ignore_errors,
ignoreErrors,
doc="""Error processing mode for DBF field value conversion
if set, failing field value conversion will return
``INVALID_VALUE`` instead of raising conversion error.
""")
## object representation
def __repr__(self):
_rv = """\
Version (signature): 0x%02x
Last update: %s
Header length: %d
Record length: %d
Record count: %d
FieldName Type Len Dec
""" % (self.signature, self.lastUpdate, self.headerLength,
self.recordLength, self.recordCount)
_rv += "\n".join(
["%10s %4s %3s %3s" % _fld.fieldInfo() for _fld in self.fields]
)
return _rv
## internal methods
def _addField(self, *defs):
"""Internal variant of the `addField` method.
This method doesn't set `self.changed` field to True.
Return value is a length of the appended records.
Note: this method doesn't modify ``recordLength`` and
``headerLength`` fields. Use `addField` instead of this
method if you don't exactly know what you're doing.
"""
# insure we have dbf.DbfFieldDef instances first (instantiation
# from the tuple could raise an error, in such a case I don't
# wanna add any of the definitions -- all will be ignored)
_defs = []
_recordLength = 0
for _def in defs:
if isinstance(_def, fields.DbfFieldDef):
_obj = _def
else:
(_name, _type, _len, _dec) = (tuple(_def) + (None,) * 4)[:4]
_cls = fields.lookupFor(_type)
_obj = _cls(_name, _len, _dec,
ignoreErrors=self._ignore_errors)
_recordLength += _obj.length
_defs.append(_obj)
# and now extend field definitions and
# update record length
self.fields += _defs
return _recordLength
## interface methods
def addField(self, *defs):
"""Add field definition to the header.
Examples:
dbfh.addField(
("name", "C", 20),
dbf.DbfCharacterFieldDef("surname", 20),
dbf.DbfDateFieldDef("birthdate"),
("member", "L"),
)
dbfh.addField(("price", "N", 5, 2))
dbfh.addField(dbf.DbfNumericFieldDef("origprice", 5, 2))
"""
_oldLen = self.recordLength
self.recordLength += self._addField(*defs)
if not _oldLen:
self.recordLength += 1
# XXX: may be just use:
# self.recordeLength += self._addField(*defs) + bool(not _oldLen)
# recalculate headerLength
self.headerLength = 32 + (32 * len(self.fields)) + 1
self.changed = True
def write(self, stream):
"""Encode and write header to the stream."""
stream.seek(0)
stream.write(self.toString())
stream.write("".join([_fld.toString() for _fld in self.fields]))
stream.write(chr(0x0D)) # cr at end of all hdr data
self.changed = False
def toString(self):
"""Returned 32 chars length string with encoded header."""
return struct.pack("<4BI2H",
self.signature,
self.year - 1900,
self.month,
self.day,
self.recordCount,
self.headerLength,
self.recordLength) + "\0" * 20
def setCurrentDate(self):
"""Update ``self.lastUpdate`` field with current date value."""
self.lastUpdate = datetime.date.today()
def __getitem__(self, item):
"""Return a field definition by numeric index or name string"""
if isinstance(item, basestring):
_name = item.upper()
for _field in self.fields:
if _field.name == _name:
return _field
else:
raise KeyError(item)
else:
# item must be field index
return self.fields[item]
# vim: et sts=4 sw=4 :
-262
View File
@@ -1,262 +0,0 @@
"""DBF record definition.
"""
"""History (most recent first):
11-feb-2007 [als] __repr__: added special case for invalid field values
10-feb-2007 [als] added .rawFromStream()
30-oct-2006 [als] fix record length in .fromStream()
04-jul-2006 [als] added export declaration
20-dec-2005 [yc] DbfRecord.write() -> DbfRecord._write();
added delete() method.
16-dec-2005 [yc] record definition moved from `dbf`.
"""
__version__ = "$Revision: 1.7 $"[11:-2]
__date__ = "$Date: 2007/02/11 09:05:49 $"[7:-2]
__all__ = ["DbfRecord"]
from itertools import izip
import utils
class DbfRecord(object):
"""DBF record.
Instances of this class shouldn't be created manualy,
use `dbf.Dbf.newRecord` instead.
Class implements mapping/sequence interface, so
fields could be accessed via their names or indexes
(names is a preffered way to access fields).
Hint:
Use `store` method to save modified record.
Examples:
Add new record to the database:
db = Dbf(filename)
rec = db.newRecord()
rec["FIELD1"] = value1
rec["FIELD2"] = value2
rec.store()
Or the same, but modify existed
(second in this case) record:
db = Dbf(filename)
rec = db[2]
rec["FIELD1"] = value1
rec["FIELD2"] = value2
rec.store()
"""
__slots__ = "dbf", "index", "deleted", "fieldData"
## creation and initialization
def __init__(self, dbf, index=None, deleted=False, data=None):
"""Instance initialiation.
Arguments:
dbf:
A `Dbf.Dbf` instance this record belonogs to.
index:
An integer record index or None. If this value is
None, record will be appended to the DBF.
deleted:
Boolean flag indicating whether this record
is a deleted record.
data:
A sequence or None. This is a data of the fields.
If this argument is None, default values will be used.
"""
self.dbf = dbf
# XXX: I'm not sure ``index`` is necessary
self.index = index
self.deleted = deleted
if data is None:
self.fieldData = [_fd.defaultValue for _fd in dbf.header.fields]
else:
self.fieldData = list(data)
# XXX: validate self.index before calculating position?
position = property(lambda self: self.dbf.header.headerLength + \
self.index * self.dbf.header.recordLength)
def rawFromStream(cls, dbf, index):
"""Return raw record contents read from the stream.
Arguments:
dbf:
A `Dbf.Dbf` instance containing the record.
index:
Index of the record in the records' container.
This argument can't be None in this call.
Return value is a string containing record data in DBF format.
"""
# XXX: may be write smth assuming, that current stream
# position is the required one? it could save some
# time required to calculate where to seek in the file
dbf.stream.seek(dbf.header.headerLength +
index * dbf.header.recordLength)
return dbf.stream.read(dbf.header.recordLength)
rawFromStream = classmethod(rawFromStream)
def fromStream(cls, dbf, index):
"""Return a record read from the stream.
Arguments:
dbf:
A `Dbf.Dbf` instance new record should belong to.
index:
Index of the record in the records' container.
This argument can't be None in this call.
Return value is an instance of the current class.
"""
return cls.fromString(dbf, cls.rawFromStream(dbf, index), index)
fromStream = classmethod(fromStream)
def fromString(cls, dbf, string, index=None):
"""Return record read from the string object.
Arguments:
dbf:
A `Dbf.Dbf` instance new record should belong to.
string:
A string new record should be created from.
index:
Index of the record in the container. If this
argument is None, record will be appended.
Return value is an instance of the current class.
"""
return cls(dbf, index, string[0]=="*",
[_fd.decodeFromRecord(string) for _fd in dbf.header.fields])
fromString = classmethod(fromString)
## object representation
def __repr__(self):
_template = "%%%ds: %%s (%%s)" % max([len(_fld)
for _fld in self.dbf.fieldNames])
_rv = []
for _fld in self.dbf.fieldNames:
_val = self[_fld]
if _val is utils.INVALID_VALUE:
_rv.append(_template %
(_fld, "None", "value cannot be decoded"))
else:
_rv.append(_template % (_fld, _val, type(_val)))
return "\n".join(_rv)
## protected methods
def _write(self):
"""Write data to the dbf stream.
Note:
This isn't a public method, it's better to
use 'store' instead publically.
Be design ``_write`` method should be called
only from the `Dbf` instance.
"""
self._validateIndex(False)
self.dbf.stream.seek(self.position)
self.dbf.stream.write(self.toString())
# FIXME: may be move this write somewhere else?
# why we should check this condition for each record?
if self.index == len(self.dbf):
# this is the last record,
# we should write SUB (ASCII 26)
self.dbf.stream.write("\x1A")
## utility methods
def _validateIndex(self, allowUndefined=True, checkRange=False):
"""Valid ``self.index`` value.
If ``allowUndefined`` argument is True functions does nothing
in case of ``self.index`` pointing to None object.
"""
if self.index is None:
if not allowUndefined:
raise ValueError("Index is undefined")
elif self.index < 0:
raise ValueError("Index can't be negative (%s)" % self.index)
elif checkRange and self.index <= self.dbf.header.recordCount:
raise ValueError("There are only %d records in the DBF" %
self.dbf.header.recordCount)
## interface methods
def store(self):
"""Store current record in the DBF.
If ``self.index`` is None, this record will be appended to the
records of the DBF this records belongs to; or replaced otherwise.
"""
self._validateIndex()
if self.index is None:
self.index = len(self.dbf)
self.dbf.append(self)
else:
self.dbf[self.index] = self
def delete(self):
"""Mark method as deleted."""
self.deleted = True
def toString(self):
"""Return string packed record values."""
return "".join([" *"[self.deleted]] + [
_def.encodeValue(_dat)
for (_def, _dat) in izip(self.dbf.header.fields, self.fieldData)
])
def asList(self):
"""Return a flat list of fields.
Note:
Change of the list's values won't change
real values stored in this object.
"""
return self.fieldData[:]
def asDict(self):
"""Return a dictionary of fields.
Note:
Change of the dicts's values won't change
real values stored in this object.
"""
return dict([_i for _i in izip(self.dbf.fieldNames, self.fieldData)])
def __getitem__(self, key):
"""Return value by field name or field index."""
if isinstance(key, (long, int)):
# integer index of the field
return self.fieldData[key]
# assuming string field name
return self.fieldData[self.dbf.indexOfFieldName(key)]
def __setitem__(self, key, value):
"""Set field value by integer index of the field or string name."""
if isinstance(key, (int, long)):
# integer index of the field
return self.fieldData[key]
# assuming string field name
self.fieldData[self.dbf.indexOfFieldName(key)] = value
# vim: et sts=4 sw=4 :
-170
View File
@@ -1,170 +0,0 @@
"""String utilities.
TODO:
- allow strings in getDateTime routine;
"""
"""History (most recent first):
11-feb-2007 [als] added INVALID_VALUE
10-feb-2007 [als] allow date strings padded with spaces instead of zeroes
20-dec-2005 [yc] handle long objects in getDate/getDateTime
16-dec-2005 [yc] created from ``strutil`` module.
"""
__version__ = "$Revision: 1.4 $"[11:-2]
__date__ = "$Date: 2007/02/11 08:57:17 $"[7:-2]
import datetime
import time
def unzfill(str):
"""Return a string without ASCII NULs.
This function searchers for the first NUL (ASCII 0) occurance
and truncates string till that position.
"""
try:
return str[:str.index('\0')]
except ValueError:
return str
def getDate(date=None):
"""Return `datetime.date` instance.
Type of the ``date`` argument could be one of the following:
None:
use current date value;
datetime.date:
this value will be returned;
datetime.datetime:
the result of the date.date() will be returned;
string:
assuming "%Y%m%d" or "%y%m%dd" format;
number:
assuming it's a timestamp (returned for example
by the time.time() call;
sequence:
assuming (year, month, day, ...) sequence;
Additionaly, if ``date`` has callable ``ticks`` attribute,
it will be used and result of the called would be treated
as a timestamp value.
"""
if date is None:
# use current value
return datetime.date.today()
if isinstance(date, datetime.date):
return date
if isinstance(date, datetime.datetime):
return date.date()
if isinstance(date, (int, long, float)):
# date is a timestamp
return datetime.date.fromtimestamp(date)
if isinstance(date, basestring):
date = date.replace(" ", "0")
if len(date) == 6:
# yymmdd
return datetime.date(*time.strptime(date, "%y%m%d")[:3])
# yyyymmdd
return datetime.date(*time.strptime(date, "%Y%m%d")[:3])
if hasattr(date, "__getitem__"):
# a sequence (assuming date/time tuple)
return datetime.date(*date[:3])
return datetime.date.fromtimestamp(date.ticks())
def getDateTime(value=None):
"""Return `datetime.datetime` instance.
Type of the ``value`` argument could be one of the following:
None:
use current date value;
datetime.date:
result will be converted to the `datetime.datetime` instance
using midnight;
datetime.datetime:
``value`` will be returned as is;
string:
*** CURRENTLY NOT SUPPORTED ***;
number:
assuming it's a timestamp (returned for example
by the time.time() call;
sequence:
assuming (year, month, day, ...) sequence;
Additionaly, if ``value`` has callable ``ticks`` attribute,
it will be used and result of the called would be treated
as a timestamp value.
"""
if value is None:
# use current value
return datetime.datetime.today()
if isinstance(value, datetime.datetime):
return value
if isinstance(value, datetime.date):
return datetime.datetime.fromordinal(value.toordinal())
if isinstance(value, (int, long, float)):
# value is a timestamp
return datetime.datetime.fromtimestamp(value)
if isinstance(value, basestring):
raise NotImplementedError("Strings aren't currently implemented")
if hasattr(value, "__getitem__"):
# a sequence (assuming date/time tuple)
return datetime.datetime(*tuple(value)[:6])
return datetime.datetime.fromtimestamp(value.ticks())
class classproperty(property):
"""Works in the same way as a ``property``, but for the classes."""
def __get__(self, obj, cls):
return self.fget(cls)
class _InvalidValue(object):
"""Value returned from DBF records when field validation fails
The value is not equal to anything except for itself
and equal to all empty values: None, 0, empty string etc.
In other words, invalid value is equal to None and not equal
to None at the same time.
This value yields zero upon explicit conversion to a number type,
empty string for string types, and False for boolean.
"""
def __eq__(self, other):
return not other
def __ne__(self, other):
return not (other is self)
def __nonzero__(self):
return False
def __int__(self):
return 0
__long__ = __int__
def __float__(self):
return 0.0
def __str__(self):
return ""
def __unicode__(self):
return u""
def __repr__(self):
return "<INVALID>"
# invalid value is a constant singleton
INVALID_VALUE = _InvalidValue()
# vim: set et sts=4 sw=4 :
-297
View File
@@ -1,297 +0,0 @@
#! /usr/bin/env python
"""DBF accessing helpers.
FIXME: more documentation needed
Examples:
Create new table, setup structure, add records:
dbf = Dbf(filename, new=True)
dbf.addField(
("NAME", "C", 15),
("SURNAME", "C", 25),
("INITIALS", "C", 10),
("BIRTHDATE", "D"),
)
for (n, s, i, b) in (
("John", "Miller", "YC", (1980, 10, 11)),
("Andy", "Larkin", "", (1980, 4, 11)),
):
rec = dbf.newRecord()
rec["NAME"] = n
rec["SURNAME"] = s
rec["INITIALS"] = i
rec["BIRTHDATE"] = b
rec.store()
dbf.close()
Open existed dbf, read some data:
dbf = Dbf(filename, True)
for rec in dbf:
for fldName in dbf.fieldNames:
print('%s:\t %s (%s)' % (fldName, rec[fldName],
type(rec[fldName])))
dbf.close()
"""
"""History (most recent first):
11-feb-2007 [als] export INVALID_VALUE;
Dbf: added .ignoreErrors, .INVALID_VALUE
04-jul-2006 [als] added export declaration
20-dec-2005 [yc] removed fromStream and newDbf methods:
use argument of __init__ call must be used instead;
added class fields pointing to the header and
record classes.
17-dec-2005 [yc] split to several modules; reimplemented
13-dec-2005 [yc] adapted to the changes of the `strutil` module.
13-sep-2002 [als] support FoxPro Timestamp datatype
15-nov-1999 [jjk] documentation updates, add demo
24-aug-1998 [jjk] add some encodeValue methods (not tested), other tweaks
08-jun-1998 [jjk] fix problems, add more features
20-feb-1998 [jjk] fix problems, add more features
19-feb-1998 [jjk] add create/write capabilities
18-feb-1998 [jjk] from dbfload.py
"""
__version__ = "$Revision: 1.7 $"[11:-2]
__date__ = "$Date: 2007/02/11 09:23:13 $"[7:-2]
__author__ = "Jeff Kunce <kuncej@mail.conservation.state.mo.us>"
__all__ = ["Dbf"]
from . import header
from . import record
from .utils import INVALID_VALUE
class Dbf(object):
"""DBF accessor.
FIXME:
docs and examples needed (dont' forget to tell
about problems adding new fields on the fly)
Implementation notes:
``_new`` field is used to indicate whether this is
a new data table. `addField` could be used only for
the new tables! If at least one record was appended
to the table it's structure couldn't be changed.
"""
__slots__ = ("name", "header", "stream",
"_changed", "_new", "_ignore_errors")
HeaderClass = header.DbfHeader
RecordClass = record.DbfRecord
INVALID_VALUE = INVALID_VALUE
# initialization and creation helpers
def __init__(self, f, readOnly=False, new=False, ignoreErrors=False):
"""Initialize instance.
Arguments:
f:
Filename or file-like object.
new:
True if new data table must be created. Assume
data table exists if this argument is False.
readOnly:
if ``f`` argument is a string file will
be opend in read-only mode; in other cases
this argument is ignored. This argument is ignored
even if ``new`` argument is True.
headerObj:
`header.DbfHeader` instance or None. If this argument
is None, new empty header will be used with the
all fields set by default.
ignoreErrors:
if set, failing field value conversion will return
``INVALID_VALUE`` instead of raising conversion error.
"""
if isinstance(f, str):
# a filename
self.name = f
if new:
# new table (table file must be
# created or opened and truncated)
self.stream = open(f, "w+b")
else:
# tabe file must exist
self.stream = open(f, ("r+b", "rb")[bool(readOnly)])
else:
# a stream
self.name = getattr(f, "name", "")
self.stream = f
if new:
# if this is a new table, header will be empty
self.header = self.HeaderClass()
else:
# or instantiated using stream
self.header = self.HeaderClass.fromStream(self.stream)
self.ignoreErrors = ignoreErrors
self._new = bool(new)
self._changed = False
# properties
closed = property(lambda self: self.stream.closed)
recordCount = property(lambda self: self.header.recordCount)
fieldNames = property(
lambda self: [_fld.name for _fld in self.header.fields])
fieldDefs = property(lambda self: self.header.fields)
changed = property(lambda self: self._changed or self.header.changed)
def ignoreErrors(self, value):
"""Update `ignoreErrors` flag on the header object and self"""
self.header.ignoreErrors = self._ignore_errors = bool(value)
ignoreErrors = property(
lambda self: self._ignore_errors,
ignoreErrors,
doc="""Error processing mode for DBF field value conversion
if set, failing field value conversion will return
``INVALID_VALUE`` instead of raising conversion error.
""")
# protected methods
def _fixIndex(self, index):
"""Return fixed index.
This method fails if index isn't a numeric object
(long or int). Or index isn't in a valid range
(less or equal to the number of records in the db).
If ``index`` is a negative number, it will be
treated as a negative indexes for list objects.
Return:
Return value is numeric object maning valid index.
"""
if not isinstance(index, int):
raise TypeError("Index must be a numeric object")
if index < 0:
# index from the right side
# fix it to the left-side index
index += len(self) + 1
if index >= len(self):
raise IndexError("Record index out of range")
return index
# iterface methods
def close(self):
self.flush()
self.stream.close()
def flush(self):
"""Flush data to the associated stream."""
if self.changed:
self.header.setCurrentDate()
self.header.write(self.stream)
self.stream.flush()
self._changed = False
def indexOfFieldName(self, name):
"""Index of field named ``name``."""
# FIXME: move this to header class
names = [f.name for f in self.header.fields]
return names.index(name.upper())
def newRecord(self):
"""Return new record, which belong to this table."""
return self.RecordClass(self)
def append(self, record):
"""Append ``record`` to the database."""
record.index = self.header.recordCount
record._write()
self.header.recordCount += 1
self._changed = True
self._new = False
def addField(self, *defs):
"""Add field definitions.
For more information see `header.DbfHeader.addField`.
"""
if self._new:
self.header.addField(*defs)
else:
raise TypeError("At least one record was added, "
"structure can't be changed")
# 'magic' methods (representation and sequence interface)
def __repr__(self):
return "Dbf stream '%s'\n" % self.stream + repr(self.header)
def __len__(self):
"""Return number of records."""
return self.recordCount
def __getitem__(self, index):
"""Return `DbfRecord` instance."""
return self.RecordClass.fromStream(self, self._fixIndex(index))
def __setitem__(self, index, record):
"""Write `DbfRecord` instance to the stream."""
record.index = self._fixIndex(index)
record._write()
self._changed = True
self._new = False
# def __del__(self):
# """Flush stream upon deletion of the object."""
# self.flush()
def demo_read(filename):
_dbf = Dbf(filename, True)
for _rec in _dbf:
print()
print(repr(_rec))
_dbf.close()
def demo_create(filename):
_dbf = Dbf(filename, new=True)
_dbf.addField(
("NAME", "C", 15),
("SURNAME", "C", 25),
("INITIALS", "C", 10),
("BIRTHDATE", "D"),
)
for (_n, _s, _i, _b) in (
("John", "Miller", "YC", (1981, 1, 2)),
("Andy", "Larkin", "AL", (1982, 3, 4)),
("Bill", "Clinth", "", (1983, 5, 6)),
("Bobb", "McNail", "", (1984, 7, 8)),
):
_rec = _dbf.newRecord()
_rec["NAME"] = _n
_rec["SURNAME"] = _s
_rec["INITIALS"] = _i
_rec["BIRTHDATE"] = _b
_rec.store()
print(repr(_dbf))
_dbf.close()
if __name__ == '__main__':
import sys
_name = len(sys.argv) > 1 and sys.argv[1] or "county.dbf"
demo_create(_name)
demo_read(_name)
# vim: set et sw=4 sts=4 :
-183
View File
@@ -1,183 +0,0 @@
#!/usr/bin/python
""".DBF creation helpers.
Note: this is a legacy interface. New code should use Dbf class
for table creation (see examples in dbf.py)
TODO:
- handle Memo fields.
- check length of the fields accoring to the
`http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
"""
"""History (most recent first)
04-jul-2006 [als] added export declaration;
updated for dbfpy 2.0
15-dec-2005 [yc] define dbf_new.__slots__
14-dec-2005 [yc] added vim modeline; retab'd; added doc-strings;
dbf_new now is a new class (inherited from object)
??-jun-2000 [--] added by Hans Fiby
"""
__version__ = "$Revision: 1.4 $"[11:-2]
__date__ = "$Date: 2006/07/04 08:18:18 $"[7:-2]
__all__ = ["dbf_new"]
from .dbf import *
from .fields import *
from .header import *
from .record import *
class _FieldDefinition(object):
"""Field definition.
This is a simple structure, which contains ``name``, ``type``,
``len``, ``dec`` and ``cls`` fields.
Objects also implement get/setitem magic functions, so fields
could be accessed via sequence iterface, where 'name' has
index 0, 'type' index 1, 'len' index 2, 'dec' index 3 and
'cls' could be located at index 4.
"""
__slots__ = "name", "type", "len", "dec", "cls"
# WARNING: be attentive - dictionaries are mutable!
FLD_TYPES = {
# type: (cls, len)
"C": (DbfCharacterFieldDef, None),
"N": (DbfNumericFieldDef, None),
"L": (DbfLogicalFieldDef, 1),
# FIXME: support memos
# "M": (DbfMemoFieldDef),
"D": (DbfDateFieldDef, 8),
# FIXME: I'm not sure length should be 14 characters!
# but temporary I use it, cuz date is 8 characters
# and time 6 (hhmmss)
"T": (DbfDateTimeFieldDef, 14),
}
def __init__(self, name, type, len=None, dec=0):
_cls, _len = self.FLD_TYPES[type]
if _len is None:
if len is None:
raise ValueError("Field length must be defined")
_len = len
self.name = name
self.type = type
self.len = _len
self.dec = dec
self.cls = _cls
def getDbfField(self):
"Return `DbfFieldDef` instance from the current definition."
return self.cls(self.name, self.len, self.dec)
def appendToHeader(self, dbfh):
"""Create a `DbfFieldDef` instance and append it to the dbf header.
Arguments:
dbfh: `DbfHeader` instance.
"""
_dbff = self.getDbfField()
dbfh.addField(_dbff)
class dbf_new(object):
"""New .DBF creation helper.
Example Usage:
dbfn = dbf_new()
dbfn.add_field("name",'C',80)
dbfn.add_field("price",'N',10,2)
dbfn.add_field("date",'D',8)
dbfn.write("tst.dbf")
Note:
This module cannot handle Memo-fields,
they are special.
"""
__slots__ = ("fields",)
FieldDefinitionClass = _FieldDefinition
def __init__(self):
self.fields = []
def add_field(self, name, typ, len, dec=0):
"""Add field definition.
Arguments:
name:
field name (str object). field name must not
contain ASCII NULs and it's length shouldn't
exceed 10 characters.
typ:
type of the field. this must be a single character
from the "CNLMDT" set meaning character, numeric,
logical, memo, date and date/time respectively.
len:
length of the field. this argument is used only for
the character and numeric fields. all other fields
have fixed length.
FIXME: use None as a default for this argument?
dec:
decimal precision. used only for the numric fields.
"""
self.fields.append(self.FieldDefinitionClass(name, typ, len, dec))
def write(self, filename):
"""Create empty .DBF file using current structure."""
_dbfh = DbfHeader()
_dbfh.setCurrentDate()
for _fldDef in self.fields:
_fldDef.appendToHeader(_dbfh)
_dbfStream = open(filename, "wb")
_dbfh.write(_dbfStream)
_dbfStream.close()
if __name__ == '__main__':
# create a new DBF-File
dbfn = dbf_new()
dbfn.add_field("name", 'C', 80)
dbfn.add_field("price", 'N', 10, 2)
dbfn.add_field("date", 'D', 8)
dbfn.write("tst.dbf")
# test new dbf
print("*** created tst.dbf: ***")
dbft = Dbf('tst.dbf', readOnly=0)
print(repr(dbft))
# add a record
rec = DbfRecord(dbft)
rec['name'] = 'something'
rec['price'] = 10.5
rec['date'] = (2000, 1, 12)
rec.store()
# add another record
rec = DbfRecord(dbft)
rec['name'] = 'foo and bar'
rec['price'] = 12234
rec['date'] = (1992, 7, 15)
rec.store()
# show the records
print("*** inserted 2 records into tst.dbf: ***")
print(repr(dbft))
for i1 in range(len(dbft)):
rec = dbft[i1]
for fldName in dbft.fieldNames:
print('%s:\t %s' % (fldName, rec[fldName]))
print()
dbft.close()
# vim: set et sts=4 sw=4 :
-466
View File
@@ -1,466 +0,0 @@
"""DBF fields definitions.
TODO:
- make memos work
"""
"""History (most recent first):
26-may-2009 [als] DbfNumericFieldDef.decodeValue: strip zero bytes
05-feb-2009 [als] DbfDateFieldDef.encodeValue: empty arg produces empty date
16-sep-2008 [als] DbfNumericFieldDef decoding looks for decimal point
in the value to select float or integer return type
13-mar-2008 [als] check field name length in constructor
11-feb-2007 [als] handle value conversion errors
10-feb-2007 [als] DbfFieldDef: added .rawFromRecord()
01-dec-2006 [als] Timestamp columns use None for empty values
31-oct-2006 [als] support field types 'F' (float), 'I' (integer)
and 'Y' (currency);
automate export and registration of field classes
04-jul-2006 [als] added export declaration
10-mar-2006 [als] decode empty values for Date and Logical fields;
show field name in errors
10-mar-2006 [als] fix Numeric value decoding: according to spec,
value always is string representation of the number;
ensure that encoded Numeric value fits into the field
20-dec-2005 [yc] use field names in upper case
15-dec-2005 [yc] field definitions moved from `dbf`.
"""
__version__ = "$Revision: 1.14 $"[11:-2]
__date__ = "$Date: 2009/05/26 05:16:51 $"[7:-2]
__all__ = ["lookupFor",] # field classes added at the end of the module
import datetime
import struct
import sys
from . import utils
## abstract definitions
class DbfFieldDef(object):
"""Abstract field definition.
Child classes must override ``type`` class attribute to provide datatype
infromation of the field definition. For more info about types visit
`http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
Also child classes must override ``defaultValue`` field to provide
default value for the field value.
If child class has fixed length ``length`` class attribute must be
overriden and set to the valid value. None value means, that field
isn't of fixed length.
Note: ``name`` field must not be changed after instantiation.
"""
__slots__ = ("name", "decimalCount",
"start", "end", "ignoreErrors")
# length of the field, None in case of variable-length field,
# or a number if this field is a fixed-length field
length = None
# field type. for more information about fields types visit
# `http://www.clicketyclick.dk/databases/xbase/format/data_types.html`
# must be overriden in child classes
typeCode = None
# default value for the field. this field must be
# overriden in child classes
defaultValue = None
def __init__(self, name, length=None, decimalCount=None,
start=None, stop=None, ignoreErrors=False,
):
"""Initialize instance."""
assert self.typeCode is not None, "Type code must be overriden"
assert self.defaultValue is not None, "Default value must be overriden"
## fix arguments
if len(name) >10:
raise ValueError("Field name \"%s\" is too long" % name)
name = str(name).upper()
if self.__class__.length is None:
if length is None:
raise ValueError("[%s] Length isn't specified" % name)
length = int(length)
if length <= 0:
raise ValueError("[%s] Length must be a positive integer"
% name)
else:
length = self.length
if decimalCount is None:
decimalCount = 0
## set fields
self.name = name
# FIXME: validate length according to the specification at
# http://www.clicketyclick.dk/databases/xbase/format/data_types.html
self.length = length
self.decimalCount = decimalCount
self.ignoreErrors = ignoreErrors
self.start = start
self.end = stop
def __cmp__(self, other):
return cmp(self.name, str(other).upper())
def __hash__(self):
return hash(self.name)
def fromString(cls, string, start, ignoreErrors=False):
"""Decode dbf field definition from the string data.
Arguments:
string:
a string, dbf definition is decoded from. length of
the string must be 32 bytes.
start:
position in the database file.
ignoreErrors:
initial error processing mode for the new field (boolean)
"""
assert len(string) == 32
_length = string[16]
return cls(utils.unzfill(string)[:11].decode('utf-8'), _length,
string[17], start, start + _length, ignoreErrors=ignoreErrors)
fromString = classmethod(fromString)
def toString(self):
"""Return encoded field definition.
Return:
Return value is a string object containing encoded
definition of this field.
"""
if sys.version_info < (2, 4):
# earlier versions did not support padding character
_name = self.name[:11] + "\0" * (11 - len(self.name))
else:
_name = self.name.ljust(11, '\0')
return (
_name +
self.typeCode +
#data address
chr(0) * 4 +
chr(self.length) +
chr(self.decimalCount) +
chr(0) * 14
)
def __repr__(self):
return "%-10s %1s %3d %3d" % self.fieldInfo()
def fieldInfo(self):
"""Return field information.
Return:
Return value is a (name, type, length, decimals) tuple.
"""
return (self.name, self.typeCode, self.length, self.decimalCount)
def rawFromRecord(self, record):
"""Return a "raw" field value from the record string."""
return record[self.start:self.end]
def decodeFromRecord(self, record):
"""Return decoded field value from the record string."""
try:
return self.decodeValue(self.rawFromRecord(record))
except:
if self.ignoreErrors:
return utils.INVALID_VALUE
else:
raise
def decodeValue(self, value):
"""Return decoded value from string value.
This method shouldn't be used publicly. It's called from the
`decodeFromRecord` method.
This is an abstract method and it must be overridden in child classes.
"""
raise NotImplementedError
def encodeValue(self, value):
"""Return str object containing encoded field value.
This is an abstract method and it must be overriden in child classes.
"""
raise NotImplementedError
## real classes
class DbfCharacterFieldDef(DbfFieldDef):
"""Definition of the character field."""
typeCode = "C"
defaultValue = b''
def decodeValue(self, value):
"""Return string object.
Return value is a ``value`` argument with stripped right spaces.
"""
return value.rstrip(b' ').decode('utf-8')
def encodeValue(self, value):
"""Return raw data string encoded from a ``value``."""
return str(value)[:self.length].ljust(self.length)
class DbfNumericFieldDef(DbfFieldDef):
"""Definition of the numeric field."""
typeCode = "N"
# XXX: now I'm not sure it was a good idea to make a class field
# `defaultValue` instead of a generic method as it was implemented
# previously -- it's ok with all types except number, cuz
# if self.decimalCount is 0, we should return 0 and 0.0 otherwise.
defaultValue = 0
def decodeValue(self, value):
"""Return a number decoded from ``value``.
If decimals is zero, value will be decoded as an integer;
or as a float otherwise.
Return:
Return value is a int (long) or float instance.
"""
value = value.strip(b' \0')
if b'.' in value:
# a float (has decimal separator)
return float(value)
elif value:
# must be an integer
return int(value)
else:
return 0
def encodeValue(self, value):
"""Return string containing encoded ``value``."""
_rv = ("%*.*f" % (self.length, self.decimalCount, value))
if len(_rv) > self.length:
_ppos = _rv.find(".")
if 0 <= _ppos <= self.length:
_rv = _rv[:self.length]
else:
raise ValueError("[%s] Numeric overflow: %s (field width: %i)"
% (self.name, _rv, self.length))
return _rv
class DbfFloatFieldDef(DbfNumericFieldDef):
"""Definition of the float field - same as numeric."""
typeCode = "F"
class DbfIntegerFieldDef(DbfFieldDef):
"""Definition of the integer field."""
typeCode = "I"
length = 4
defaultValue = 0
def decodeValue(self, value):
"""Return an integer number decoded from ``value``."""
return struct.unpack("<i", value)[0]
def encodeValue(self, value):
"""Return string containing encoded ``value``."""
return struct.pack("<i", int(value))
class DbfCurrencyFieldDef(DbfFieldDef):
"""Definition of the currency field."""
typeCode = "Y"
length = 8
defaultValue = 0.0
def decodeValue(self, value):
"""Return float number decoded from ``value``."""
return struct.unpack("<q", value)[0] / 10000.
def encodeValue(self, value):
"""Return string containing encoded ``value``."""
return struct.pack("<q", round(value * 10000))
class DbfLogicalFieldDef(DbfFieldDef):
"""Definition of the logical field."""
typeCode = "L"
defaultValue = -1
length = 1
def decodeValue(self, value):
"""Return True, False or -1 decoded from ``value``."""
# Note: value always is 1-char string
if value == "?":
return -1
if value in "NnFf ":
return False
if value in "YyTt":
return True
raise ValueError("[%s] Invalid logical value %r" % (self.name, value))
def encodeValue(self, value):
"""Return a character from the "TF?" set.
Return:
Return value is "T" if ``value`` is True
"?" if value is -1 or False otherwise.
"""
if value is True:
return "T"
if value == -1:
return "?"
return "F"
class DbfMemoFieldDef(DbfFieldDef):
"""Definition of the memo field.
Note: memos aren't currenly completely supported.
"""
typeCode = "M"
defaultValue = " " * 10
length = 10
def decodeValue(self, value):
"""Return int .dbt block number decoded from the string object."""
#return int(value)
raise NotImplementedError
def encodeValue(self, value):
"""Return raw data string encoded from a ``value``.
Note: this is an internal method.
"""
#return str(value)[:self.length].ljust(self.length)
raise NotImplementedError
class DbfDateFieldDef(DbfFieldDef):
"""Definition of the date field."""
typeCode = "D"
defaultValue = utils.classproperty(lambda cls: datetime.date.today())
# "yyyymmdd" gives us 8 characters
length = 8
def decodeValue(self, value):
"""Return a ``datetime.date`` instance decoded from ``value``."""
if value.strip():
return utils.getDate(value)
else:
return None
def encodeValue(self, value):
"""Return a string-encoded value.
``value`` argument should be a value suitable for the
`utils.getDate` call.
Return:
Return value is a string in format "yyyymmdd".
"""
if value:
return utils.getDate(value).strftime("%Y%m%d")
else:
return " " * self.length
class DbfDateTimeFieldDef(DbfFieldDef):
"""Definition of the timestamp field."""
# a difference between JDN (Julian Day Number)
# and GDN (Gregorian Day Number). note, that GDN < JDN
JDN_GDN_DIFF = 1721425
typeCode = "T"
defaultValue = utils.classproperty(lambda cls: datetime.datetime.now())
# two 32-bits integers representing JDN and amount of
# milliseconds respectively gives us 8 bytes.
# note, that values must be encoded in LE byteorder.
length = 8
def decodeValue(self, value):
"""Return a `datetime.datetime` instance."""
assert len(value) == self.length
# LE byteorder
_jdn, _msecs = struct.unpack("<2I", value)
if _jdn >= 1:
_rv = datetime.datetime.fromordinal(_jdn - self.JDN_GDN_DIFF)
_rv += datetime.timedelta(0, _msecs / 1000.0)
else:
# empty date
_rv = None
return _rv
def encodeValue(self, value):
"""Return a string-encoded ``value``."""
if value:
value = utils.getDateTime(value)
# LE byteorder
_rv = struct.pack("<2I", value.toordinal() + self.JDN_GDN_DIFF,
(value.hour * 3600 + value.minute * 60 + value.second) * 1000)
else:
_rv = "\0" * self.length
assert len(_rv) == self.length
return _rv
_fieldsRegistry = {}
def registerField(fieldCls):
"""Register field definition class.
``fieldCls`` should be subclass of the `DbfFieldDef`.
Use `lookupFor` to retrieve field definition class
by the type code.
"""
assert fieldCls.typeCode is not None, "Type code isn't defined"
# XXX: use fieldCls.typeCode.upper()? in case of any decign
# don't forget to look to the same comment in ``lookupFor`` method
_fieldsRegistry[fieldCls.typeCode] = fieldCls
def lookupFor(typeCode):
"""Return field definition class for the given type code.
``typeCode`` must be a single character. That type should be
previously registered.
Use `registerField` to register new field class.
Return:
Return value is a subclass of the `DbfFieldDef`.
"""
# XXX: use typeCode.upper()? in case of any decign don't
# forget to look to the same comment in ``registerField``
return _fieldsRegistry[chr(typeCode)]
## register generic types
for (_name, _val) in list(globals().items()):
if isinstance(_val, type) and issubclass(_val, DbfFieldDef) \
and (_name != "DbfFieldDef"):
__all__.append(_name)
registerField(_val)
del _name, _val
# vim: et sts=4 sw=4 :
-273
View File
@@ -1,273 +0,0 @@
"""DBF header definition.
TODO:
- handle encoding of the character fields
(encoding information stored in the DBF header)
"""
"""History (most recent first):
16-sep-2010 [als] fromStream: fix century of the last update field
11-feb-2007 [als] added .ignoreErrors
10-feb-2007 [als] added __getitem__: return field definitions
by field name or field number (zero-based)
04-jul-2006 [als] added export declaration
15-dec-2005 [yc] created
"""
__version__ = "$Revision: 1.6 $"[11:-2]
__date__ = "$Date: 2010/09/16 05:06:39 $"[7:-2]
__all__ = ["DbfHeader"]
import io
import datetime
import struct
import time
import sys
from . import fields
from .utils import getDate
class DbfHeader(object):
"""Dbf header definition.
For more information about dbf header format visit
`http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_STRUCT`
Examples:
Create an empty dbf header and add some field definitions:
dbfh = DbfHeader()
dbfh.addField(("name", "C", 10))
dbfh.addField(("date", "D"))
dbfh.addField(DbfNumericFieldDef("price", 5, 2))
Create a dbf header with field definitions:
dbfh = DbfHeader([
("name", "C", 10),
("date", "D"),
DbfNumericFieldDef("price", 5, 2),
])
"""
__slots__ = ("signature", "fields", "lastUpdate", "recordLength",
"recordCount", "headerLength", "changed", "_ignore_errors")
## instance construction and initialization methods
def __init__(self, fields=None, headerLength=0, recordLength=0,
recordCount=0, signature=0x03, lastUpdate=None, ignoreErrors=False,
):
"""Initialize instance.
Arguments:
fields:
a list of field definitions;
recordLength:
size of the records;
headerLength:
size of the header;
recordCount:
number of records stored in DBF;
signature:
version number (aka signature). using 0x03 as a default meaning
"File without DBT". for more information about this field visit
``http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_NOTE_1_TARGET``
lastUpdate:
date of the DBF's update. this could be a string ('yymmdd' or
'yyyymmdd'), timestamp (int or float), datetime/date value,
a sequence (assuming (yyyy, mm, dd, ...)) or an object having
callable ``ticks`` field.
ignoreErrors:
error processing mode for DBF fields (boolean)
"""
self.signature = signature
if fields is None:
self.fields = []
else:
self.fields = list(fields)
self.lastUpdate = getDate(lastUpdate)
self.recordLength = recordLength
self.headerLength = headerLength
self.recordCount = recordCount
self.ignoreErrors = ignoreErrors
# XXX: I'm not sure this is safe to
# initialize `self.changed` in this way
self.changed = bool(self.fields)
# @classmethod
def fromString(cls, string):
"""Return header instance from the string object."""
return cls.fromStream(io.StringIO(str(string)))
fromString = classmethod(fromString)
# @classmethod
def fromStream(cls, stream):
"""Return header object from the stream."""
stream.seek(0)
first_32 = stream.read(32)
if type(first_32) != bytes:
_data = bytes(first_32, sys.getfilesystemencoding())
_data = first_32
(_cnt, _hdrLen, _recLen) = struct.unpack("<I2H", _data[4:12])
#reserved = _data[12:32]
_year = _data[1]
if _year < 80:
# dBase II started at 1980. It is quite unlikely
# that actual last update date is before that year.
_year += 2000
else:
_year += 1900
## create header object
_obj = cls(None, _hdrLen, _recLen, _cnt, _data[0],
(_year, _data[2], _data[3]))
## append field definitions
# position 0 is for the deletion flag
_pos = 1
_data = stream.read(1)
while _data != b'\r':
_data += stream.read(31)
_fld = fields.lookupFor(_data[11]).fromString(_data, _pos)
_obj._addField(_fld)
_pos = _fld.end
_data = stream.read(1)
return _obj
fromStream = classmethod(fromStream)
## properties
year = property(lambda self: self.lastUpdate.year)
month = property(lambda self: self.lastUpdate.month)
day = property(lambda self: self.lastUpdate.day)
def ignoreErrors(self, value):
"""Update `ignoreErrors` flag on self and all fields"""
self._ignore_errors = value = bool(value)
for _field in self.fields:
_field.ignoreErrors = value
ignoreErrors = property(
lambda self: self._ignore_errors,
ignoreErrors,
doc="""Error processing mode for DBF field value conversion
if set, failing field value conversion will return
``INVALID_VALUE`` instead of raising conversion error.
""")
## object representation
def __repr__(self):
_rv = """\
Version (signature): 0x%02x
Last update: %s
Header length: %d
Record length: %d
Record count: %d
FieldName Type Len Dec
""" % (self.signature, self.lastUpdate, self.headerLength,
self.recordLength, self.recordCount)
_rv += "\n".join(
["%10s %4s %3s %3s" % _fld.fieldInfo() for _fld in self.fields]
)
return _rv
## internal methods
def _addField(self, *defs):
"""Internal variant of the `addField` method.
This method doesn't set `self.changed` field to True.
Return value is a length of the appended records.
Note: this method doesn't modify ``recordLength`` and
``headerLength`` fields. Use `addField` instead of this
method if you don't exactly know what you're doing.
"""
# insure we have dbf.DbfFieldDef instances first (instantiation
# from the tuple could raise an error, in such a case I don't
# wanna add any of the definitions -- all will be ignored)
_defs = []
_recordLength = 0
for _def in defs:
if isinstance(_def, fields.DbfFieldDef):
_obj = _def
else:
(_name, _type, _len, _dec) = (tuple(_def) + (None,) * 4)[:4]
_cls = fields.lookupFor(_type)
_obj = _cls(_name, _len, _dec,
ignoreErrors=self._ignore_errors)
_recordLength += _obj.length
_defs.append(_obj)
# and now extend field definitions and
# update record length
self.fields += _defs
return _recordLength
## interface methods
def addField(self, *defs):
"""Add field definition to the header.
Examples:
dbfh.addField(
("name", "C", 20),
dbf.DbfCharacterFieldDef("surname", 20),
dbf.DbfDateFieldDef("birthdate"),
("member", "L"),
)
dbfh.addField(("price", "N", 5, 2))
dbfh.addField(dbf.DbfNumericFieldDef("origprice", 5, 2))
"""
_oldLen = self.recordLength
self.recordLength += self._addField(*defs)
if not _oldLen:
self.recordLength += 1
# XXX: may be just use:
# self.recordeLength += self._addField(*defs) + bool(not _oldLen)
# recalculate headerLength
self.headerLength = 32 + (32 * len(self.fields)) + 1
self.changed = True
def write(self, stream):
"""Encode and write header to the stream."""
stream.seek(0)
stream.write(self.toString())
fields = [_fld.toString() for _fld in self.fields]
stream.write(''.join(fields).encode(sys.getfilesystemencoding()))
stream.write(b'\x0D') # cr at end of all header data
self.changed = False
def toString(self):
"""Returned 32 chars length string with encoded header."""
return struct.pack("<4BI2H",
self.signature,
self.year - 1900,
self.month,
self.day,
self.recordCount,
self.headerLength,
self.recordLength) + (b'\x00' * 20)
#TODO: figure out if bytes(utf-8) is correct here.
def setCurrentDate(self):
"""Update ``self.lastUpdate`` field with current date value."""
self.lastUpdate = datetime.date.today()
def __getitem__(self, item):
"""Return a field definition by numeric index or name string"""
if isinstance(item, str):
_name = item.upper()
for _field in self.fields:
if _field.name == _name:
return _field
else:
raise KeyError(item)
else:
# item must be field index
return self.fields[item]
# vim: et sts=4 sw=4 :
-266
View File
@@ -1,266 +0,0 @@
"""DBF record definition.
"""
"""History (most recent first):
11-feb-2007 [als] __repr__: added special case for invalid field values
10-feb-2007 [als] added .rawFromStream()
30-oct-2006 [als] fix record length in .fromStream()
04-jul-2006 [als] added export declaration
20-dec-2005 [yc] DbfRecord.write() -> DbfRecord._write();
added delete() method.
16-dec-2005 [yc] record definition moved from `dbf`.
"""
__version__ = "$Revision: 1.7 $"[11:-2]
__date__ = "$Date: 2007/02/11 09:05:49 $"[7:-2]
__all__ = ["DbfRecord"]
import sys
from . import utils
class DbfRecord(object):
"""DBF record.
Instances of this class shouldn't be created manualy,
use `dbf.Dbf.newRecord` instead.
Class implements mapping/sequence interface, so
fields could be accessed via their names or indexes
(names is a preffered way to access fields).
Hint:
Use `store` method to save modified record.
Examples:
Add new record to the database:
db = Dbf(filename)
rec = db.newRecord()
rec["FIELD1"] = value1
rec["FIELD2"] = value2
rec.store()
Or the same, but modify existed
(second in this case) record:
db = Dbf(filename)
rec = db[2]
rec["FIELD1"] = value1
rec["FIELD2"] = value2
rec.store()
"""
__slots__ = "dbf", "index", "deleted", "fieldData"
## creation and initialization
def __init__(self, dbf, index=None, deleted=False, data=None):
"""Instance initialiation.
Arguments:
dbf:
A `Dbf.Dbf` instance this record belonogs to.
index:
An integer record index or None. If this value is
None, record will be appended to the DBF.
deleted:
Boolean flag indicating whether this record
is a deleted record.
data:
A sequence or None. This is a data of the fields.
If this argument is None, default values will be used.
"""
self.dbf = dbf
# XXX: I'm not sure ``index`` is necessary
self.index = index
self.deleted = deleted
if data is None:
self.fieldData = [_fd.defaultValue for _fd in dbf.header.fields]
else:
self.fieldData = list(data)
# XXX: validate self.index before calculating position?
position = property(lambda self: self.dbf.header.headerLength + \
self.index * self.dbf.header.recordLength)
def rawFromStream(cls, dbf, index):
"""Return raw record contents read from the stream.
Arguments:
dbf:
A `Dbf.Dbf` instance containing the record.
index:
Index of the record in the records' container.
This argument can't be None in this call.
Return value is a string containing record data in DBF format.
"""
# XXX: may be write smth assuming, that current stream
# position is the required one? it could save some
# time required to calculate where to seek in the file
dbf.stream.seek(dbf.header.headerLength +
index * dbf.header.recordLength)
return dbf.stream.read(dbf.header.recordLength)
rawFromStream = classmethod(rawFromStream)
def fromStream(cls, dbf, index):
"""Return a record read from the stream.
Arguments:
dbf:
A `Dbf.Dbf` instance new record should belong to.
index:
Index of the record in the records' container.
This argument can't be None in this call.
Return value is an instance of the current class.
"""
return cls.fromString(dbf, cls.rawFromStream(dbf, index), index)
fromStream = classmethod(fromStream)
def fromString(cls, dbf, string, index=None):
"""Return record read from the string object.
Arguments:
dbf:
A `Dbf.Dbf` instance new record should belong to.
string:
A string new record should be created from.
index:
Index of the record in the container. If this
argument is None, record will be appended.
Return value is an instance of the current class.
"""
return cls(dbf, index, string[0]=="*",
[_fd.decodeFromRecord(string) for _fd in dbf.header.fields])
fromString = classmethod(fromString)
## object representation
def __repr__(self):
_template = "%%%ds: %%s (%%s)" % max([len(_fld)
for _fld in self.dbf.fieldNames])
_rv = []
for _fld in self.dbf.fieldNames:
_val = self[_fld]
if _val is utils.INVALID_VALUE:
_rv.append(_template %
(_fld, "None", "value cannot be decoded"))
else:
_rv.append(_template % (_fld, _val, type(_val)))
return "\n".join(_rv)
## protected methods
def _write(self):
"""Write data to the dbf stream.
Note:
This isn't a public method, it's better to
use 'store' instead publically.
Be design ``_write`` method should be called
only from the `Dbf` instance.
"""
self._validateIndex(False)
self.dbf.stream.seek(self.position)
self.dbf.stream.write(bytes(self.toString(),
sys.getfilesystemencoding()))
# FIXME: may be move this write somewhere else?
# why we should check this condition for each record?
if self.index == len(self.dbf):
# this is the last record,
# we should write SUB (ASCII 26)
self.dbf.stream.write(b"\x1A")
## utility methods
def _validateIndex(self, allowUndefined=True, checkRange=False):
"""Valid ``self.index`` value.
If ``allowUndefined`` argument is True functions does nothing
in case of ``self.index`` pointing to None object.
"""
if self.index is None:
if not allowUndefined:
raise ValueError("Index is undefined")
elif self.index < 0:
raise ValueError("Index can't be negative (%s)" % self.index)
elif checkRange and self.index <= self.dbf.header.recordCount:
raise ValueError("There are only %d records in the DBF" %
self.dbf.header.recordCount)
## interface methods
def store(self):
"""Store current record in the DBF.
If ``self.index`` is None, this record will be appended to the
records of the DBF this records belongs to; or replaced otherwise.
"""
self._validateIndex()
if self.index is None:
self.index = len(self.dbf)
self.dbf.append(self)
else:
self.dbf[self.index] = self
def delete(self):
"""Mark method as deleted."""
self.deleted = True
def toString(self):
"""Return string packed record values."""
# for (_def, _dat) in zip(self.dbf.header.fields, self.fieldData):
#
return "".join([" *"[self.deleted]] + [
_def.encodeValue(_dat)
for (_def, _dat) in zip(self.dbf.header.fields, self.fieldData)
])
def asList(self):
"""Return a flat list of fields.
Note:
Change of the list's values won't change
real values stored in this object.
"""
return self.fieldData[:]
def asDict(self):
"""Return a dictionary of fields.
Note:
Change of the dicts's values won't change
real values stored in this object.
"""
return dict([_i for _i in zip(self.dbf.fieldNames, self.fieldData)])
def __getitem__(self, key):
"""Return value by field name or field index."""
if isinstance(key, int):
# integer index of the field
return self.fieldData[key]
# assuming string field name
return self.fieldData[self.dbf.indexOfFieldName(key)]
def __setitem__(self, key, value):
"""Set field value by integer index of the field or string name."""
if isinstance(key, int):
# integer index of the field
return self.fieldData[key]
# assuming string field name
self.fieldData[self.dbf.indexOfFieldName(key)] = value
# vim: et sts=4 sw=4 :
-170
View File
@@ -1,170 +0,0 @@
"""String utilities.
TODO:
- allow strings in getDateTime routine;
"""
"""History (most recent first):
11-feb-2007 [als] added INVALID_VALUE
10-feb-2007 [als] allow date strings padded with spaces instead of zeroes
20-dec-2005 [yc] handle long objects in getDate/getDateTime
16-dec-2005 [yc] created from ``strutil`` module.
"""
__version__ = "$Revision: 1.4 $"[11:-2]
__date__ = "$Date: 2007/02/11 08:57:17 $"[7:-2]
import datetime
import time
def unzfill(str):
"""Return a string without ASCII NULs.
This function searchers for the first NUL (ASCII 0) occurance
and truncates string till that position.
"""
try:
return str[:str.index(b'\0')]
except ValueError:
return str
def getDate(date=None):
"""Return `datetime.date` instance.
Type of the ``date`` argument could be one of the following:
None:
use current date value;
datetime.date:
this value will be returned;
datetime.datetime:
the result of the date.date() will be returned;
string:
assuming "%Y%m%d" or "%y%m%dd" format;
number:
assuming it's a timestamp (returned for example
by the time.time() call;
sequence:
assuming (year, month, day, ...) sequence;
Additionaly, if ``date`` has callable ``ticks`` attribute,
it will be used and result of the called would be treated
as a timestamp value.
"""
if date is None:
# use current value
return datetime.date.today()
if isinstance(date, datetime.date):
return date
if isinstance(date, datetime.datetime):
return date.date()
if isinstance(date, (int, float)):
# date is a timestamp
return datetime.date.fromtimestamp(date)
if isinstance(date, str):
date = date.replace(" ", "0")
if len(date) == 6:
# yymmdd
return datetime.date(*time.strptime(date, "%y%m%d")[:3])
# yyyymmdd
return datetime.date(*time.strptime(date, "%Y%m%d")[:3])
if hasattr(date, "__getitem__"):
# a sequence (assuming date/time tuple)
return datetime.date(*date[:3])
return datetime.date.fromtimestamp(date.ticks())
def getDateTime(value=None):
"""Return `datetime.datetime` instance.
Type of the ``value`` argument could be one of the following:
None:
use current date value;
datetime.date:
result will be converted to the `datetime.datetime` instance
using midnight;
datetime.datetime:
``value`` will be returned as is;
string:
*** CURRENTLY NOT SUPPORTED ***;
number:
assuming it's a timestamp (returned for example
by the time.time() call;
sequence:
assuming (year, month, day, ...) sequence;
Additionaly, if ``value`` has callable ``ticks`` attribute,
it will be used and result of the called would be treated
as a timestamp value.
"""
if value is None:
# use current value
return datetime.datetime.today()
if isinstance(value, datetime.datetime):
return value
if isinstance(value, datetime.date):
return datetime.datetime.fromordinal(value.toordinal())
if isinstance(value, (int, float)):
# value is a timestamp
return datetime.datetime.fromtimestamp(value)
if isinstance(value, str):
raise NotImplementedError("Strings aren't currently implemented")
if hasattr(value, "__getitem__"):
# a sequence (assuming date/time tuple)
return datetime.datetime(*tuple(value)[:6])
return datetime.datetime.fromtimestamp(value.ticks())
class classproperty(property):
"""Works in the same way as a ``property``, but for the classes."""
def __get__(self, obj, cls):
return self.fget(cls)
class _InvalidValue(object):
"""Value returned from DBF records when field validation fails
The value is not equal to anything except for itself
and equal to all empty values: None, 0, empty string etc.
In other words, invalid value is equal to None and not equal
to None at the same time.
This value yields zero upon explicit conversion to a number type,
empty string for string types, and False for boolean.
"""
def __eq__(self, other):
return not other
def __ne__(self, other):
return not (other is self)
def __bool__(self):
return False
def __int__(self):
return 0
__long__ = __int__
def __float__(self):
return 0.0
def __str__(self):
return ""
def __unicode__(self):
return ""
def __repr__(self):
return "<INVALID>"
# invalid value is a constant singleton
INVALID_VALUE = _InvalidValue()
# vim: set et sts=4 sw=4 :
-24
View File
@@ -1,24 +0,0 @@
from __future__ import division
def median(data):
"""
Return the median (middle value) of numeric data, using the common
"mean of middle two" method. If data is empty, ValueError is raised.
Mimics the behaviour of Python3's statistics.median
>>> median([1, 3, 5])
3
>>> median([1, 3, 5, 7])
4.0
"""
data = sorted(data)
n = len(data)
if not n:
raise ValueError("No median for empty data")
i = n // 2
if n % 2:
return data[i]
return (data[i - 1] + data[i]) / 2
+1
View File
@@ -0,0 +1 @@
from core import *
+313
View File
@@ -0,0 +1,313 @@
# -*- coding: utf-8 -*-
# _____ ______ ______ _________
# __ /_______ ____ /_ ___ /_ _____ ______ /
# _ __/_ __ `/__ __ \__ __ \_ _ \_ __ /
# / /_ / /_/ / _ /_/ /_ /_/ // __// /_/ /
# \__/ \__,_/ /_.___/ /_.___/ \___/ \__,_/
import csv
import cStringIO
import random
import simplejson as json
import xlwt
import yaml
from helpers import *
# __all__ = ['Dataset', 'DataBook']
__name__ = 'tablib'
__version__ = '0.6.4'
__build__ = 0x000604
__author__ = 'Kenneth Reitz'
__license__ = 'MIT'
__copyright__ = 'Copyright 2010 Kenneth Reitz'
class Dataset(object):
"""Epic Tabular-Dataset object. """
def __init__(self, *args, **kwargs):
self._data = None
self._saved_file = None
self._saved_format = None
self._data = list(args)
self.__headers = None
try:
self.headers = kwargs['headers']
except KeyError, why:
self.headers = None
try:
self.title = kwargs['title']
except KeyError, why:
self.title = None
def __len__(self):
return self.height
def __getitem__(self, key):
if isinstance(key, basestring):
if key in self.headers:
pos = self.headers.index(key) # get 'key' index from each data
return [row[pos] for row in self._data]
else:
raise KeyError
else:
return self._data[key]
def __setitem__(self, key, value):
self._validate(value)
self._data[key] = tuple(value)
def __delitem__(self, key):
del self._data[key]
def __repr__(self):
try:
return '<%s dataset>' % (self.title.lower())
except AttributeError:
return '<dataset object>'
def _validate(self, row=None, col=None, safety=False):
"""Assures size of every row in dataset is of proper proportions."""
if row:
is_valid = (len(row) == self.width) if self.width else True
elif col:
if self.headers:
is_valid = (len(col) - 1) == self.height
else:
is_valid = (len(col) == self.height) if self.height else True
else:
is_valid = all((len(x)== self.width for x in self._data))
if is_valid:
return True
else:
if not safety:
raise InvalidDimensions
return False
def _package(self, dicts=True):
"""Packages Dataset into lists of dictionaries for transmission."""
if self.headers:
if dicts:
data = [dict(zip(self.headers, data_row)) for data_row in self ._data]
else:
data = [list(self.headers)] + list(self._data)
else:
data = [list(row) for row in self._data]
return data
@property
def height(self):
"""Returns the height of the Dataset."""
return len(self._data)
@property
def width(self):
"""Returns the width of the Dataset."""
try:
return len(self._data[0])
except IndexError, why:
try:
return len(self.headers)
except TypeError, e:
return 0
@property
def headers(self):
"""Headers property."""
return self.__headers
@headers.setter
def headers(self, collection):
"""Validating headers setter."""
self._validate(collection)
if collection:
try:
self.__headers = list(collection)
except TypeError, why:
raise TypeError
else:
self.__headers = None
@property
def dict(self):
"""Returns python dict of Dataset."""
return self._package()
@property
def json(self):
"""Returns JSON representation of Dataset."""
return json.dumps(self.dict)
@property
def yaml(self):
"""Returns YAML representation of Dataset."""
return yaml.dump(self.dict)
@property
def csv(self):
"""Returns CSV representation of Dataset."""
stream = cStringIO.StringIO()
_csv = csv.writer(stream)
for row in self._package(dicts=False):
_csv.writerow(row)
return stream.getvalue()
@property
def xls(self):
"""Returns XLS representation of Dataset."""
stream = cStringIO.StringIO()
wb = xlwt.Workbook()
ws = wb.add_sheet(self.title if self.title else 'Tabbed Dataset')
for i, row in enumerate(self._package(dicts=False)):
for j, col in enumerate(row):
ws.write(i, j, col.decode('utf8'))
wb.save(stream)
return stream.getvalue()
def append(self, row=None, col=None):
"""Adds a row to the end of Dataset"""
if row:
self._validate(row)
self._data.append(tuple(row))
elif col:
self._validate(col=col)
if self.headers:
# pop the first item off, add to headers
self.headers.append(col[0])
col = col[1:]
if self.height and self.width:
for i, row in enumerate(self._data):
_row = list(row)
_row.append(col[i])
self._data[i] = tuple(_row)
else:
self._data = [tuple([row]) for row in col]
def insert(self, i, row=None, col=None):
"""Inserts a row at given position in Dataset"""
if row:
self._validate(row)
self._data.insert(i, tuple(row))
elif col:
pass
class DataBook(object):
"""A book of Dataset objects.
Currently, this exists only for XLS workbook support.
"""
def __init__(self, sets=[]):
self._datasets = sets
def __repr__(self):
try:
return '<%s databook>' % (self.title.lower())
except AttributeError:
return '<databook object>'
def add_sheet(self, dataset):
"""Add given dataset ."""
if type(dataset) is Dataset:
self._datasets.append(dataset)
else:
raise InvalidDatasetType
def _package(self):
collector = []
for dset in self._datasets:
collector.append(dict(
title = dset.title,
data = dset.dict
))
return collector
@property
def size(self):
"""The number of the Datasets within DataBook."""
return len(self._datasets)
@property
def xls(self):
"""Returns XLS representation of DataBook."""
stream = cStringIO.StringIO()
wb = xlwt.Workbook()
for i, dset in enumerate(self._datasets):
ws = wb.add_sheet(dset.title if dset.title else 'Sheet%s' % (i))
#for row in self._package(dicts=False):
for i, row in enumerate(dset._package(dicts=False)):
for j, col in enumerate(row):
ws.write(i, j, str(col))
wb.save(stream)
return stream.getvalue()
@property
def json(self):
"""Returns JSON representation of Databook."""
return json.dumps(self._package())
@property
def yaml(self):
"""Returns YAML representation of Databook."""
return yaml.dump(self._package())
class InvalidDatasetType(Exception):
"Only Datasets can be added to a DataBook"
class InvalidDimensions(Exception):
"Invalid size"
class UnsupportedFormat(NotImplementedError):
"Format is not supported"
+21
View File
@@ -0,0 +1,21 @@
# -*- coding: utf-8 -*-
import sys
class Struct(object):
"""Your attributes are belong to us."""
def __init__(self, **entries):
self.__dict__.update(entries)
def __getitem__(self, key):
return getattr(self, key, None)
def piped():
"""Returns piped input via stdin, else False."""
with sys.stdin as stdin:
# TTY is only way to detect if stdin contains data
return stdin.read() if not stdin.isatty() else None
Executable
+186
View File
@@ -0,0 +1,186 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Tests for tablib."""
import unittest
import tablib
class TablibTestCase(unittest.TestCase):
"""Tablib test cases."""
def setUp(self):
"""Create simple data set with headers."""
global data
data = tablib.Dataset()
self.headers = ('first_name', 'last_name', 'gpa')
self.john = ('John', 'Adams', 90)
self.george = ('George', 'Washington', 67)
self.tom = ('Thomas', 'Jefferson', 50)
self.founders = tablib.Dataset(headers=self.headers)
self.founders.append(self.john)
self.founders.append(self.george)
self.founders.append(self.tom)
def tearDown(self):
"""Teardown."""
pass
def test_empty_append(self):
"""Verify append() correctly adds tuple with no headers."""
new_row = (1, 2, 3)
data.append(new_row)
# Verify width/data
self.assertTrue(data.width == len(new_row))
self.assertTrue(data[0] == new_row)
def test_empty_append_with_headers(self):
"""Verify append() correctly detects mismatch of number of
headers and data.
"""
data.headers = ['first', 'second']
new_row = (1, 2, 3, 4)
self.assertRaises(tablib.InvalidDimensions, data.append, new_row)
def test_add_column(self):
"""Verify adding column works with/without headers."""
data.append(['kenneth'])
data.append(['bessie'])
new_col = ['reitz', 'monke']
data.append(col=new_col)
self.assertEquals(data[0], ('kenneth', 'reitz'))
self.assertEquals(data.width, 2)
# With Headers
data.headers = ('fname', 'lname')
new_col = ['age', 21, 22]
data.append(col=new_col)
self.assertEquals(data[new_col[0]], new_col[1:])
def test_add_column_no_data_no_headers(self):
"""Verify adding new column with no headers."""
new_col = ('reitz', 'monke')
data.append(col=new_col)
self.assertEquals(data[0], tuple([new_col[0]]))
self.assertEquals(data.width, 1)
self.assertEquals(data.height, len(new_col))
def test_add_column_no_data_with_headers(self):
"""Verify adding new column with headers."""
data.headers = ('first', 'last')
new_col = ('age',)
data.append(col=new_col)
self.assertEquals(len(data.headers), 3)
self.assertEquals(data.width, 3)
new_col = ('foo', 'bar')
self.assertRaises(tablib.InvalidDimensions, data.append, col=new_col)
def test_header_slicing(self):
"""Verify slicing by headers."""
self.assertEqual(self.founders['first_name'],
[self.john[0], self.george[0], self.tom[0]])
self.assertEqual(self.founders['last_name'],
[self.john[1], self.george[1], self.tom[1]])
self.assertEqual(self.founders['gpa'],
[self.john[2], self.george[2], self.tom[2]])
def test_data_slicing(self):
"""Verify slicing by data."""
# Slice individual rows
self.assertEqual(self.founders[0], self.john)
self.assertEqual(self.founders[:1], [self.john])
self.assertEqual(self.founders[1:2], [self.george])
self.assertEqual(self.founders[-1], self.tom)
self.assertEqual(self.founders[3:], [])
# Slice multiple rows
self.assertEqual(self.founders[:], [self.john, self.george, self.tom])
self.assertEqual(self.founders[0:2], [self.john, self.george])
self.assertEqual(self.founders[1:3], [self.george, self.tom])
self.assertEqual(self.founders[2:], [self.tom])
def test_delete(self):
"""Verify deleting from dataset works."""
# Delete from front of object
del self.founders[0]
self.assertEqual(self.founders[:], [self.george, self.tom])
# Verify dimensions, width should NOT change
self.assertEqual(self.founders.height, 2)
self.assertEqual(self.founders.width, 3)
# Delete from back of object
del self.founders[1]
self.assertEqual(self.founders[:], [self.george])
# Verify dimensions, width should NOT change
self.assertEqual(self.founders.height, 1)
self.assertEqual(self.founders.width, 3)
# Delete from invalid index
self.assertRaises(IndexError, self.founders.__delitem__, 3)
def test_csv_export(self):
"""Verify exporting dataset object as CSV."""
# Build up the csv string with headers first, followed by each row
csv = ''
for col in self.headers:
csv += col + ','
csv = csv.strip(',') + '\r\n'
for founder in self.founders:
for col in founder:
csv += str(col) + ','
csv = csv.strip(',') + '\r\n'
self.assertEqual(csv, self.founders.csv)
def test_unicode_append(self):
"""Passes in a single unicode charecter and exports."""
new_row = ('å', 'é')
data.append(new_row)
data.json
data.yaml
data.csv
data.xls
if __name__ == '__main__':
unittest.main()
-10
View File
@@ -1,10 +0,0 @@
pytest
pytest-cov
backports.csv; python_version < '3.0'
MarkupPy
odfpy
openpyxl>=2.4.0
pandas
pyyaml
xlrd
xlwt
-1105
View File
File diff suppressed because it is too large Load Diff
-40
View File
@@ -1,40 +0,0 @@
[tox]
usedevelop = true
minversion = 2.4
envlist =
py{27,35,36,37,38}-tests,
py37-{docs,lint}
[testenv]
basepython =
py27: python2.7
py35: python3.5
py36: python3.6
py37: python3.7
py38: python3.8
deps =
tests: -rtests/requirements.txt
docs: sphinx
extras = pandas
commands =
tests: pytest {posargs:tests}
docs: sphinx-build -b html -d {envtmpdir}/doctrees docs {envtmpdir}/html
[testenv:py37-lint]
basepython = python3.7
deps =
flake8
# flake8-black
# flake8-isort
twine
check-manifest
commands =
# flake8 src/tablib tests/
check-manifest -v
python setup.py sdist
twine check dist/*
[flake8]
exclude =
.tox
ignore=E501,E127,E128,E124