[ADD] some import doc
bzr revid: xmo@openerp.com-20120928091519-ksdprqdq0tpklggw
This commit is contained in:
parent
aaff4459c4
commit
879886decd
|
@ -0,0 +1,231 @@
|
||||||
|
.. _bulk-import:
|
||||||
|
|
||||||
|
Bulk Import
|
||||||
|
===========
|
||||||
|
|
||||||
|
OpenERP has included a bulk import facility for CSV-ish files for a
|
||||||
|
long time. With 7.0, both the interface and internal implementation
|
||||||
|
have been redone, resulting in
|
||||||
|
:meth:`~openerp.osv.orm.BaseModel.load`.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
the previous bulk-loading method,
|
||||||
|
:meth:`~openerp.osv.orm.BaseModel.import_data`, remains for
|
||||||
|
backwards compatibility but was re-implemented on top of
|
||||||
|
:meth:`~openerp.osv.orm.BaseModel.load`, while its interface is
|
||||||
|
unchanged its precise behavior has likely been altered for some
|
||||||
|
cases (it shouldn't throw exceptions anymore in many cases where
|
||||||
|
it previously did)
|
||||||
|
|
||||||
|
This document attempts to explain the behavior and limitations of
|
||||||
|
:meth:`~openerp.osv.orm.BaseModel.load`.
|
||||||
|
|
||||||
|
Data
|
||||||
|
====
|
||||||
|
|
||||||
|
The input ``data`` is a regular row-major matrix of strings (in Python
|
||||||
|
datatype terms, a ``list`` of rows, each row being a ``list`` of
|
||||||
|
``str``, all rows must be of equal length). Each row must be the same
|
||||||
|
length as the ``fields`` list preceding it in the argslist.
|
||||||
|
|
||||||
|
Each field of ``fields`` maps to a (potentially relational and nested)
|
||||||
|
field of the model under import, and the corresponding column of the
|
||||||
|
``data`` matrix provides a value for the field for each record.
|
||||||
|
|
||||||
|
Generally speaking each row of the input yields a record of output,
|
||||||
|
and each cell of a row yields a value for the corresponding field of
|
||||||
|
the row's record. There is currently one exception for this rule:
|
||||||
|
|
||||||
|
One to Many fields
|
||||||
|
------------------
|
||||||
|
|
||||||
|
Because O2M fields contain multiple records "embedded" in the main
|
||||||
|
one, and these sub-records are fully dependent on the main record (are
|
||||||
|
no other references to the sub-records in the system), they have to be
|
||||||
|
spliced into the matrix somehow. This is done by adding lines composed
|
||||||
|
*only* of o2m record fields below the main record:
|
||||||
|
|
||||||
|
.. literalinclude:: o2m.txt
|
||||||
|
|
||||||
|
the sections in double-lines represent the span of two o2m
|
||||||
|
fields. During parsing, they are extracted into their own ``data``
|
||||||
|
matrix for the o2m field they correspond to.
|
||||||
|
|
||||||
|
Import process
|
||||||
|
==============
|
||||||
|
|
||||||
|
Here are the phases of import. Note that the concept of "phases" is
|
||||||
|
fuzzy as it's currently more of a pipeline, each record moves through
|
||||||
|
the entire pipeline before the next one is processed.
|
||||||
|
|
||||||
|
Extraction
|
||||||
|
----------
|
||||||
|
|
||||||
|
The first phase of the import is the extraction of the current row
|
||||||
|
(and potentially a section of rows following it if it has One to Many
|
||||||
|
fields) into a record dictionary. The keys are the ``fields``
|
||||||
|
originally passed to :meth:`~openerp.osv.orm.BaseModel.load`, and the
|
||||||
|
values are either the string value at the corresponding cell (for
|
||||||
|
non-relational fields) or a list of sub-records (for all relational
|
||||||
|
fields).
|
||||||
|
|
||||||
|
This phase also generates the ``rows`` indexes for any
|
||||||
|
:ref:`import-message` produced thereafter.
|
||||||
|
|
||||||
|
Conversion
|
||||||
|
----------
|
||||||
|
|
||||||
|
This second phase takes the record dicts, extracts the :ref:`dbid` and
|
||||||
|
:ref:`xid` if present and attempts to convert each field to a type
|
||||||
|
matching what OpenERP expects to write. Empty fields (empty strings)
|
||||||
|
are replaced with the ``False`` value, and non-empty fields are
|
||||||
|
converted through
|
||||||
|
:class:`~openerp.addons.base.ir.ir_fields.ir_fields_converter`:
|
||||||
|
|
||||||
|
Char, text and binary fields
|
||||||
|
++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
Are returned as-is, without any alteration.
|
||||||
|
|
||||||
|
Boolean fields
|
||||||
|
++++++++++++++
|
||||||
|
|
||||||
|
The string value is compared (in a case-insensitive manner) to ``0``,
|
||||||
|
``false`` and ``no`` as well of any translation thereof loaded in the
|
||||||
|
database. If the value matches one of these, the field is set to
|
||||||
|
``False``.
|
||||||
|
|
||||||
|
Otherwise the field is compared to ``1``, ``true`` and ``yes`` (and
|
||||||
|
any translation of these in the database). The field is always set to
|
||||||
|
``True``, but if the value does not match one of these a warning will
|
||||||
|
also be output.
|
||||||
|
|
||||||
|
Integers and float fields
|
||||||
|
+++++++++++++++++++++++++
|
||||||
|
|
||||||
|
The field is parsed with Python's built-in conversion routines
|
||||||
|
(``int`` and ``float`` respectively), if the conversion fails an error
|
||||||
|
is generated.
|
||||||
|
|
||||||
|
Selection fields
|
||||||
|
++++++++++++++++
|
||||||
|
|
||||||
|
The field is compared to 1. the values of the selection (first part of
|
||||||
|
each selection tuple) and 2. all translations of the selection label
|
||||||
|
found in the database.
|
||||||
|
|
||||||
|
If one of these is matched, the corresponding value is set on the
|
||||||
|
field.
|
||||||
|
|
||||||
|
Otherwise an error is generated.
|
||||||
|
|
||||||
|
The same process applies to both list-type and function-type selection
|
||||||
|
fields.
|
||||||
|
|
||||||
|
Many to One field
|
||||||
|
+++++++++++++++++
|
||||||
|
|
||||||
|
If the specified field is the relational field itself (``m2o``), the
|
||||||
|
value is used in a ``name_search``. The first record returned by
|
||||||
|
``name_search`` is used as the field's value.
|
||||||
|
|
||||||
|
If ``name_search`` finds no value, an error is generated. If
|
||||||
|
``name_search`` finds multiple value, a warning is generated to warn
|
||||||
|
the user of ``name_search`` collisions.
|
||||||
|
|
||||||
|
If the specified field is a :ref:`xid` (``m2o/id``), the
|
||||||
|
corresponding record it looked up in the database and used as the
|
||||||
|
field's value. If no record is found matching the provided external
|
||||||
|
ID, an error is generated.
|
||||||
|
|
||||||
|
If the specified field is a :ref:`dbid` (``m2o/.id``), the process is
|
||||||
|
the same as for external ids (on database identifiers instead of
|
||||||
|
external ones).
|
||||||
|
|
||||||
|
Many to Many field
|
||||||
|
++++++++++++++++++
|
||||||
|
|
||||||
|
The field's value is interpreted as a comma-separated list of names,
|
||||||
|
external ids or database ids. For each one, the process previously
|
||||||
|
used for the many to one field is applied.
|
||||||
|
|
||||||
|
One to Many field
|
||||||
|
+++++++++++++++++
|
||||||
|
|
||||||
|
For each o2m record extracted, if the record has a ``name``,
|
||||||
|
:ref:`xid` or :ref:`dbid` the :ref:`dbid` is looked up and checked
|
||||||
|
through the same process as for m2o fields.
|
||||||
|
|
||||||
|
If a :ref:`dbid` was found, a LINK_TO command is emmitted, followed by
|
||||||
|
an UPDATE with the non-db values for the relational field.
|
||||||
|
|
||||||
|
Otherwise a CREATE command is emmitted.
|
||||||
|
|
||||||
|
Create/Write
|
||||||
|
------------
|
||||||
|
|
||||||
|
If the conversion was successful, the converted record is then saved
|
||||||
|
to the database via ``(ir.model.data)._update``.
|
||||||
|
|
||||||
|
Error handling
|
||||||
|
--------------
|
||||||
|
|
||||||
|
The import process will only catch 2 types of exceptions to convert
|
||||||
|
them to error messages: ``ValueError`` during the conversion process,
|
||||||
|
and sub-exceptions of ``psycopg2.Error`` during the create/write
|
||||||
|
process.
|
||||||
|
|
||||||
|
The import process uses savepoint to:
|
||||||
|
|
||||||
|
* protect the overall transaction from the failure of each ``_update``
|
||||||
|
call, if an ``_update`` call fails the savepoint is rolled back and
|
||||||
|
the import process keeps going in order to obtain as many error
|
||||||
|
messages as possible during each run.
|
||||||
|
|
||||||
|
* protect the import as a whole, a savepoint is created before
|
||||||
|
starting and if any error is generated that savepoint is rolled
|
||||||
|
back. The rest of the transaction (anything not within the import
|
||||||
|
process) will be left untouched.
|
||||||
|
|
||||||
|
.. _import-message:
|
||||||
|
.. _import-messages:
|
||||||
|
|
||||||
|
Messages
|
||||||
|
========
|
||||||
|
|
||||||
|
A message is a dictionary with 5 mandatory keys and one optional key:
|
||||||
|
|
||||||
|
``type``
|
||||||
|
the type of message, either ``warning`` or ``error``. Any
|
||||||
|
``error`` message indicates the import failed and was rolled back.
|
||||||
|
|
||||||
|
``message``
|
||||||
|
the message's actual text, which should be translated and can be
|
||||||
|
shown to the user directly
|
||||||
|
|
||||||
|
``rows``
|
||||||
|
a dict with 2 keys ``from`` and ``to``, indicates the range of
|
||||||
|
rows in ``data`` which generated the message
|
||||||
|
|
||||||
|
``record``
|
||||||
|
a single integer, for warnings the index of the record which
|
||||||
|
generated the message (can be obtained from a non-false ``ids``
|
||||||
|
result)
|
||||||
|
|
||||||
|
``field``
|
||||||
|
the name of the (logical) OpenERP field for which the error or
|
||||||
|
warning was generated
|
||||||
|
|
||||||
|
``moreinfo`` (optional)
|
||||||
|
A string, a list or a dict, leading to more information about the
|
||||||
|
warning.
|
||||||
|
|
||||||
|
* If ``moreinfo`` is a string, it is a supplementary warnings
|
||||||
|
message which should be hidden by default
|
||||||
|
* If ``moreinfo`` is a list, it provides a number of possible or
|
||||||
|
alternative values for the string
|
||||||
|
* If ``moreinfo`` is a dict, it is an OpenERP action descriptor
|
||||||
|
which can be executed to get more information about the issues
|
||||||
|
with the field. If present, the ``help`` key serves as a label
|
||||||
|
for the action (e.g. the text of the link).
|
|
@ -5,6 +5,7 @@ OpenERP Server
|
||||||
.. toctree::
|
.. toctree::
|
||||||
:maxdepth: 1
|
:maxdepth: 1
|
||||||
|
|
||||||
|
import
|
||||||
test-framework
|
test-framework
|
||||||
|
|
||||||
Changed in 7.0
|
Changed in 7.0
|
||||||
|
|
|
@ -0,0 +1,13 @@
|
||||||
|
+-------+-------+===========+===========+-------+-------+
|
||||||
|
|value01|value02‖o2m/value01|o2m/value02‖value03|value04|
|
||||||
|
+-------+-------+-----------+-----------+-------+-------+
|
||||||
|
| | ‖o2m/value11|o2m/value12‖ | |
|
||||||
|
+-------+-------+-----------+-----------+-------+-------+
|
||||||
|
| | ‖o2m/value21|o2m/value22‖ | |
|
||||||
|
+-------+-------+===========+===========+-------+-------+
|
||||||
|
|value11|value12‖o2m/value01|o2m/value02‖value13|value14|
|
||||||
|
+-------+-------+-----------+-----------+-------+-------+
|
||||||
|
| | ‖o2m/value11|o2m/value12‖ | |
|
||||||
|
+-------+-------+===========+===========+-------+-------+
|
||||||
|
|value21|value22| | |value23|value24|
|
||||||
|
+-------+-------+-----------+-----------+-------+-------+
|
|
@ -1310,37 +1310,6 @@ class BaseModel(object):
|
||||||
the same order they were extracted from the file. They can be passed
|
the same order they were extracted from the file. They can be passed
|
||||||
directly to :meth:`~read`
|
directly to :meth:`~read`
|
||||||
|
|
||||||
Each message is a dictionary with the following keys:
|
|
||||||
|
|
||||||
``type``
|
|
||||||
the type of message, either ``warning`` or ``error``. Any ``error``
|
|
||||||
message indicates the import failed and was rolled back.
|
|
||||||
``message``
|
|
||||||
the message's actual text, which should be translated and can be
|
|
||||||
shown to the user directly
|
|
||||||
``rows``
|
|
||||||
a dict with 2 keys ``from`` and ``to``, indicates the range of rows
|
|
||||||
in ``data`` which generated the message
|
|
||||||
``record``
|
|
||||||
a single integer, for warnings the index of the record which
|
|
||||||
generated the message (can be obtained from a non-false ``ids``
|
|
||||||
result)
|
|
||||||
``field``
|
|
||||||
the name of the (logical) OpenERP field for which the error or
|
|
||||||
warning was generated
|
|
||||||
``moreinfo`` (optional)
|
|
||||||
A string, a list or a dict, leading to more information about the
|
|
||||||
warning.
|
|
||||||
|
|
||||||
* If ``moreinfo`` is a string, it is a supplementary warnings
|
|
||||||
message which should be hidden by default
|
|
||||||
* If ``moreinfo`` is a list, it provides a number of possible or
|
|
||||||
alternative values for the string
|
|
||||||
* If ``moreinfo`` is a dict, it is an OpenERP action descriptor
|
|
||||||
which can be executed to get more information about the issues
|
|
||||||
with the field. If present, the ``help`` key serves as a label
|
|
||||||
for the action (e.g. the text of the link).
|
|
||||||
|
|
||||||
:param cr: cursor for the request
|
:param cr: cursor for the request
|
||||||
:param int uid: ID of the user attempting the data import
|
:param int uid: ID of the user attempting the data import
|
||||||
:param fields: list of fields to import, at the same index as the corresponding data
|
:param fields: list of fields to import, at the same index as the corresponding data
|
||||||
|
|
Loading…
Reference in New Issue