264 lines
8.9 KiB
ReStructuredText
264 lines
8.9 KiB
ReStructuredText
.. _bulk-import:
|
|
|
|
Bulk Import
|
|
===========
|
|
|
|
OpenERP has included a bulk import facility for CSV-ish files for a
|
|
long time. With 7.0, both the interface and internal implementation
|
|
have been redone, resulting in
|
|
:meth:`~openerp.osv.orm.BaseModel.load`.
|
|
|
|
.. note::
|
|
|
|
the previous bulk-loading method,
|
|
:meth:`~openerp.osv.orm.BaseModel.import_data`, remains for
|
|
backwards compatibility but was re-implemented on top of
|
|
:meth:`~openerp.osv.orm.BaseModel.load`, while its interface is
|
|
unchanged its precise behavior has likely been altered for some
|
|
cases (it shouldn't throw exceptions anymore in many cases where
|
|
it previously did)
|
|
|
|
This document attempts to explain the behavior and limitations of
|
|
:meth:`~openerp.osv.orm.BaseModel.load`.
|
|
|
|
Data
|
|
----
|
|
|
|
The input ``data`` is a regular row-major matrix of strings (in Python
|
|
datatype terms, a ``list`` of rows, each row being a ``list`` of
|
|
``str``, all rows must be of equal length). Each row must be the same
|
|
length as the ``fields`` list preceding it in the argslist.
|
|
|
|
Each field of ``fields`` maps to a (potentially relational and nested)
|
|
field of the model under import, and the corresponding column of the
|
|
``data`` matrix provides a value for the field for each record.
|
|
|
|
Generally speaking each row of the input yields a record of output,
|
|
and each cell of a row yields a value for the corresponding field of
|
|
the row's record. There is currently one exception for this rule:
|
|
|
|
One to Many fields
|
|
++++++++++++++++++
|
|
|
|
Because O2M fields contain multiple records "embedded" in the main
|
|
one, and these sub-records are fully dependent on the main record (are
|
|
no other references to the sub-records in the system), they have to be
|
|
spliced into the matrix somehow. This is done by adding lines composed
|
|
*only* of o2m record fields below the main record:
|
|
|
|
.. literalinclude:: 06_misc_import_o2m.txt
|
|
|
|
the sections in double-lines represent the span of two o2m
|
|
fields. During parsing, they are extracted into their own ``data``
|
|
matrix for the o2m field they correspond to.
|
|
|
|
Import process
|
|
--------------
|
|
|
|
Here are the phases of import. Note that the concept of "phases" is
|
|
fuzzy as it's currently more of a pipeline, each record moves through
|
|
the entire pipeline before the next one is processed.
|
|
|
|
Extraction
|
|
++++++++++
|
|
|
|
The first phase of the import is the extraction of the current row
|
|
(and potentially a section of rows following it if it has One to Many
|
|
fields) into a record dictionary. The keys are the ``fields``
|
|
originally passed to :meth:`~openerp.osv.orm.BaseModel.load`, and the
|
|
values are either the string value at the corresponding cell (for
|
|
non-relational fields) or a list of sub-records (for all relational
|
|
fields).
|
|
|
|
This phase also generates the ``rows`` indexes for any
|
|
:ref:`import-message` produced thereafter.
|
|
|
|
Conversion
|
|
++++++++++
|
|
|
|
This second phase takes the record dicts, extracts the :term:`database
|
|
ID` and :term:`external ID` if present and attempts to convert each
|
|
field to a type matching what OpenERP expects to write.
|
|
|
|
* Empty fields (empty strings) are replaced with the ``False`` value
|
|
|
|
* Non-empty fields are converted through
|
|
:class:`~openerp.addons.base.ir.ir_fields.ir_fields_converter`
|
|
|
|
.. note:: if a field is specified in the import, its default will *never* be
|
|
used. If some records need to have a value and others need to use
|
|
the model's default, either specify that default explicitly or do
|
|
the import in two phases.
|
|
|
|
Char, text and binary fields
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Are returned as-is, without any alteration.
|
|
|
|
Boolean fields
|
|
~~~~~~~~~~~~~~
|
|
|
|
The string value is compared (in a case-insensitive manner) to ``0``,
|
|
``false`` and ``no`` as well of any translation thereof loaded in the
|
|
database. If the value matches one of these, the field is set to
|
|
``False``.
|
|
|
|
Otherwise the field is compared to ``1``, ``true`` and ``yes`` (and
|
|
any translation of these in the database). The field is always set to
|
|
``True``, but if the value does not match one of these a warning will
|
|
also be output.
|
|
|
|
Integers and float fields
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The field is parsed with Python's built-in conversion routines
|
|
(``int`` and ``float`` respectively), if the conversion fails an error
|
|
is generated.
|
|
|
|
Selection fields
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
The field is compared to 1. the values of the selection (first part of
|
|
each selection tuple) and 2. all translations of the selection label
|
|
found in the database.
|
|
|
|
If one of these is matched, the corresponding value is set on the
|
|
field.
|
|
|
|
Otherwise an error is generated.
|
|
|
|
The same process applies to both list-type and function-type selection
|
|
fields.
|
|
|
|
Many to One field
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
If the specified field is the relational field itself (``m2o``), the
|
|
value is used in a ``name_search``. The first record returned by
|
|
``name_search`` is used as the field's value.
|
|
|
|
If ``name_search`` finds no value, an error is generated. If
|
|
``name_search`` finds multiple value, a warning is generated to warn
|
|
the user of ``name_search`` collisions.
|
|
|
|
If the specified field is a :term:`external ID` (``m2o/id``), the
|
|
corresponding record it looked up in the database and used as the
|
|
field's value. If no record is found matching the provided external
|
|
ID, an error is generated.
|
|
|
|
If the specified field is a :term:`database ID` (``m2o/.id``), the
|
|
process is the same as for external ids (on database identifiers
|
|
instead of external ones).
|
|
|
|
Many to Many field
|
|
~~~~~~~~~~~~~~~~~~
|
|
|
|
The field's value is interpreted as a comma-separated list of names,
|
|
external ids or database ids. For each one, the process previously
|
|
used for the many to one field is applied.
|
|
|
|
One to Many field
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
For each o2m record extracted, if the record has a ``name``,
|
|
:term:`external ID` or :term:`database ID` the :term:`database ID` is
|
|
looked up and checked through the same process as for m2o fields.
|
|
|
|
If a :term:`database ID` was found, a LINK_TO command is emmitted,
|
|
followed by an UPDATE with the non-db values for the relational field.
|
|
|
|
Otherwise a CREATE command is emmitted.
|
|
|
|
Date fields
|
|
~~~~~~~~~~~
|
|
|
|
The value's format is checked against
|
|
:data:`~openerp.tools.misc.DEFAULT_SERVER_DATE_FORMAT`, an error is
|
|
generated if it does not match the specified format.
|
|
|
|
Datetime fields
|
|
~~~~~~~~~~~~~~~
|
|
|
|
The value's format is checked against
|
|
:data:`~openerp.tools.misc.DEFAULT_SERVER_DATETIME_FORMAT`, an error
|
|
is generated if it does not match.
|
|
|
|
The value is then interpreted as a datetime in the user's
|
|
timezone. The timezone is specified thus:
|
|
|
|
* If the import ``context`` contains a ``tz`` key with a valid
|
|
timezone name, this is the timezone of the datetime.
|
|
|
|
* Otherwise if the user performing the import has a ``tz`` attribute
|
|
set to a valid timezone name, this is the timezone of the datetime.
|
|
|
|
* Otherwise interpret the datetime as being in the ``UTC`` timezone.
|
|
|
|
Create/Write
|
|
++++++++++++
|
|
|
|
If the conversion was successful, the converted record is then saved
|
|
to the database via ``(ir.model.data)._update``.
|
|
|
|
Error handling
|
|
++++++++++++++
|
|
|
|
The import process will only catch 2 types of exceptions to convert
|
|
them to error messages: ``ValueError`` during the conversion process,
|
|
and sub-exceptions of ``psycopg2.Error`` during the create/write
|
|
process.
|
|
|
|
The import process uses savepoint to:
|
|
|
|
* protect the overall transaction from the failure of each ``_update``
|
|
call, if an ``_update`` call fails the savepoint is rolled back and
|
|
the import process keeps going in order to obtain as many error
|
|
messages as possible during each run.
|
|
|
|
* protect the import as a whole, a savepoint is created before
|
|
starting and if any error is generated that savepoint is rolled
|
|
back. The rest of the transaction (anything not within the import
|
|
process) will be left untouched.
|
|
|
|
.. _import-message:
|
|
.. _import-messages:
|
|
|
|
Messages
|
|
--------
|
|
|
|
A message is a dictionary with 5 mandatory keys and one optional key:
|
|
|
|
``type``
|
|
the type of message, either ``warning`` or ``error``. Any
|
|
``error`` message indicates the import failed and was rolled back.
|
|
|
|
``message``
|
|
the message's actual text, which should be translated and can be
|
|
shown to the user directly
|
|
|
|
``rows``
|
|
a dict with 2 keys ``from`` and ``to``, indicates the range of
|
|
rows in ``data`` which generated the message
|
|
|
|
``record``
|
|
a single integer, for warnings the index of the record which
|
|
generated the message (can be obtained from a non-false ``ids``
|
|
result)
|
|
|
|
``field``
|
|
the name of the (logical) OpenERP field for which the error or
|
|
warning was generated
|
|
|
|
``moreinfo`` (optional)
|
|
A string, a list or a dict, leading to more information about the
|
|
warning.
|
|
|
|
* If ``moreinfo`` is a string, it is a supplementary warnings
|
|
message which should be hidden by default
|
|
* If ``moreinfo`` is a list, it provides a number of possible or
|
|
alternative values for the string
|
|
* If ``moreinfo`` is a dict, it is an OpenERP action descriptor
|
|
which can be executed to get more information about the issues
|
|
with the field. If present, the ``help`` key serves as a label
|
|
for the action (e.g. the text of the link).
|