Number of ressources in osv.osv_memory object

Forgot to mention that i'm running latest version of openerp-server (5.0.7) on a debian stable release

Here is a test module putting the problem in evidence : just choose a size above 250 for school size, and then import...

Note: we'll need a YAML test for this.

On Sunday 04 April 2010, you wrote:
> Public bug reported:
>
> On a wizard based on two osv.osv_memory objects with one2many
> relationship, i canno't create more than 211 lines in the "many" object.
>
I will close this bug as "Invalid", because it is a feature of orm_memory
objects to have a hard limit on the number of records they can hold.

If you see at server/bin/osv/orm.py:2030 (latest trunk, more or less), you
will notice the "_max_count" member of the orm_memory objects. There is also a
"_check_time" one that means the limit is only enforced every so many calls to
self.create().

So, the absolute count of any orm_memory class is _max_count + _check_time -1
This, of course, applies to every class separately, not on aggregate.

If you need to have more data, define a class like:

class my_bigmem(osv.osv_memory):
    _name = "foobar.big_mem"
    _max_count = 1000 # or more..

In such a case, I would challenge your design decision to use an orm_memory
object for so many records. Why not a regular orm object after all? Memory
ones are volatile, you know..

Actually, as we've seen with a recent duplicate of this bug, this is a real issue - let me reopen it.
We're working on a solution...

Olivier,

a few remarks:
1) some other robust platforms (example Java servers) use to have a global
budget for session memory. Once you reach that limit (no matter how you fill
it), you get an ou of memory Exception. I think this is much more flexible
and explicit. It's alright OpenERP to say it's better than that the Java
platform, but in that case it just at least do an equivalent job or better.
Not just implement it's own hacky memory limit and session garbage... Isn't
there some robust standardized equivalent of Session management in Python?
2) at least an Exception should be raised when you reach the limit. Imagine
you reach it while processing accounting data, currently you could miss half
the records silently... Failing is OK, memory is limited anyway, but you
should be warned.
3) if there is such a hard limit, at least it should go in the global
configuration so it's explicit it's a parameter you can tweak to adapt your
need, just like you could give more memory to some application for instance.

Thanks for re-opening that bug. For the record, here is the cross link to my
original report where a large picking couldn't be processed:
https://bugs.launchpad.net/openobject-addons/+bug/709559
I'll make my best to look at the perf aspect of it on my side.

Regards

On Mon, Jan 31, 2011 at 7:47 AM, Olivier Dony (OpenERP) <
<email address hidden>> wrote:

> Actually, as we've seen with a recent duplicate of this bug, this is a real
> issue - let me reopen it.
> We're working on a solution...
>
> ** Changed in: openobject-server
> Importance: Undecided => High
>
> ** Changed in: openobject-server
> Status: Invalid => Confirmed
>
> ** Changed in: openobject-server
> Milestone: None => 6.0.2
>
> --
> You received this bug notification because you are subscribed to OpenERP
> Server.
> https://bugs.launchpad.net/bugs/555271
>
> Title:
> Number of ressources in osv.osv_memory object
>
> Status in OpenERP Server:
> Confirmed
>
> Bug description:
> On a wizard based on two osv.osv_memory objects with one2many
> relationship, i canno't create more than 211 lines in the "many"
> object.
>
> I want to do a "pre" import of many csv lines into the osv.osv_memory
> object before validating in the wizard and do the actual import in an
> osv.osv object.
>
> i made a simple scenario to check if it was a bug :
>
> from osv import fields,osv
>
> class test_school(osv.osv_memory):
> def _do_import(self, cr, uid, ids, context={}):
> myself=self.browse(cr, uid, ids, context)[0]
> for i in range(0,myself.size):
> self.pool.get("test.person").create(cr, uid, {
> "name":"Doe",
> "age":i,
> "school_id":myself.id
> })
> return True
>
> _name = "test.school"
> _columns = {
> "name":fields.char("name",size=20),
> "size":fields.integer("size"),
> "students":fields.one2many("test.person","school_id","students"),
> }
> test_school()
>
> class test_person(osv.osv_memory):
> _name = "test.person"
> _columns = {
> "name":fields.char("name",size=20),
> "age":fields.integer(...

We have implemented a global fix for this issue in the following server revisions:
- 6.0: rev 3339 <email address hidden>
- trunk: rev 3352 revision-id: <email address hidden>

This revision introduces two new config parameters and the corresponding server startup parameters:
- osv_memory_count_limit: limit on number or records in each osv_memory virtual table
- osv_memory_age_limit: limit on maximum age of records in each osv_memory virtual table

The default values have been changed to count=False (disabled, it used to be 200) and age=1.0 hour, so that with the default config the server will only expire osv_memory records that are 1 hour old or more, avoiding problems related with the use of osv_memory classes that instantiate a large number of short-lived objects.

Server administrators are free to use the configuration parameters to tweak these parameters globally if needed.
Module developers can also alter the self._max_count and self._max_hours values in each osv_memory class directly (but should keep in mind that the server admin expects the configured values to be applied!)

On Monday 14 February 2011, you wrote:
> The default values have been changed to count=False (disabled, it used
> to be 200) and age=1.0 hour, so that with the default config the server
> will only expire osv_memory records...

Yes, but may I remind, Olivier, this simple design fact about osv_memory
records:

   *** osv_memory records are VOLATILE, they are designed to be only temporary
storage of data for the purposes of intermediate steps, like the wizards ***

we must keep that in mind[2] and never expect them to reliably _store_ data.
And this design fact does invalidate the proposals to have "orm_memory pointed
to by plain orm records", or just have high limits etc.

My personal opinion is this: if we need to store some small amount[1] of
wizard decisions, yes, orm_memory storage is appropriate. If we need to edit
thousands of lines (which a human would take some hours to review), then we
should reconsider and use some regular ORM model instead. If, like in the
stock.partial.picking, we only use the list to display the set of records we
are about to process (but don't expect the user to edit them all one-by-one),
an orm_record may marginally be valid, though.

[1] a "small amount" can be subjective. In some cases 1-10 records, sometimes
even 1000.
[2] please don't hammer us describing that as a bug. It was designed to work
that way and you have to understand that.