Nested collection record decode error

[kerozen]

when using field traversal in projections, I get error :

PHP Fatal error:  Cannot access empty property in /var/www/paulbot/OrientDB/OrientDBRecordDecoder.php on line 746

the query was :

SELECT tweets.size() as nb_tw, id, tweets.tout.rt as nb_rted  FROM U WHERE tweets.size() > 0 and tweets.tout.rt.size() > 0 order by nb_tw desc LIMIT 20 

trying to debug by myself I added l.745

if ($name == "") {  
error_log("Decode error ! ".$values);
error_log("Decode error ! ".$this->content);
} else { ...

and got values and content vars logged :

Decode error ! a:1:{i:0;O:16:"OrientDBTypeLink":3:{s:22:"
Decode error ! nb_tw:37,id:"583489647",nb_rted:[[#15:5],[#15:1],[#15:10,#15:88],[#15:15,#15:24],[#15:48],[#15:53,#15:294],[#15:1],[#15:10,#15:88],[#15:99,#15:131],[#15:93],[#15:109],[#15:99,#15:131],[#15:135,#15:140,#15:192,#15:197,#15:246,#15:247,#15:249,#15:250,#15:251,#15:252,#15:253,#15:255,#15:256],[#15:157,#15:162,#15:169,#15:180,#15:265,#15:269,#15:274,#15:280],[#15:151,#15:179,#15:262,#15:279],[#15:177,#15:278],[#15:135,#15:140,#15:192,#15:197,#15:246,#15:247,#15:249,#15:250,#15:251,#15:252,#15:253,#15:255,#15:256],[#15:227],[#15:234,#15:235,#15:236,#15:239],[#15:135,#15:140,#15:192,#15:197,#15:246,#15:247,#15:249,#15:250,#15:251,#15:252,#15:253,#15:255,#15:256],[#15:157,#15:162,#15:169,#15:180,#15:265,#15:269,#15:274,#15:280],[#15:151,#15:179,#15:262,#15:279],[#15:177,#15:278],[#15:53,#15:294]]

but after that the script ran out of memory.

PHP Fatal error:  Allowed memory size of 746586112 bytes exhausted (tried to allocate 35 bytes) in /var/www/paulbot/OrientDB/OrientDBRecordDecoder.php on line 799

any clue how to solve this ?

[AntonTerekhov]

Hi!
Can you please report version of OrientDB and full record by using
$OrientDB->setDebug(true);

[kerozen]

the version is the last snapshot : 1.2.0

here is the debug output of the query :

0 : 29 00 00 00 78 73 00 00 00 e7 00 00 00 3a 63 6f [)...xs.......:co] 10 : 6d 2e 6f 72 69 65 6e 74 65 63 68 6e 6f 6c 6f 67 [m.orientechnolog] 20 : 69 65 73 2e 6f 72 69 65 6e 74 2e 63 6f 72 65 2e [ies.orient.core.] 30 : 73 71 6c 2e 71 75 65 72 79 2e 4f 53 51 4c 53 79 [sql.query.OSQLSy] 40 : 6e 63 68 51 75 65 72 79 00 00 00 96 53 45 4c 45 [nchQuery....SELE] 50 : 43 54 20 74 77 65 65 74 73 2e 73 69 7a 65 28 29 [CT tweets.size()] 60 : 20 61 73 20 6e 62 5f 74 77 2c 20 69 64 2c 20 74 [ as nb_tw, id, t] 70 : 77 65 65 74 73 2e 74 6f 75 74 2e 72 74 20 61 73 [weets.tout.rt as] 80 : 20 6e 62 5f 72 74 65 64 20 20 46 52 4f 4d 20 55 [ nb_rted FROM U] 90 : 20 57 48 45 52 45 20 74 77 65 65 74 73 2e 73 69 [ WHERE tweets.si] A0 : 7a 65 28 29 20 3e 20 30 20 61 6e 64 20 74 77 65 [ze() > 0 and twe] B0 : 65 74 73 2e 74 6f 75 74 2e 72 74 2e 73 69 7a 65 [ets.tout.rt.size] C0 : 28 29 20 3e 20 30 20 6f 72 64 65 72 20 62 79 20 [() > 0 order by ] D0 : 6e 62 5f 74 77 20 64 65 73 63 20 4c 49 4d 49 54 [nb_tw desc LIMIT] E0 : 20 31 ff ff ff ff 00 00 00 03 2a 3a 30 00 00 00 [ 1........*:0...] F0 : 00 [.] >request_status 0 : 00 [.] >TransactionID 0 : 00 00 00 78 [...x] >status 0 : 6c [l] >records_count 0 : 00 00 00 01 [....] >record_marker 0 : 00 00 [..] >record_type 0 : 64 [d] >record_clusterID 0 : ff fe [..] >record_position 0 : 00 00 00 00 [....] 0 : 00 00 00 1f [....] >record_version 0 : 00 00 00 00 [....] >record_content 0 : 00 00 03 1f [....] 0 : 6e 62 5f 74 77 3a 33 37 2c 69 64 3a 22 35 38 33 [nb_tw:37,id:"583] 10 : 34 38 39 36 34 37 22 2c 6e 62 5f 72 74 65 64 3a [489647",nb_rted:] 20 : 5b 5b 23 31 35 3a 35 5d 2c 5b 23 31 35 3a 31 5d [[[#15:5],[#15:1]] 30 : 2c 5b 23 31 35 3a 31 30 2c 23 31 35 3a 38 38 5d [,[#15:10,#15:88]] 40 : 2c 5b 23 31 35 3a 31 35 2c 23 31 35 3a 32 34 5d [,[#15:15,#15:24]] 50 : 2c 5b 23 31 35 3a 34 38 5d 2c 5b 23 31 35 3a 35 [,[#15:48],[#15:5] 60 : 33 2c 23 31 35 3a 32 39 34 5d 2c 5b 23 31 35 3a [3,#15:294],[#15:] 70 : 31 5d 2c 5b 23 31 35 3a 31 30 2c 23 31 35 3a 38 [1],[#15:10,#15:8] 80 : 38 5d 2c 5b 23 31 35 3a 39 39 2c 23 31 35 3a 31 [8],[#15:99,#15:1] 90 : 33 31 5d 2c 5b 23 31 35 3a 39 33 5d 2c 5b 23 31 [31],[#15:93],[#1] A0 : 35 3a 31 30 39 5d 2c 5b 23 31 35 3a 39 39 2c 23 [5:109],[#15:99,#] B0 : 31 35 3a 31 33 31 5d 2c 5b 23 31 35 3a 31 33 35 [15:131],[#15:135] C0 : 2c 23 31 35 3a 31 34 30 2c 23 31 35 3a 31 39 32 [,#15:140,#15:192] D0 : 2c 23 31 35 3a 31 39 37 2c 23 31 35 3a 32 34 36 [,#15:197,#15:246] E0 : 2c 23 31 35 3a 32 34 37 2c 23 31 35 3a 32 34 39 [,#15:247,#15:249] F0 : 2c 23 31 35 3a 32 35 30 2c 23 31 35 3a 32 35 31 [,#15:250,#15:251] 100 : 2c 23 31 35 3a 32 35 32 2c 23 31 35 3a 32 35 33 [,#15:252,#15:253] 110 : 2c 23 31 35 3a 32 35 35 2c 23 31 35 3a 32 35 36 [,#15:255,#15:256] 120 : 5d 2c 5b 23 31 35 3a 31 35 37 2c 23 31 35 3a 31 [],[#15:157,#15:1] 130 : 36 32 2c 23 31 35 3a 31 36 39 2c 23 31 35 3a 31 [62,#15:169,#15:1] 140 : 38 30 2c 23 31 35 3a 32 36 35 2c 23 31 35 3a 32 [80,#15:265,#15:2] 150 : 36 39 2c 23 31 35 3a 32 37 34 2c 23 31 35 3a 32 [69,#15:274,#15:2] 160 : 38 30 5d 2c 5b 23 31 35 3a 31 35 31 2c 23 31 35 [80],[#15:151,#15] 170 : 3a 31 37 39 2c 23 31 35 3a 32 36 32 2c 23 31 35 [:179,#15:262,#15] 180 : 3a 32 37 39 5d 2c 5b 23 31 35 3a 31 37 37 2c 23 [:279],[#15:177,#] 190 : 31 35 3a 32 37 38 5d 2c 5b 23 31 35 3a 31 33 35 [15:278],[#15:135] 1A0 : 2c 23 31 35 3a 31 34 30 2c 23 31 35 3a 31 39 32 [,#15:140,#15:192] 1B0 : 2c 23 31 35 3a 31 39 37 2c 23 31 35 3a 32 34 36 [,#15:197,#15:246] 1C0 : 2c 23 31 35 3a 32 34 37 2c 23 31 35 3a 32 34 39 [,#15:247,#15:249] 1D0 : 2c 23 31 35 3a 32 35 30 2c 23 31 35 3a 32 35 31 [,#15:250,#15:251] 1E0 : 2c 23 31 35 3a 32 35 32 2c 23 31 35 3a 32 35 33 [,#15:252,#15:253] 1F0 : 2c 23 31 35 3a 32 35 35 2c 23 31 35 3a 32 35 36 [,#15:255,#15:256] 200 : 5d 2c 5b 23 31 35 3a 32 32 37 5d 2c 5b 23 31 35 [],[#15:227],[#15] 210 : 3a 32 33 34 2c 23 31 35 3a 32 33 35 2c 23 31 35 [:234,#15:235,#15] 220 : 3a 32 33 36 2c 23 31 35 3a 32 33 39 5d 2c 5b 23 [:236,#15:239],[#] 230 : 31 35 3a 31 33 35 2c 23 31 35 3a 31 34 30 2c 23 [15:135,#15:140,#] 240 : 31 35 3a 31 39 32 2c 23 31 35 3a 31 39 37 2c 23 [15:192,#15:197,#] 250 : 31 35 3a 32 34 36 2c 23 31 35 3a 32 34 37 2c 23 [15:246,#15:247,#] 260 : 31 35 3a 32 34 39 2c 23 31 35 3a 32 35 30 2c 23 [15:249,#15:250,#] 270 : 31 35 3a 32 35 31 2c 23 31 35 3a 32 35 32 2c 23 [15:251,#15:252,#] 280 : 31 35 3a 32 35 33 2c 23 31 35 3a 32 35 35 2c 23 [15:253,#15:255,#] 290 : 31 35 3a 32 35 36 5d 2c 5b 23 31 35 3a 31 35 37 [15:256],[#15:157] 2A0 : 2c 23 31 35 3a 31 36 32 2c 23 31 35 3a 31 36 39 [,#15:162,#15:169] 2B0 : 2c 23 31 35 3a 31 38 30 2c 23 31 35 3a 32 36 35 [,#15:180,#15:265] 2C0 : 2c 23 31 35 3a 32 36 39 2c 23 31 35 3a 32 37 34 [,#15:269,#15:274] 2D0 : 2c 23 31 35 3a 32 38 30 5d 2c 5b 23 31 35 3a 31 [,#15:280],[#15:1] 2E0 : 35 31 2c 23 31 35 3a 31 37 39 2c 23 31 35 3a 32 [51,#15:179,#15:2] 2F0 : 36 32 2c 23 31 35 3a 32 37 39 5d 2c 5b 23 31 35 [62,#15:279],[#15] 300 : 3a 31 37 37 2c 23 31 35 3a 32 37 38 5d 2c 5b 23 [:177,#15:278],[#] 310 : 31 35 3a 35 33 2c 23 31 35 3a 32 39 34 5d 5d [15:53,#15:294]]]

[AntonTerekhov]

Thanks, this is bug =(
I'll fix it.

[kerozen]

thanks a lot :)

reading the code I didn't understood much ... how is the record decoded ?

[AntonTerekhov]

Well, Its just a simple state machine.
A bit messy (as splitting this huge code into functions slow execution a lot) with some speedups (using regex instead byte-after-byte byte reading on long strings).

[F21]

Has this been fixed yet?

[AntonTerekhov]

Nope, its a lot of work. Support one level of nesting is not too complex, but this is only half of work, as we can expect much more deeper nesting levels.
However, I'll fix that, but no due date yet.

[F21]

Do you have any ideas/plans as to how this can be done? I would like contribute if possible :)

[AntonTerekhov]

The Plan is simple: add "some" code to the state machine. To be precise,
state machine must now have more complex stack to support nesting states.

[F21]

By the state machine, you mean the state machine in OrientDBRecordDecoder.php right?

If that is the case, I will study the code over the next few days and see what I can contribute :)

[fsicre]

Waiting for the official patch too

Meanwhile I can propose you this small gist

https://gist.github.com/fsicre/1f3b24529ad4bf8acd1d

Hope this will help.

[AntonTerekhov]

Smart patch =)

But, in case of nested documents it will not pass, I think.

[fsicre]

Your right ! ;-)

Please look at this one https://gist.github.com/anonymous/4986634

I'm trying to understand the existing state machine adding a stack of nested "containers".

This "gist" I have written handles correctly these cases :

INSERT INTO cluster:default SET a=[1,2,3], b=[{"a":1,"b":2,"c":3},{"d":4,"e":5,"f":6}], c=[[1,2,3],[4,5,6]]

INSERT INTO cluster:default SET a=[1,2,3], b=[{"a":1,"b":2,"c":3},{"d":4,"e":5,"f":6}], c=[[1,2,3],[4,5,6]], d=["a","b","c"]

INSERT INTO cluster:default SET a={"a":1,"b":[2,3,4],"c":{"d":5}}

INSERT INTO cluster:default SET a={"a":1,"b":2,"c":3}, b={"a":[1,2,3],"b":[4,5,6],"c":[7,8,9]}, c={"a":[{"a1":11,"a2":12},{"b1":21,"b2":22}]}

INSERT INTO cluster:default SET a={"a":1,"b":2,"c":3}, b={"a":[1,2,3],"b":[4,5,6],"c":[7,8,9]}, c={"a":[{"a1":11,"a2":12},{"b1":21,"b2":22}],"b":[{"c1":31,"c2":32},{"d1":41,"d2":42}]}
The order of the [] are respected, but the order of the keys in the {} can not because OrientDB seems to loose it.

Needs much much more testing.

[fsicre]

The gist seems anonymous but it is mine in fact. sorry for that.