General Python Unicode / ASCII Casting Issue Causing Trouble in Pyorient
<p>UPDATE: I opened an issue on github based on a  Ivan Mainetti's suggestion. If you want to weigh in there, it is :<a href="https://github.com/orientechnologies/orientdb/issues/6757" rel="nofollow">https://github.com/orientechnologies/orientdb/issues/6757</a></p>

<p>I am working on a database based on OrienDB and using a python interface for it. I've had pretty good luck with it, but I've run into a problem that seems to be the driver's (pyorient) wonkiness when dealing with certain unicode characters.</p>

<p>The data structure I'm uploading to the database looks like this:</p>

<pre><code>new_Node = {'@Nodes':
                {
                    "Abs_Address":Ono.absolute_address,
                    'Content':Ono.content,
                    'Heading':Ono.heading,
                    'Type':Ono.type,
                    'Value':Ono.value
                }
}
</code></pre>

<p>I have created literally hundreds of records flawlessly on OrientDB / pyorient. I don't think the problem is necessarily a pyorient specific question, however, as I think the reason it is failing on a particular record is because the Ono.absolute_address element has a unicode character that pyorient is somehow choking on. </p>

<p>The record I want to create has an Abs_address of /u/c/2/a1â2, but the node I get when I pass the value to the my data structure above is this:</p>

<pre><code>{'@Nodes': {'Content': '', 'Abs_Address': u'/u/c/2/a1\u20132', 'Type': 'section', 'Heading': ' Transferred', 'Value': u'1\u20132'}}
</code></pre>

<p>I think that somehow my problem is python is mixing unicode and ascii strings / chars? I'm a bit new to python and not declaring types, so I'm hoping this isn't an issue with pyorient perse given that the new_Node object doesn't output the properly formatted string...? Or is this an instance of pyorient not liking unicode? I'm tearing my hair out on this one. Any help is appreciated. </p>

<p>In case the error is coming from pyorient and not some kind of text encoding, here's the pyorient-related info. I am creating the record using this code:</p>

<pre><code>rec_position = self.pyo_client.record_create(14, new_Node)
</code></pre>

<p>And this is the error I'm getting:</p>

<pre><code>com.orientechnologies.orient.core.storage.ORecordDuplicatedException - Cannot index record Nodes{Content:,Abs_Address:null,Type:section,Heading: Transferred,Value:null}: found duplicated key 'null' in index 'Nodes.Abs_Address' previously assigned to the record #14:558
</code></pre>

<p>The error is odd as it suggests that the backend database is getting a null object for the address. Apparently it did create an entry for this "address," but it's not what I want it to do. I don't know why address strings with unicode are coming up null in the database... I can create it through orientDB studio using the exact string I fed into the new_Node data structure... but I can't use python to do the same thing. </p>

<p>Someone help?</p>

<p>EDIT:</p>

<p>Thanks to Laurent, I've narrowed the problem down to something to do with unicode objects and pyorient. Whenever a the variable I am passing is type unicode, the pyorient adapter sends a null value to the OrientDB database. I determined the value that is causing the problem is an ndash symbol, and Laurent helped me replace it with a minus sign using this code</p>

<pre><code>.replace(u"\u2013",u"-")
</code></pre>

<p>When I do that, however, pyorient gets unicode objects which it then passes as null values... This is not good. I can fix this short term by recasting the string using str(...) and this appears to solve my immediate problem:</p>

<pre><code>str(Ono.absolute_address.replace(u"\u2013",u"-"))
</code></pre>

<p>. Problem is, I know I will have symbols and other unusual characters in my DB data. I know the database supports the unicode strings because I can add them manually or use SQL syntax to do what I cannot do via pyorient and python...  I am assuming this is a dicey casting issue somewhere, but I'm not really sure where. This seems very similar to this problem: <a href="http://stackoverflow.duapp.com/questions/34757352/how-do-i-create-a-linked-record-in-orientdb-using-pyorient-library" rel="nofollow">http://stackoverflow.duapp.com/questions/34757352/how-do-i-create-a-linked-record-in-orientdb-using-pyorient-library</a></p>

<p>Any pyorient people out there? Python gods? Lucky s0bs? =)</p>

<p>I have tried your example on Python 3 with the development branch of pyorient with the latest version of OrientDB 2.2.11. If I pass the values without escaping them, your example seems to work for me and I get the right values back.</p>

<p>So this test works:</p>

<pre><code>def test_test1(self):
    new_Node = {'@Nodes': {'Content': '',
                           'Abs_Address': '/u/c/2/a1â2',
                           'Type': 'section',
                           'Heading': ' Transferred',
                           'Value': u'1\u20132'}
                }

    self.client.record_create(14, new_Node)

    result = self.client.query('SELECT * FROM V where Abs_Address="/u/c/2/a1â2"')
    assert result[0].Abs_Address == '/u/c/2/a1â2'
</code></pre>

<p>I think you may be saving the unicode value as an escaped value and that's where things get tricky.</p>

<p>I don't trust replacing values myself so I usually escape the unicode values I send to orientdb with the following code:</p>

<pre><code>import json
def _escape(string):
    return json.dumps(string)[1:-1]
</code></pre>

<p>The following test would fail because the escaped value won't match the escaped value in the DB so no record will be returned:</p>

<pre><code>def test_test2(self):
    new_Node = {'@Nodes': {'Content': '',
                           'Abs_Address': _escape('/u/c/2/a1â2'),
                           'Type': 'section',
                           'Heading': ' Transferred',
                           'Value': u'1\u20132'}
                }

    self.client.record_create(14, new_Node)

    result = self.client.query('SELECT * FROM V where Abs_Address="%s"' % _escape('/u/c/2/a1â2'))
    assert  result[0].Abs_Address.encode('UTF-8').decode('unicode_escape') == '/u/c/2/a1â2'
</code></pre>

<p>In order to fix this, you have to escape the value twice:</p>

<pre><code>def test_test3(self):
    new_Node = {'@Nodes': {'Content': '',
                           'Abs_Address': _escape('/u/c/2/a1â2'),
                           'Type': 'section',
                           'Heading': ' Transferred',
                           'Value': u'1\u20132'}
                }

    self.client.record_create(14, new_Node)

    result = self.client.query('SELECT * FROM V where Abs_Address="%s"' % _escape(_escape('/u/c/2/a1â2')))
    assert  result[0].Abs_Address.encode('UTF-8').decode('unicode_escape') == '/u/c/2/a1â2'
</code></pre>

<p>This test will succeed because you will now be asking for the escaped value in the DB.</p>

python string unicode orientdb pyorient