Besoin d'aide pour créer une classe de chargeur GAE Datastore?

Question

Besoin d'aide pour créer une classe de chargeur de banque de données GAE pour télécharger des données à l'aide de appcfg.py? Toute autre façon de simplifier ce processus? est-il un exemple détaillé mieux que here

Quand essayez d'utiliser bulkloader.yaml:

Uploading data records. 
[INFO ] Logging to bulkloader-log-20100701.041515 
[INFO ] Throttling transfers: 
[INFO ] Bandwidth: 250000 bytes/second 
[INFO ] HTTP connections: 8/second 
[INFO ] Entities inserted/fetched/modified: 20/second 
[INFO ] Batch Size: 10 
[INFO ] Opening database: bulkloader-progress-20100701.041515.sql3 
[INFO ] Connecting to livelihoodproducer.appspot.com/remote_api 
[INFO ] Starting import; maximum 10 entities per post 
[ERROR ] [Thread-1] WorkerThread: 
Traceback (most recent call last): 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 150, in WorkOnItems 
    status, instruction = item.PerformWork(self.__thread_pool) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 693, in PerformWork 
    transfer_time = self._TransferItem(thread_pool) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 848, in _TransferItem 
    self.content = self.request_manager.EncodeContent(self.rows) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1269, in EncodeContent 
    entity = loader.create_entity(values, key_name=key, parent=parent) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 385, in create_entity 
    return self.dict_to_entity(input_dict, self.bulkload_state) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 133, in dict_to_entity 
    self.__run_import_transforms(input_dict, instance, bulkload_state_copy) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 233, in __run_import_transforms 
    value = self.__dict_to_prop(transform, input_dict, bulkload_state) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 188, in __dict_to_prop 
    value = transform.import_transform(value) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_parser.py", line 93, in __call__ 
    return self.method(*args, **kwargs) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/transform.py", line 143, in generate_foreign_key_lambda 
    return datastore.Key.from_path(kind, value) 
    File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_types.py", line 387, in from_path 
    'received %r (a %s).' % (i + 2, id_or_name, typename(id_or_name))) 
BadArgumentError: Expected an integer id or string name as argument 2; received None (a NoneType). 
[INFO ] [Thread-3] Backing off due to errors: 1.0 seconds 
[INFO ] Unexpected thread death: Thread-1 
[INFO ] An error occurred. Shutting down... 
[ERROR ] Error in Thread-1: Expected an integer id or string name as argument 2; received None (a NoneType). 

[INFO ] 30 entites total, 0 previously transferred 
[INFO ] 0 entities (733 bytes) transferred in 2.8 seconds 
[INFO ] Some entities not successfully transferred 

Dans le processus, j'ai downloadeded données csv insérées manuellement sur appspot.com. Pendant que j'essaye de télécharger mes propres données de csv, l'ordre de colonne devrait faire exactement comme CSV téléchargé à partir de appspot.com? que diriez-vous de la valeur vide?

Answer 1

Il semble que vous ayez des propriétés de référence avec des valeurs None, ces valeurs ne sont pas gérées correctement par les helpers de bulkloader.

Answer 2

J'ai créé config.yaml avec la configuration de bulkloader, et j'ai aussi écrit une fonction simple d'assistance pour traiter les références sans. Je ne sais pas pourquoi ce n'est pas fait en aide d'origine.

L'assistant (fichier helpers.py est très simple, il suffit de placer au même répertoire où vous avez placé config.yaml):

from google.appengine.api import datastore 
def create_foreign_key(kind, key_is_id=False): 
    def generate_foreign_key_lambda(value): 
    if value is None: 
     return None 

    if key_is_id: 
     value = int(value) 
    return datastore.Key.from_path(kind, value) 

    return generate_foreign_key_lambda 

Et cela est coupé de mon config.yaml:

python_preamble: 
- import: helpers # this will import our helper 
[other imports] 
... 
- kind: ArticleComment 
    connector: simplexml 
    connector_options: 
    xpath_to_nodes: "/blog/Comments/Comment" 
    style: element_centric 

    property_map: 
    - property: __key__ 
     external_name: key 
     export_transform: transform.key_id_or_name_as_string 

    - property: parent_comment 
     external_name: parent-comment 
     export_transform: transform.key_id_or_name_as_string 
     import_transform: helpers.create_foreign_key('ArticleComment') 
     #     ^^^^^^^ here it is 
     #     use this instead of transform.create_foreign_key