Presto S3 AccessDenied lors de l'insertion de données dans le compartiment de chiffrement KMS

Question

J'ai un compartiment sur AWS s3 qui impose à tous les objets d'être chiffrés KMS. Je cours Presto sur emr-5.2.1

J'ai la table externe sur s3 (aucune donnée). Lorsque j'utilise

INSERT INTO hive.s3.new_table 
SELECT * FROM src_table 

Je reçois erreur AccessDenied. J'ai testé quelques options différentes et atteint le support mais sans chance. Si je supprime une stratégie du compartiment, Presto fonctionne correctement, mais les fichiers créés sur s3 ne sont pas cryptés.

Presto n'a aucun problème à lire les tables s3 externes cryptées ou à les créer localement sur hdfs. Je ne peux pas autoriser les données non cryptées.

exemple de la politique:

{ 
    "Version":"2012-10-17", 
    "Id":"PutObjPolicy", 
    "Statement":[{ 
     "Sid":"DenyUnEncryptedObjectUploads", 
     "Effect":"Deny", 
     "Principal":"*", 
     "Action":"s3:PutObject", 
     "Resource":"arn:aws:s3:::YourBucket/*", 
     "Condition":{ 
      "StringNotEquals":{ 
       "s3:x-amz-server-side-encryption":"aws:kms" 
      } 
     } 
     } 
    ] 
} 

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html

config Presto /etc/presto/conf/catalog/hive.properties

hive.s3.ssl.enabled=true 
hive.s3.use-instance-credentials=true 
hive.s3.sse.enabled = true 
hive.s3.kms-key-id = long_key_id_here 

...

Error: 
com.facebook.presto.spi.PrestoException: Error committing write to Hive 
    at com.facebook.presto.hive.HiveRecordWriter.commit(HiveRecordWriter.java:132) 
    at com.facebook.presto.hive.HiveWriter.commit(HiveWriter.java:49) 
    at com.facebook.presto.hive.HivePageSink.doFinish(HivePageSink.java:152) 
    at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23) 
    at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:76) 
    at com.facebook.presto.hive.HivePageSink.finish(HivePageSink.java:144) 
    at com.facebook.presto.spi.classloader.ClassLoaderSafeConnectorPageSink.finish(ClassLoaderSafeConnectorPageSink.java:49) 
    at com.facebook.presto.operator.TableWriterOperator.finish(TableWriterOperator.java:156) 
    at com.facebook.presto.operator.Driver.processInternal(Driver.java:394) 
    at com.facebook.presto.operator.Driver.processFor(Driver.java:301) 
    at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622) 
    at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:534) 
    at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:670) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxx+xxx= 
    at com.facebook.presto.hive.PrestoS3FileSystem$PrestoS3OutputStream.uploadObject(PrestoS3FileSystem.java:1003) 
    at com.facebook.presto.hive.PrestoS3FileSystem$PrestoS3OutputStream.close(PrestoS3FileSystem.java:967) 
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:74) 
    at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:108) 
    at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2429) 
    at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:106) 
    at com.facebook.presto.hive.HiveRecordWriter.commit(HiveRecordWriter.java:129) 
    ... 15 more 
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxx) 
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1387) 
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:940) 
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:715) 
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:466) 
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:427) 
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:376) 
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4039) 
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1583) 
    at com.amazonaws.services.s3.AmazonS3EncryptionClient.access$101(AmazonS3EncryptionClient.java:80) 
    at com.amazonaws.services.s3.AmazonS3EncryptionClient$S3DirectImpl.putObject(AmazonS3EncryptionClient.java:603) 
    at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.putObjectUsingMetadata(S3CryptoModuleBase.java:175) 
    at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.putObjectSecurely(S3CryptoModuleBase.java:161) 
    at com.amazonaws.services.s3.internal.crypto.CryptoModuleDispatcher.putObjectSecurely(CryptoModuleDispatcher.java:108) 
    at com.amazonaws.services.s3.AmazonS3EncryptionClient.putObject(AmazonS3EncryptionClient.java:483) 
    at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131) 
    at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123) 
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139) 
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
    ... 3 more 

Suis-je mi ssing quelque chose dans la configuration ou Presto n'utilise pas KMS lors de l'insertion dans les tables? Selon amazon: "Toutes les requêtes GET et PUT pour un objet protégé par AWS KMS échoueront si elles ne sont pas réalisées via SSL ou en utilisant SigV4."

Answer 1

Presto prend désormais en charge SSE-KMS via la propriété de configuration du connecteur Hive hive.s3.sse.kms-key-id.