Problema com criação de instantâneo

Question

Estou com problemas quando tento reparar um espaço de chaves:

myuser@host-1:~$ nodetool repair -pr mykeyspace
[2014-11-07 13:33:10,132] Starting repair command #13, repairing 1 ranges for keyspace mykeyspace
[2014-11-07 13:33:20,668] Repair session 3baeeb40-667a-11e4-a6e9-3562b4c30e9e for range (-3651079129967449,-2] failed with error java.io.IOException: Failed during snapshot creation.
[2014-11-07 13:33:20,668] Repair command #13 finished

Quando olho em system.log on host-1 , diz:

ERROR [RepairJobTask:1] 2014-11-07 13:33:20,665 RepairJob.java (line 125) Error occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /X.X.X.33
        at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:81)
        at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:344)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [AntiEntropySessions:12] 2014-11-07 13:33:20,666 RepairSession.java (line 288) [repair #3baeeb40-667a-11e4-a6e9-3562b4c30e9e] session completed with the following error
java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [AntiEntropySessions:12] 2014-11-07 13:33:20,667 CassandraDaemon.java (line 199) Exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        ... 3 more
ERROR [Thread-1607] 2014-11-07 13:33:20,668 StorageService.java (line 2599) Repair session 3baeeb40-667a-11e4-a6e9-3562b4c30e9e for range (-3651079129967449,-2] failed with error java.io.IOException: Failed during snapshot creation.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2591)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        ... 1 more
Caused by: java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        ... 3 more

Quando olho no log em X.X.X.33 , não vejo nenhum erro de log.

Alguém que poderia me dar um ponteiro na direção certa?

    

Answer 1

Corrigido. Eu emiti um manual nodetool snapshot mykeyspace on X.X.X.33 e consegui obter um erro melhor no log. Acontece que eu tinha um arquivo de dados faltando, provavelmente de descartar uma tabela e criá-la mais rápido do que essa alteração de esquema se propagou por todo o cluster. Reiniciei o nó Cassandra (para descobrir que o arquivo estava faltando) e TRUNCATEd a tabela.