/L8Again

Musings about distributed data infrastructure, cloud, high performance computing, and a bit of philosophy.

I am interested in high performance computing in general, and more specifically distributed systems and cloud services (admittedly biased towards AWS). I work in the data infrastructure team of a social media company that handles user generated content (UGC) for major retailer sites.

Deleting hints in a Cassandra node

December 28, 2014  /  Fahd Siddiqui

This is primarily to remind myself of an often useful command of clearing hints in my cassandra node when faced with a "stop the world" kind of scenario. To be clear, always be sure to repair the ring before proceeding to clear hints on any node. Also, this should be an exception and not a norm. With those disclaimers out of the way, here goes:

The "hints" column family is where the hints are stored in Cassandra 1.2.x and onwards. Each row key is an endpoint with pending hints. 

cassandra-cli -h <host>

use system;
del hints['a095ce27-c240-4937-8b06-f744510c528f'];
row removed.
Elapsed time: 17 msec(s).

list hints;
Using default limit of 100
Using default cell limit of 100
-------------------
RowKey: a095ce27-c240-4937-8b06-f744510c528f

1 Row Returned.
Elapsed time: 1707 msec(s).

Notice that Cassandra still shows the row key that we just deleted. This is because it is marked with a tombstone and therefore we still see the key but no columns are returned. Also, the listEndpointsPendingHints() of HintedHandoffManager will still return 1, until the hints column family is compacted. To force a major compaction, run the following:

nodetool flush system hints && nodetool compact system hints
Comment