com.sleepycat.je.incomp

Class INCompressor

java.lang.Object

com.sleepycat.je.utilint.DaemonThread

com.sleepycat.je.incomp.INCompressor

All Implemented Interfaces:

DaemonRunner, ExceptionListenerUser, java.lang.Runnable

public class INCompressor
extends DaemonThread

JE compression consists of removing deleted entries from BINs, and pruning empty IN/BINs from the tree which is also called a reverse split. One of the reasons compression is treated specially is that slot compression cannot be performed inline as part of a delete operation. When we delete an LN, a cursor is always present on the LN. The API dictates that the cursor will remain positioned on the deleted record. In addition, if the deleting transaction aborts we must restore the slot and the possibility of a split during an abort is something we wish to avoid; for this reason, compression will not occur if the slot's LSN is locked. In principle, slot compression could be performed during transaction commit, but that would be expensive because a Btree lookup would be required, and this would negatively impact operation latency. For all these reasons, slot compression is performed after the delete operation is complete and committed, and not in the thread performing the operation or transaction commit. Compression is of two types: + "Queued compression" is carried out by the INCompressor daemon thread. Both slot compression and pruning are performed. + "Lazy compression" is carried out as part of logging a BIN by certain operations (namely checkpointing and eviction). Only slot compression is performed by lazy compression, not pruning. The use of BINDeltas has a big impact on slot compression because slots cannot be compressed until we know that a full BIN will next be logged. If a slot were compressed prior to logging a BINDelta, the record of the compression would be lost and the slot would "reappear" when the BIN is reconstituted; therefore, this is not permitted. Queued compression prior to logging a BINDelta is also wasteful because the dequeued entry cannot be processed. Therefore, lazy compression is used when a BINDelta will next be logged, and queued compression is used only when a full BIN will next be logged. Because, in general, BINDeltas are logged more often than BINs, lazy compression is used for slot compression more often than queued compression. You may wonder, since lazy compression is used most of the time for slot compression, why use queued compression for slot compression at all? Queued compression is useful for slot compression for the following reasons: + When a cursor is on a BIN, queuing has an advantage over lazy compression. If we can't compress during logging because of a cursor, we have to log anyway, and we must delay compression and log again later. If we can't compress when processing a queue entry, we requeue it and retry later, which increases the chances that we will be able to compress before logging. + The code to process a queue entry must do slot compression anyway, even if we only want to prune the BIN. We have to account for the case where all slots are deleted but not yet compressed. So the code to process the queue entry could not be simplified even if we were to decide not to queue entries for slot compression. + Because BINDeltas are not used for DeferredWrite mode, queued compression is much more appropriate and efficient in this mode. The mainstream algorithm for compression is as follows. 1. When a delete operation occurs (CursorImpl.delete) we call Locker.addDeleteInfo, which determines whether a BINDelta will next be logged (BIN.shouldLogDelta). If so, it does nothing, meaning that lazy compression will be used. If not (a full BIN will next be logged), it adds a BINReference to the Locker.deleteInfo map. 2. When the operation is successful and the Locker releases its locks, it copies the BINReferences from the deleteInfo map to the compressor queue. For a transaction this happens at commit (Txn.commit). For a non-transaction locker this happens when the cursor moves or is closed (BasicLocker.releaseNonTxnLocks). 3. The INCompressor thread processes its queue entries periodically, based on EnvironmentConfig.COMPRESSOR_WAKEUP_INTERVAL. For each BINReference that was queued, it tries to compress all deleted slots (BIN.compress). If the BIN is then empty, it prunes the Btree (Tree.delete). If a slot cannot be compressed or an empty BIN cannot be pruned because a slot's LSN is locked or a cursor is present, the entry is requeued for retry. 4. Lazy compression occurs via the checkpointer and evictor. When logging an IN, these components pass true for the allowCompress parameter. If a full BIN is logged (BIN.shouldLogDelta returns false), the lazyCompress method is called by BIN.beforeLog. lazyCompress will attempt to compress all deleted slots (BIN.compress). If the BIN is then empty, it will queue a BINReference so that pruning will occur later. If a slot cannot be compressed (because the LSN is locked or a cursor is present), the BIN.afterLog method will queue a BINReference. In this last case, two full BINs will be logged consecutively. Special cases are as follows. A. Before performing a split, we call lazyCompress in order to avoid the split if possible (Tree.searchSubTreeUntilSplit). It is important to avoid splitting when compression is deferred due to BINDeltas. B. When we undo an LN insertion (via abort, rollback or recovery undo in RecoveryManager.undo), or redo an LN deletion during recovery (RecoveryManager.redo), we queue a BINReference if a full BIN will next be logged (BIN.queueSlotDeletion). This mimics what happens during a mainstream delete operation.

Field Summary

Fields inherited from class com.sleepycat.je.utilint.DaemonThread

envImpl, logger, name, nWakeupRequests, stifleExceptionChatter

Constructor Summary

Constructors

Constructor and Description
INCompressor(EnvironmentImpl env, long waitTime, java.lang.String name)

Method Summary

Methods

Modifier and Type	Method and Description
void	addBinRefToQueue(BINReference binRef, boolean doWakeup) Adds the BINReference to the queue if the BIN is not already in the queue, or adds the deleted keys to an existing entry if one exists.
void	addBinToQueue(BIN bin, boolean doWakeup) Adds the BIN to the queue if the BIN is not already in the queue.
void	addMultipleBinRefsToQueue(java.util.Collection<BINReference> binRefs, boolean doWakeup) Adds an entire collection of BINReferences to the queue at once.
void	clearEnv()
void	doCompress() The real work to doing a compress.
boolean	exists(long nodeId)
int	getBinRefQueueSize()
void	lazyCompress(IN in) Lazily compress prior to logging a full version of a BIN; the caller is responsible for ensuring that a full version is likely to be logged next.
StatGroup	loadStats(StatsConfig config) Return stats
protected long	nDeadlockRetries() Return the number of retries when a deadlock exception occurs.
void	onWakeup() onWakeup is synchronized to ensure that multiple invocations of the DaemonThread aren't made.
BIN	searchForBIN(DatabaseImpl db, BINReference binRef) Search the tree for the BIN that corresponds to this BINReference.
void	setBeforeFlushTrackerHook(TestHook hook)
void	verifyCursors()

Methods inherited from class com.sleepycat.je.utilint.DaemonThread

checkErrorListener, createLogger, getExceptionListener, getNWakeupRequests, getThread, isPaused, isRunning, isShutdownRequested, requestShutdown, run, runOrPause, setExceptionListener, shutdown, toString, wakeup

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

INCompressor

public INCompressor(EnvironmentImpl env,
            long waitTime,
            java.lang.String name)

Method Detail

clearEnv

public void clearEnv()

setBeforeFlushTrackerHook

public void setBeforeFlushTrackerHook(TestHook hook)

verifyCursors

public void verifyCursors()
                   throws DatabaseException

Throws:

DatabaseException

getBinRefQueueSize

public int getBinRefQueueSize()

addBinToQueue

public void addBinToQueue(BIN bin,
                 boolean doWakeup)

Adds the BIN to the queue if the BIN is not already in the queue.

addBinRefToQueue

public void addBinRefToQueue(BINReference binRef,
                    boolean doWakeup)

Adds the BINReference to the queue if the BIN is not already in the queue, or adds the deleted keys to an existing entry if one exists.

addMultipleBinRefsToQueue

public void addMultipleBinRefsToQueue(java.util.Collection<BINReference> binRefs,
                             boolean doWakeup)

Adds an entire collection of BINReferences to the queue at once. Use this to avoid latching for each add.

exists

public boolean exists(long nodeId)

loadStats

public StatGroup loadStats(StatsConfig config)

Return stats

nDeadlockRetries

protected long nDeadlockRetries()

Return the number of retries when a deadlock exception occurs.

Overrides:

nDeadlockRetries in class DaemonThread

onWakeup

public void onWakeup()
              throws DatabaseException

Description copied from class: DaemonThread

onWakeup is synchronized to ensure that multiple invocations of the DaemonThread aren't made.

Specified by:

onWakeup in class DaemonThread

Throws:

DatabaseException

doCompress

public void doCompress()
                throws DatabaseException

The real work to doing a compress. This may be called by the compressor thread or programatically.

Throws:

DatabaseException

searchForBIN

public BIN searchForBIN(DatabaseImpl db,
               BINReference binRef)

Search the tree for the BIN that corresponds to this BINReference.

Parameters:

binRef - the BINReference that indicates the bin we want.

Returns:

the BIN that corresponds to this BINReference. The node is latched upon return. Returns null if the BIN can't be found.

lazyCompress

public void lazyCompress(IN in)

Lazily compress prior to logging a full version of a BIN; the caller is responsible for ensuring that a full version is likely to be logged next. Do not do any pruning. The target IN should be latched when we enter, and it will be remain latched. When an LN is deleted and a delta will be logged next (see BIN.shouldLogDelta), we do not add the slot to the compressor queue because compression must be deferred until the full version is logged. Therefore we cannot rely on the compressor to delete all slots and we do the final deferred compression here. Note that we do not bother to delete queue entries for the BIN if compression succeeds. Queue entries are normally removed quickly by the compressor. In the case where queue entries happen to exist when we do the final compression below, we rely on the compressor to clean them up later on when they are processed.