Package edu.caltech.nanodb.storage.writeahead

Class WALManager

  • public class WALManager
extends java.lang.Object

    This class manages the write-ahead logs of the database. There are methods to write the different kinds of log records, and also to perform recovery based on the contents of the write-ahead logs. Note that many details of transaction coordination are handled by the Transaction Manager, such as syncing the write-ahead log, forcing the WAL out to a specific log sequence number, and so forth.

    The actual details of the write-ahead log format are in the package Javadocs: edu.caltech.nanodb.storage.writeahead.

    Some of the writeXXXX() methods require explicit transaction details, while others retrieve the transaction state from thread-local storage. The main difference is that methods that require explicit transaction details are needed during recovery processing, when transaction state is dictated by the log file, not what is in thread-local storage.

    Thread Safety

    This class is written to be thread-safe when invoked from multiple transaction threads. Operations that involve reading logs may overlap with each other, since they will always read log entries that have been written earlier by the same thread, and those entries will not change. Operations that involve writing logs always write each log-entry atomically so that individual records cannot be interleaved with each other; but, the records from two transactions may of course be interleaved.

    Recovery processing is written to occur atomically, but the reality is that recovery processing always occurs from a single thread when the database starts up, so no thread-safety is required.

    • Field Detail

      • logger

        private static org.apache.logging.log4j.Logger logger
        A logging object for reporting anything interesting that happens.

      • WAL_FILENAME_PATTERN

        private static final java.lang.String WAL_FILENAME_PATTERN
        Write-ahead log files follow this pattern.
        See Also:
        Constant Field Values

      • MAX_WAL_FILE_NUMBER

        public static final int MAX_WAL_FILE_NUMBER
        Maximum file number for a write-ahead log file.
        See Also:
        Constant Field Values

      • MAX_WAL_FILE_SIZE

        private static final int MAX_WAL_FILE_SIZE
        Maximum size of a write-ahead log file is 10MB. When the current WAL file reaches this size, it is closed and a new WAL file is created with the next increasing file number.
        See Also:
        Constant Field Values
        To Do:
        Make this a configuration propery

      • OFFSET_PREV_FILE_END

        private static final int OFFSET_PREV_FILE_END
        This is the file-offset just past the last byte written in the previous WAL file, or 0 for the first WAL file. The value is an unsigned 32-bit integer.
        See Also:
        Constant Field Values

      • OFFSET_FIRST_RECORD

        public static final int OFFSET_FIRST_RECORD
        This is the file-offset of the first log entry in a WAL file.
        See Also:
        Constant Field Values

      • firstLSN

        private LogSequenceNumber firstLSN
        This object holds the log sequence number of the first write-ahead log record where recovery would need to start from.

      • nextLSN

        private LogSequenceNumber nextLSN
        This object holds the log sequence number where the next write-ahead log record will be written.

      • guard

        private final java.lang.Object guard
        A simple object for synchronizing on, so that the WAL manager will be thread-safe.

    • Constructor Detail

    • Method Detail

      • getWALFileName

        public static java.lang.String getWALFileName​(int fileNo)
        This static helper method simply takes a WAL file number and translates it into a corresponding filename based on that number.
        Parameters:
        fileNo - the WAL file number to get the filename for
        Returns:
        the string file-name for the corresponding WAL file

      • createWALFile

        private DBFile createWALFile​(int fileNo)
        This helper method creates a brand new write-ahead log file using the Storage Manager, generating a suitable filename and passing the appropriate arguments to the Storage Manager.
        Parameters:
        fileNo - the number of the WAL file to create
        Returns:
        a DBFile for the newly created and opened WAL file
        Throws:
        StorageException - if the file cannot be created for some reason

      • openWALFile

        private DBFile openWALFile​(int fileNo)
        This helper method opens an existing write-ahead log file using the Storage Manager, generating a suitable filename and passing the appropriate arguments to the Storage Manager. Of course, if the file has already been opened, the Storage Manager will return the DBFile from its cache.
        Parameters:
        fileNo - the number of the WAL file to open
        Returns:
        a DBFile for the opened WAL file
        Throws:
        StorageException - if the file cannot be opened for some reason, such as if the file doesn't exist, or if permissions are incorrect

      • doRecovery

        public RecoveryInfo doRecovery​(LogSequenceNumber storedFirstLSN,
                               LogSequenceNumber storedNextLSN)

        Performs recovery processing starting at the specified log sequence number, and returns the LSN where the next recovery process should start from.

        This method is not thread-safe. It is expected that when recovery processing occurs, nothing else will be happening in the database system.

        This function is thread-safe merely for the sake of completeness, since it will always be called from a single thread of execution at start-up. Its approach to thread-safety is simple: the guard lock is held for the entirety of the recovery operation.

        Parameters:
        storedFirstLSN - the location of the write-ahead log record where recovery should start from
        storedNextLSN - the location in the write-ahead log that is just past the last valid log record in the WAL
        Returns:
        the new location where recovery should start from the next time recovery processing is performed
        Throws:
        StorageException - if an IO error occurs during recovery processing

      • performRedo

        private void performRedo​(RecoveryInfo recoveryInfo)

        This helper function performs redo processing using the write-ahead log. As the log is traversed, the RecoveryInfo object is also updated with important redo/undo information.

        This method is not thread-safe. It is expected that when recovery processing occurs, nothing else will be happening in the database system.

        Parameters:
        recoveryInfo - the object used to track information about specific transactions during recovery processing. This object will be passed to performUndo(edu.caltech.nanodb.storage.writeahead.RecoveryInfo).

      • performUndo

        private void performUndo​(RecoveryInfo recoveryInfo)

        This helper function performs undo processing using the write-ahead log. As the log is traversed backwards, the RecoveryInfo object is also updated with important redo/undo information.

        This method is not thread-safe. It is expected that when recovery processing occurs, nothing else will be happening in the database system.

        Parameters:
        recoveryInfo - the object used to track information about specific transactions during recovery processing. This object is populated by a previous call to performRedo(edu.caltech.nanodb.storage.writeahead.RecoveryInfo).

      • computeNextLSN

        public static LogSequenceNumber computeNextLSN​(int fileNo,
                                               int fileOffset)
        This static helper function takes the file number of a WAL file, and the offset in the WAL file where the next write-ahead log record would go if the WAL file can hold more data, and then creates a new LogSequenceNumber object, wrapping to the next file if necessary.
        Parameters:
        fileNo - the number of the current WAL file
        fileOffset - the offset of where the next write-ahead log record would go, in the absence of wrapping to the next file
        Returns:
        a LogSequenceNumber object that takes wrapping into account
        Design Note:
        This is a method on WALManager and not on LogSequenceNumber since the calculation must also consider the format of WAL files as well as their size limit. The class simply doesn't have that information.

      • getWALFileWriter

        private DBFileWriter getWALFileWriter​(LogSequenceNumber lsn)

        This method opens the WAL file specified in the passed-in Log Sequence Number, wraps it with a DBFileWriter so that it can be read and written, and then seeks to the specified file offset.

        Because we are writing, the file may not yet exist. In that case, this method will also create a new WAL file for the specified file number, initializing it with the proper values, and then seeking to the location where the first WAL record may be written.

        By itself, this method is not thread-safe. However, it is definitely called concurrently from multiple threads by code within the WALManager. It is expected that callers of this method have acquired the guard monitor before calling this method.

        Parameters:
        lsn - The log sequence number specifying the WAL file and the offset in the WAL file to go to.
        Returns:
        the WAL file, with the file position moved to the specified offset.
        Throws:
        StorageException - if the corresponding WAL file cannot be opened, or if some other IO error occurs.

      • getWALFileReader

        private DBFileReader getWALFileReader​(LogSequenceNumber lsn,
                                      DBFileReader reader)

        This method opens the WAL file specified in the passed-in Log Sequence Number, wraps it with a DBFileReader so that it can be read from, and then seeks to the specified file offset. Since we are reading, the expectation is that the file already exists.

        By itself, this method is not thread-safe. However, it is definitely called concurrently from multiple threads by code within the WALManager. It is expected that callers of this method have acquired the guard monitor before calling this method.

        Parameters:
        lsn - The log sequence number specifying the WAL file and the offset in the WAL file to go to.
        Returns:
        the WAL file, with the file position moved to the specified offset.

      • writeUpdatePageRecord

        public void writeUpdatePageRecord​(DBPage dbPage)
        This method writes an update-page record to the write-ahead log, including both undo and redo details.
        Parameters:
        dbPage - The data page whose changes are to be recorded in the log.
        Throws:
        java.lang.IllegalArgumentException - if dbPage is null, or if it shows no updates.

      • applyRedo

        private void applyRedo​(WALRecordType type,
                       DBFileReader walReader,
                       DBPage dbPage,
                       int numSegments)
        This helper function writes a sequence of redo-segments from an WALRecordType.UPDATE_PAGE or WALRecordType.UPDATE_PAGE_REDO_ONLY record. Note that the walReader argument is expected to be positioned at the start of the segments containing the old and new versions of the page data (or just the new versions, for redo-only records). Additionally, the reader position will be advanced by this method.
        Parameters:
        type - The record type, either WALRecordType.UPDATE_PAGE or WALRecordType.UPDATE_PAGE_REDO_ONLY.
        walReader - A reader positioned at the start of the redo/undo data to apply to the data page. This method will advance the reader's position past this redo/undo data.
        dbPage - the page that the redo should be applied to
        numSegments - the number of segments containing redo[/undo] data; this value is expected to already be unpacked from the log record

      • applyUndoAndGenRedoOnlyData

        private byte[] applyUndoAndGenRedoOnlyData​(DBFileReader walReader,
                                           DBPage dbPage,
                                           int numSegments)
        This helper method uses a WALRecordType.UPDATE_PAGE record to undo changes to a data page, and at the same time the method generates the data that must go into a corresponding redo-only WAL record.
        Parameters:
        walReader - A reader positioned at the start of the redo/undo data to apply to the data page. This method will advance the reader's position past this redo/undo data.
        dbPage - the data page that undo operations should be applied to
        numSegments - the number of segments in the redo/undo data
        Returns:
        a byte-array containing the same number of segments as the original update record, with only the data necessary for the redo-only record.

      • writeRedoOnlyUpdatePageRecord

        private LogSequenceNumber writeRedoOnlyUpdatePageRecord​(int transactionID,
                                                        LogSequenceNumber prevLSN,
                                                        DBPage dbPage,
                                                        int numSegments,
                                                        byte[] changes)
        This method writes a redo-only update-page record to the write-ahead log, including only redo details. The transaction state is passed explicitly so that this method can be used during recovery processing. The alternate method writeRedoOnlyUpdatePageRecord(DBPage, int, byte[]) retrieves the transaction state from thread-local storage, and should be used during normal operation.
        Parameters:
        transactionID - the transaction ID that the WAL record is for.
        prevLSN - the log sequence number of the transaction's immediately previous WAL record.
        dbPage - The data page whose changes are to be recorded in the log.
        numSegments - The number of segments in the change-data to record.
        changes - The actual changes themselves, serialized to a byte array.
        Returns:
        the Log Sequence Number of the WAL record that was written
        Throws:
        java.lang.IllegalArgumentException - if dbPage is null, or if changes is null.

      • writeRedoOnlyUpdatePageRecord

        private LogSequenceNumber writeRedoOnlyUpdatePageRecord​(DBPage dbPage,
                                                        int numSegments,
                                                        byte[] changes)
        This method writes a redo-only update-page record to the write-ahead log, including only redo details. The transaction state is taken from thread-local storage; this method should be used during normal operation. The alternate method writeRedoOnlyUpdatePageRecord(int, LogSequenceNumber, DBPage, int, byte[]) should be used during recovery processing when transaction state is specifically known.
        Parameters:
        dbPage - The data page whose changes are to be recorded in the log.
        numSegments - The number of segments in the change-data to record.
        changes - The actual changes themselves, serialized to a byte array.
        Returns:
        the Log Sequence Number of the WAL record that was written
        Throws:
        java.lang.IllegalArgumentException - if dbPage is null, or if changes is null.

      • rollbackTransaction

        public void rollbackTransaction()
        This method performs the operations necessary to rollback the current transaction from the database. The transaction details are taken from the transaction state stored in thread-local storage. This method is not used during recovery processing; the performUndo(edu.caltech.nanodb.storage.writeahead.RecoveryInfo) method is used to rollback all incomplete transactions in the logs.