Package edu.caltech.nanodb.storage

Class BufferManager

  • public class BufferManager
extends java.lang.Object
    The buffer manager reduces the number of disk IO operations by managing an in-memory cache of data pages. It also imposes a limit on the maximum amount of space that can be used for data pages in the database. Other components can register with the Buffer Manager (as BufferManagerObservers) to be informed when pages are being evicted.
    To Do:
    Eventually add integrity checks, e.g. to make sure every cached page's file appears in the collection of cached files.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      private class  BufferManager.BufferPropertyObserver  
      private static class  BufferManager.CachedPageInfo
      This helper class keeps track of a data page that is currently cached.
      private static class  BufferManager.SessionPinCount
      This helper class records the pin-count of a data page as imposed by a given session, so that we can forcibly release the session's pins after each command the session completes.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      private java.util.LinkedHashMap<java.lang.String,​DBFile> cachedFiles
      This collection holds the DBFile objects corresponding to various opened files the database is currently using.
      private java.util.LinkedHashMap<BufferManager.CachedPageInfo,​DBPage> cachedPages
      This collection holds database pages (not WAL pages) that the database is currently working with, so that they don't continually need to be reloaded.
      private FileManager fileManager  
      private java.lang.Object guard
      A simple object for synchronizing on, so that the buffer manager will be thread-safe.
      private static org.apache.logging.log4j.Logger logger
      A logging object for reporting anything interesting that happens.
      private int maxCacheSize
      This field records the maximum allowed cache size.
      private java.util.ArrayList<BufferManagerObserver> observers  
      private java.lang.String replacementPolicy
      A string indicating the buffer manager's page replacement policy.
      private java.util.HashMap<java.lang.Integer,​java.util.HashMap<DBPageID,​BufferManager.SessionPinCount>> sessionPinCounts
      This collection maps session IDs to the files and pages that each session has pinned, so that we can forcibly unpin pages used by a given session when the session is done with the current command.
      private int totalBytesCached
      This field records how many bytes are currently cached, in total.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void addFile​(DBFile dbFile)  
      void addObserver​(BufferManagerObserver obs)
      Add another observer to the buffer manager.
      byte[] allocBuffer​(int size)
      This method attempts to allocate a buffer of the specified size, possibly evicting some existing buffers in order to make space.
      private void ensureSpaceAvailable​(int bytesRequired)
      This helper function ensures that the buffer manager has the specified amount of space available.
      void flushAll()
      This method removes all cached pages from the buffer manager, writing out any dirty pages in the process.
      void flushDBFile​(DBFile dbFile)
      This method removes all cached pages in the specified file from the buffer manager, writing out any dirty pages in the process.
      DBFile getFile​(java.lang.String filename)
      Retrieves the specified DBFile from the buffer manager, if it has already been opened.
      int getMaxCacheSize()
      Returns the current maximum buffer-cache size in bytes.
      DBPage getPage​(DBFile dbFile, int pageNo, boolean create)
      Retrieves the specified DBPage from the Buffer Manager if it's currently buffered, or null if the page is not currently buffered.
      void recordPageInvalidated​(DBPage dbPage)  
      void recordPagePinned​(DBPage dbPage)
      Records that the page was pinned by the current session.
      void recordPageUnpinned​(DBPage dbPage)
      Records that the page was unpinned by the current session.
      void releaseBuffer​(byte[] buffer)  
      java.util.List<DBFile> removeAll()
      This method removes ALL files from the cache, first flushing all pages from the cache so that any dirty pages will be saved to disk (possibly updating the write-ahead log in the process).
      void removeDBFile​(DBFile dbFile)
      This method removes a file from the cache, first flushing all pages from the file out of the cache.
      void setMaxCacheSize​(int maxCacheSize)
      Sets the maximum buffer-cache size in bytes.
      void unpinAllSessionPages()
      This method unpins all pages pinned by the current session.
      void writeAll​(boolean sync)
      This method writes all dirty pages in the buffer manager to disk.
      void writeDBFile​(DBFile dbFile, boolean sync)
      This method writes all dirty pages in the specified file, optionally syncing the file after performing the write.
      void writeDBFile​(DBFile dbFile, int minPageNo, int maxPageNo, boolean sync)
      This method writes all dirty pages in the specified file, optionally syncing the file after performing the write.
      private void writeDirtyPages​(java.util.List<DBPage> dirtyPages, boolean invalidate)
      This helper method writes out a list of dirty pages from the buffer manager, ensuring that if transactions are enabled, the write-ahead-logging rule is satisfied.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • logger

        private static org.apache.logging.log4j.Logger logger
        A logging object for reporting anything interesting that happens.

      • guard

        private final java.lang.Object guard
        A simple object for synchronizing on, so that the buffer manager will be thread-safe.

      • cachedFiles

        private java.util.LinkedHashMap<java.lang.String,​DBFile> cachedFiles
        This collection holds the DBFile objects corresponding to various opened files the database is currently using.

      • cachedPages

        private java.util.LinkedHashMap<BufferManager.CachedPageInfo,​DBPage> cachedPages
        This collection holds database pages (not WAL pages) that the database is currently working with, so that they don't continually need to be reloaded.

      • sessionPinCounts

        private java.util.HashMap<java.lang.Integer,​java.util.HashMap<DBPageID,​BufferManager.SessionPinCount>> sessionPinCounts
        This collection maps session IDs to the files and pages that each session has pinned, so that we can forcibly unpin pages used by a given session when the session is done with the current command.

      • totalBytesCached

        private int totalBytesCached
        This field records how many bytes are currently cached, in total.

      • maxCacheSize

        private int maxCacheSize
        This field records the maximum allowed cache size.

      • replacementPolicy

        private java.lang.String replacementPolicy
        A string indicating the buffer manager's page replacement policy. Currently it can be "LRU" or "FIFO".

    • Method Detail

      • setMaxCacheSize

        public void setMaxCacheSize​(int maxCacheSize)
        Sets the maximum buffer-cache size in bytes. If in-use buffer allocations exceed this limit then allocBuffer(int) will throw an exception.
        Parameters:
        maxCacheSize - the maximum size for the buffer cache.

      • getMaxCacheSize

        public int getMaxCacheSize()
        Returns the current maximum buffer-cache size in bytes.
        Returns:
        the maximum size for the buffer cache.

      • addObserver

        public void addObserver​(BufferManagerObserver obs)
        Add another observer to the buffer manager.
        Parameters:
        obs - the observer to add to the buffer manager

      • allocBuffer

        public byte[] allocBuffer​(int size)
        This method attempts to allocate a buffer of the specified size, possibly evicting some existing buffers in order to make space.
        Parameters:
        size - the size of the buffer to allocate
        Returns:
        an array of bytes, of the specified size

      • releaseBuffer

        public void releaseBuffer​(byte[] buffer)

      • getFile

        public DBFile getFile​(java.lang.String filename)
        Retrieves the specified DBFile from the buffer manager, if it has already been opened.
        Parameters:
        filename - The filename of the database file to retrieve. This should be ONLY the database filename, no path. The path is expected to be relative to the database's base directory.
        Returns:
        the DBFile corresponding to the filename, if it has already been opened, or null if the file isn't currently open.

      • addFile

        public void addFile​(DBFile dbFile)

      • recordPagePinned

        public void recordPagePinned​(DBPage dbPage)
        Records that the page was pinned by the current session. This method does not actually pin the page; it is presumed that the page is already pinned.
        Parameters:
        dbPage - the page that was pinned by the session

      • recordPageUnpinned

        public void recordPageUnpinned​(DBPage dbPage)
        Records that the page was unpinned by the current session. This method does not actually unpin the page; it is presumed that the page will be unpinned after this call.
        Parameters:
        dbPage - the page that was unpinned

      • unpinAllSessionPages

        public void unpinAllSessionPages()
        This method unpins all pages pinned by the current session. This is generally done at the end of each transaction so that pages aren't pinned forever, and can actually be evicted from the buffer manager.

      • recordPageInvalidated

        public void recordPageInvalidated​(DBPage dbPage)

      • getPage

        public DBPage getPage​(DBFile dbFile,
                      int pageNo,
                      boolean create)
        Retrieves the specified DBPage from the Buffer Manager if it's currently buffered, or null if the page is not currently buffered. If a page is returned, it is pinned before it is returned.
        Parameters:
        dbFile - the file containing the page to retrieve
        pageNo - the page number in the DBFile to retrieve
        Returns:
        the requested DBPage, or null if not found

      • ensureSpaceAvailable

        private void ensureSpaceAvailable​(int bytesRequired)

        This helper function ensures that the buffer manager has the specified amount of space available. This is done by looking for pages that may be evicted from the cache, until the requested amount of space is available. Of course, if the requested amount of space is already available, this function returns immediately.

        Pinned pages are not considered to be eligible for eviction.

        Parameters:
        bytesRequired - the amount of space that should be made available in the cache, in bytes

      • writeDirtyPages

        private void writeDirtyPages​(java.util.List<DBPage> dirtyPages,
                             boolean invalidate)
        This helper method writes out a list of dirty pages from the buffer manager, ensuring that if transactions are enabled, the write-ahead-logging rule is satisfied.
        Parameters:
        dirtyPages - the list of dirty pages to write
        invalidate - if true then the dirty pages are invalidated so they must be reloaded from disk

      • writeDBFile

        public void writeDBFile​(DBFile dbFile,
                        int minPageNo,
                        int maxPageNo,
                        boolean sync)
        This method writes all dirty pages in the specified file, optionally syncing the file after performing the write. The pages are not removed from the buffer manager after writing them; their dirty state is simply cleared.
        Parameters:
        dbFile - the file whose dirty pages should be written to disk
        minPageNo - dirty pages with a page-number less than this value will not be written to disk
        maxPageNo - dirty pages with a page-number greater than this value will not be written to disk
        sync - If true then the database file will be sync'd to disk; if false then no sync will occur. The sync will always occur, in case dirty pages had previously been flushed to disk without syncing.

      • writeDBFile

        public void writeDBFile​(DBFile dbFile,
                        boolean sync)
        This method writes all dirty pages in the specified file, optionally syncing the file after performing the write. The pages are not removed from the buffer manager after writing them; their dirty state is simply cleared.
        Parameters:
        dbFile - the file whose dirty pages should be written to disk
        sync - If true then the database file will be sync'd to disk; if false then no sync will occur. The sync will always occur, in case dirty pages had previously been flushed to disk without syncing.

      • writeAll

        public void writeAll​(boolean sync)
        This method writes all dirty pages in the buffer manager to disk. The pages are not removed from the buffer manager after writing them; their dirty state is simply cleared.
        Parameters:
        sync - if true, this method will sync all files in which dirty pages were found, with the exception of WAL files and the transaction-state file. If false, no file syncing will be performed.

      • flushDBFile

        public void flushDBFile​(DBFile dbFile)
        This method removes all cached pages in the specified file from the buffer manager, writing out any dirty pages in the process. This method is not generally recommended to be used, as it basically defeats the purpose of the buffer manager in the first place; rather, the writeDBFile(edu.caltech.nanodb.storage.DBFile, int, int, boolean) method should be used instead. There is a specific situation in which it is used, when a file is being removed from the Buffer Manager by the Storage Manager.
        Parameters:
        dbFile - the file whose pages should be flushed from the cache

      • flushAll

        public void flushAll()
        This method removes all cached pages from the buffer manager, writing out any dirty pages in the process. This method is not generally recommended to be used, as it basically defeats the purpose of the buffer manager in the first place; rather, the writeAll(boolean) method should be used instead. However, this method is useful to cause certain performance issues to manifest with individual commands, and the Storage Manager also uses it during shutdown processing to ensure all data is saved to disk.

      • removeDBFile

        public void removeDBFile​(DBFile dbFile)
        This method removes a file from the cache, first flushing all pages from the file out of the cache. This operation is used by the Storage Manager to close a data file.
        Parameters:
        dbFile - the file to remove from the cache.

      • removeAll

        public java.util.List<DBFile> removeAll()
        This method removes ALL files from the cache, first flushing all pages from the cache so that any dirty pages will be saved to disk (possibly updating the write-ahead log in the process). This operation is used by the Storage Manager during shutdown.
        Returns:
        a list of the files that were in the cache, so that they can be used by the caller if necessary (e.g. to sync and close each one)