Merge branch 'master' of github.com:jonaswinkler/paperless-ng

CONTRIBUTING.md

@@ -1,13 +1,26 @@
 # Contributing
 
-If you feel that somethings is not working, please submit an issue. You can also ask questions on the issue tracker by tagging your question with the question tag.
+There's still lots of things to be done, just have a look at that issue log. If you feel like conctributing to the project, please do! Bug fixes and improvements to the front end (I just can't seem to get some of these CSS things right) are always welcome.
 
-Pull requests are welcome, however, I will be a little bit more strict about what goes into the code and what does not. If you want to make a big change, please ask me about it first.
+If you want to implement something big: Please start a discussion about that in the issues! Maybe I've already had something similar in mind and we can make it happen together. However, keep in mind that the general roadmap is to make the existing features stable and get them tested. See the roadmap in the readme.
 
 * When making additions to the project, consider if the majority of users will benefit from your change. If not, you're probably better of forking the project.
 * Also consider if your change will get in the way of other users. A good change is a change that enhances the experience of some users who want that change and does not affect users who do not care about the change.
 
-However:
+## Python
 
-* Bug fixes and are always welcome. Docker makes things easier, however, I alone cannot ensure that this runs on all platforms.
-* Improvements to the styling of the front-end are always welcome. I'm no expert in things UX, and simply copied one of the Bootstrap examples. I think it turned out rather good, but I just can't seem to get some things working properly.
+Use python 3.6 for development. Paperless supports python 3.6, 3.7 and 3.8.
+
+## Branches
+
+master always reflects the latest release.
+
+dev contains all changes that will be part of the next release. Use this branch to start making your changes.
+
+feature-X branches is for experimental stuff that will eventually be merged into dev, and then released as part of the next release.
+
+## Testing:
+
+I'm trying to get most of paperless tested, so please do the same for your code! I know its a hassle, but it makes sure that your code works now and will allow us to detect regressions easily.
+
+To test your code, execute `pytest` in the src/ directory. Executing that in the project root is no good. This also generates a html coverage report, which you can use to see if you missed anything important during testing.

README.md

@@ -35,43 +35,61 @@ The gist of the changes is the following:
 * New full text search.
 * New email processing.
 * Machine learning powered document matching.
-* Code cleanup in many, MANY areas.
+* A task processor that processes documents in parallel and also tells you when something goes wrong.
+* Code cleanup in many, MANY areas. Some of the code was just overly complicated.
+* More tests, more stability.
 
 If you want to see some screenshots of paperless-ng in action, [some are available in the documentation](https://paperless-ng.readthedocs.io/en/latest/screenshots.html).
 
 For a complete list of changes, check out the [changelog](https://paperless-ng.readthedocs.io/en/latest/changelog.html)
 
-## Planned
+# Roadmap for 1.0
 
-These features will make it into the application at some point, sorted by priority.
+- Make the front end nice (except mobile).
+- Test coverage at 90%.
+- Store archived documents with an embedded OCR text layer, while keeping originals available. Making good progress in the `feature-ocrmypdf` branch.
+- Fix whatever bugs I and you find.
+
+## Roadmap for versions beyond 1.0
 
 - **More search.** The search backend is incredibly versatile and customizable. Searching is the most important feature of this project and thus, I want to implement things like:
   - Group and limit search results by correspondent, show “more from this” links in the results.
   - Ability to search for “Similar documents” in the search results
-  - Provide corrections for mispelled queries
-- **More robust consumer** that shows its progress on the web page.
+- **An interactive consumer** that shows its progress for documents it processes on the web page.
+	- With live updates ans websockets. This already works on a dev branch, but requires a lot of new dependencies, which I'm not particular happy about.
+	- Notifications when a document was added with buttons to open the new document right away.
 - **Arbitrary tag colors**. Allow the selection of any color with a color picker.
 
 ## On the chopping block.
 
-- **GnuPG encrypion.** Since its disabled by default and the website allows transparent access to encrypted documents anyway, this doesn’t really provide any benefit over having the application stored on an encrypted file system.
+- **GnuPG encrypion.** [Here's a note about encryption in paperless](https://paperless-ng.readthedocs.io/en/latest/administration.html#managing-encryption). The gist of it is that I don't see which attacks this implementation protects against. It gives a false sense of security to users who don't care about how it works.
 
 # Getting started
 
-The recommended way to deploy paperless is docker-compose. Grab the latest release to get started. the dockerfiles archive contains just the docker files which will pull the image from docker hub. The source archive contains everything you need to build the docker image yourself.
+The recommended way to deploy paperless is docker-compose. Don't clone the repository, grab the latest release to get started instead. The dockerfiles archive contains just the docker files which will pull the image from docker hub. The source archive contains everything you need to build the docker image yourself (i.e. if you want to run on Raspberry Pi).
 
 Read the [documentation](https://paperless-ng.readthedocs.io/en/latest/setup.html#installation) on how to get started.
 
-Alternatively, you can install the dependencies and setup apache and a database server yourself. Details for that will be available in the documentation at some point.
+Alternatively, you can install the dependencies and setup apache and a database server yourself. The documenation has information about the individual components of paperless that you need to take care of.
 
 # Migrating to paperless-ng
 
-Read the section about [migration](https://paperless-ng.readthedocs.io/en/latest/setup.html#migration-to-paperless-ng) in the documentation.
+Read the section about [migration](https://paperless-ng.readthedocs.io/en/latest/setup.html#migration-to-paperless-ng) in the documentation. Its also entirely possible to go back to paperless by reverting the database migrations.
 
 # Documentation
 
 The documentation for Paperless-ng is available on [ReadTheDocs](https://paperless-ng.readthedocs.io/).
 
+# Suggestions? Questions? Something not working?
+
+Please open an issue and start a discussion about it!
+
+## Feel like helping out?
+
+There's still lots of things to be done, just have a look at that issue log. If you feel like conctributing to the project, please do! Bug fixes and improvements to the front end (I just can't seem to get some of these CSS things right) are always welcome.
+
+If you want to implement something big: Please start a discussion about that in the issues! Maybe I've already had something similar in mind and we can make it happen together. However, keep in mind that the general roadmap is to make the existing features stable and get them tested. See the roadmap above.
+
 # Affiliated Projects
 
 Paperless has been around a while now, and people are starting to build stuff on top of it.  If you're one of those people, we can add your project to this list:

docker/hub/docker-compose.postgres.yml

@@ -15,7 +15,7 @@ services:
       POSTGRES_PASSWORD: paperless
 
   webserver:
-    image: jonaswinkler/paperless-ng:0.9.3
+    image: jonaswinkler/paperless-ng:0.9.4
     restart: always
     depends_on:
       - db

docker/hub/docker-compose.sqlite.yml

@@ -5,7 +5,7 @@ services:
     restart: always
 
   webserver:
-    image: jonaswinkler/paperless-ng:0.9.3
+    image: jonaswinkler/paperless-ng:0.9.4
     restart: always
     depends_on:
       - broker

docs/administration.rst

@@ -274,6 +274,7 @@ management command:
 
 This command takes no arguments.
 
+.. _`administration-index`:
 
 Managing the document search index
 ==================================

docs/changelog.rst

@@ -5,6 +5,34 @@
 Changelog
 *********
 
+paperless-ng 0.9.4
+##################
+
+* Searching:
+
+  * Paperless now supports searching by tags, types and dates and correspondents. In order to have this applied to your
+    existing documents, you need to perform a ``document_index reindex`` management command
+    (see :ref:`administration-index`)
+    that adds the data to the search index. You only need to do this once, since the schema of the search index changed.
+    Paperless keeps the index updated after that whenever something changes.
+  * Paperless now has spelling corrections ("Did you mean") for miss-typed queries.
+  * The documentation contains :ref:`information about the query syntax <basic-searching>`.
+
+* Front end:
+
+  * Clickable tags, correspondents and types allow quick filtering for related documents.
+  * Saved views are now editable.
+  * Preview documents directly in the browser.
+  * Navigation from the dashboard to saved views.
+
+* Fixes:
+
+  * A severe error when trying to use post consume scripts.
+  * An error in the consumer that cause invalid messages of missing files to show up in the log.
+
+* The documentation now contains information about bare metal installs and a section about
+  how to setup the development environment.
+
 paperless-ng 0.9.3
 ##################
 
@@ -20,7 +48,7 @@ paperless-ng 0.9.3
     aware of.
   * Issue with the automatic classifier not working with only one tag.
   * A couple issues with the search index being opened to eagerly.
-  
+
 * Added lots of tests for various parts of the application.
 
 paperless-ng 0.9.2

docs/extending.rst

@@ -1,5 +1,120 @@
 .. _extending:
 
+Paperless development
+#####################
+
+This section describes the steps you need to take to start development on paperless-ng.
+
+1.  Check out the source from github. The repository is organized in the following way:
+
+    *   ``master`` always represents the latest release and will only see changes
+        when a new release is made.
+    *   ``dev`` contains the code that will be in the next release.
+    *   ``feature-X`` contain bigger changes that will be in some release, but not
+        necessarily the next one.
+    
+    Apart from that, the folder structure is as follows:
+
+    *   ``docs/`` - Documentation.
+    *   ``src-ui/`` - Code of the front end.
+    *   ``src/`` - Code of the back end.
+    *   ``scripts/`` - Various scripts that help with different parts of development.
+    *   ``docker/`` - Files required to build the docker image.
+
+2.  Install some dependencies.
+
+    *   Python 3.6.
+    *   All dependencies listed in the :ref:`Bare metal route <setup-bare_metal>`
+    *   redis. You can either install redis or use the included scritps/start-redis.sh
+        to use docker to fire up a redis instance.
+
+Back end development
+====================
+
+The backend is a django application. I use PyCharm for development, but you can use whatever
+you want.
+
+Install the python dependencies by performing ``pipenv install --dev`` in the src/ directory.
+This will also create a virtual environment, which you can enter with ``pipenv shell`` or
+execute one-shot commands in with ``pipenv run``.
+
+In ``src/paperless.conf``, enable debug mode.
+
+Configure the IDE to use the src/ folder as the base source folder. Configure the following
+launch configurations in your IDE:
+
+*   python3 manage.py runserver
+*   python3 manage.py qcluster
+*   python3 manage.py consumer
+
+Depending on which part of paperless you're developing for, you need to have some or all of
+them running.
+
+Testing and code style:
+
+*   Run ``pytest`` in the src/ directory to execute all tests. This also generates a HTML coverage
+    report. When runnings test, paperless.conf is loaded as well. However: the tests rely on the default
+    configuration. This is not ideal. But for now, make sure no settings except for DEBUG are overridden when testing.
+*   Run ``pycodestyle`` to test your code for issues with the configured code style settings.
+
+    .. note::
+
+        The line length rule E501 is generally useful for getting multiple source files
+        next to each other on the screen. However, in some cases, its just not possible
+        to make some lines fit, especially complicated IF cases. Append ``  # NOQA: E501``
+        to disable this check for certain lines.
+
+Front end development
+=====================
+
+The front end is build using angular. I use the ``Code - OSS`` IDE for development.
+
+In order to get started, you need ``npm``. Install the Angular CLI interface with
+
+.. code:: shell-session
+
+    $ npm install -g @angular/cli
+
+and make sure that it's on your path. Next, in the src-ui/ directory, install the
+required dependencies of the project.
+
+.. code:: shell-session
+
+    $ npm install
+
+You can launch a development server by running
+
+.. code:: shell-session
+
+    $ ng serve
+
+This will automatically update whenever you save. However, in-place compilation might fail
+on syntax errors, in which case you need to restart it.
+
+By default, the development server is available on ``http://localhost:4200/`` and is configured
+to access the API at ``http://localhost:8000/api/``, which is the default of the backend.
+If you enabled DEBUG on the back end, several security overrides for allowed hosts, CORS and
+X-Frame-Options are in place so that the front end behaves exactly as in production. This also
+relies on you being logged into the back end. Without a valid session, The front end will simply
+not work.
+
+In order to build the front end and serve it as part of django, execute
+
+.. code:: shell-session
+
+    $ ng build --prod --output-path ../src/documents/static/frontend/
+
+This will build the front end and put it in a location from which the Django server will serve
+it as static content. This way, you can verify that authentication is working.
+
+Making a release
+================
+
+Execute the ``make-release.sh <ver>`` script.
+
+This will test and assemble everything and also build and tag a docker image.
+
+
 Extending Paperless
 ===================
 

docs/setup.rst

@@ -204,13 +204,152 @@ Docker Route
    simplifies deployment immensely. If you know your way around Docker, feel
    free to tinker around without using compose!
 
+.. _`setup-bare_metal`:
 
 Bare Metal Route
 ================
 
-.. warning::
+Paperless runs on linux only. The following procedure has been tested on a minimal
+installation of Debian/Buster, which is the current stable release at the time of
+writing. Windows is not and will never be supported.
 
-    TBD. User docker for now.
+1.  Install dependencies. Paperless requires the following packages.
+
+    *   ``python3`` 3.6, 3.7, 3.8 (3.9 is untested).
+    *   ``python3-pip``, optionally ``pipenv`` for package installation
+    *   ``python3-dev``
+
+    *   ``imagemagick`` >= 6 for PDF conversion
+    *   ``unpaper`` for cleaning documents before OCR
+    *   ``ghostscript``
+    *   ``optipng`` for optimising thumbnails
+    *   ``tesseract-ocr`` >= 4.0.0 for OCR
+    *   ``tesseract-ocr`` language packs (``tesseract-ocr-eng``, ``tesseract-ocr-deu``, etc)
+    *   ``gnupg`` for handling encrypted documents
+    *   ``libpoppler-cpp-dev`` for PDF to text conversion
+    *   ``libmagic-dev`` for mime type detection
+    *   ``libpq-dev`` for PostgreSQL
+
+    You will also need ``build-essential``, ``python3-setuptools`` and ``python3-wheel``
+    for installing some of the python dependencies. You can remove that
+    again after installation.
+
+2.  Install ``redis`` >= 5.0 and configure it to start automatically.
+
+3.  Optional. Install ``postgresql`` and configure a database, user and password for paperless. If you do not wish
+    to use PostgreSQL, SQLite is avialable as well.
+
+4.  Get the release archive. If you pull the git repo as it is, you also have to compile the front end by yourself.
+    Extract the frontend to a place from where you wish to execute it, such as ``/opt/paperless``.
+
+5.  Configure paperless. See :ref:`configuration` for details. Edit the included ``paperless.conf`` and adjust the
+    settings to your needs. Required settings for getting paperless running are:
+
+    *   ``PAPERLESS_REDIS`` should point to your redis server, such as redis://localhost:6379.
+    *   ``PAPERLESS_DBHOST`` should be the hostname on which your PostgreSQL server is running. Do not configure this
+        to use SQLite instead. Also configure port, database name, user and password as necessary.
+    *   ``PAPERLESS_CONSUMPTION_DIR`` should point to a folder which paperless should watch for documents. You might
+        want to have this somewhere else. Likewise, ``PAPERLESS_DATA_DIR`` and ``PAPERLESS_MEDIA_ROOT`` define where
+        paperless stores its data. If you like, you can point both to the same directory.
+    *   ``PAPERLESS_SECRET_KEY`` should be a random sequence of characters. It's used for authentication. Failure
+        to do so allows third parties to forge authentication credentials.
+    
+    Many more adjustments can be made to paperless, especially the OCR part. The following options are recommended
+    for everyone:
+
+    *   Set ``PAPERLESS_OCR_LANGUAGE`` to the language most of your documents are written in.
+    *   Set ``PAPERLESS_TIME_ZONE`` to your local time zone.
+
+6.  Setup permissions. Create a system users under which you wish to run paperless. Ensure that these directories exist
+    and that the user has write permissions to the following directories
+    
+    *   ``/opt/paperless/media``
+    *   ``/opt/paperless/data``
+    *   ``/opt/paperless/consume``
+
+    Adjust as necessary if you configured different folders.
+
+7.  Install python requirements. Paperless comes with both Pipfiles for ``pipenv`` as well as with a ``requirements.txt``.
+    Both will install exactly the same requirements. It is up to you if you wish to use a virtual environment or not.
+
+8.  Go to ``/opt/paperless/src``, and execute the following commands:
+
+    .. code:: bash
+
+        # This collects static files from paperless and django.
+        python3 manage.py collectstatic --clear --no-input
+        
+        # This creates the database schema.
+        python3 manage.py migrate
+
+        # This creates your first paperless user
+        python3 manage.py createsuperuser
+
+9.  Optional: Test that paperless is working by executing
+
+      .. code:: bash
+
+        # This collects static files from paperless and django.
+        python3 manage.py runserver
+    
+    and pointing your browser to http://localhost:8000/.
+
+    .. warning::
+
+        This is a development server which should not be used in
+        production.
+
+    .. hint::
+
+        This will not start the consumer. Paperless does this in a
+        separate process.
+
+10. Setup systemd services to run paperless automatically. You may
+    use the service definition files included in the ``scripts`` folder
+    as a starting point.
+
+    Paperless needs the ``webserver`` script to run the webserver, the
+    ``consumer`` script to watch the input folder, and the ``scheduler``
+    script to run tasks such as email checking and document consumption.
+
+    These services rely on redis and optionally the database server, but
+    don't need to be started in any particular order. The example files
+    depend on redis being started. If you use a database server, you should
+    add additinal dependencies.
+
+    .. hint::
+
+        You may optionally set up your preferred web server to serve
+        paperless as a wsgi application directly instead of running the
+        ``webserver`` service. The module containing the wsgi application
+        is named ``paperless.wsgi``.
+
+    .. caution::
+
+        The included scripts run a ``gunicorn`` standalone server,
+        which is fine for running paperless. It does support SSL,
+        however, the documentation of GUnicorn states that you should
+        use a proxy server in front of gunicorn instead.
+
+11. Optional: Install a samba server and make the consumption folder
+    available as a network share.
+
+12. Configure ImageMagick to allow processing of PDF documents. Most distributions have
+    this disabled by default, since PDF documents can contain malware. If
+    you don't do this, paperless will fall back to ghostscript for certain steps
+    such as thumbnail generation.
+
+    Edit ``/etc/ImageMagick-6/policy.xml`` and adjust
+
+    .. code::
+
+        <policy domain="coder" rights="none" pattern="PDF" />
+    
+    to
+
+    .. code::
+
+        <policy domain="coder" rights="read|write" pattern="PDF" />
 
 Migration to paperless-ng
 #########################

docs/usage_overview.rst

@@ -156,6 +156,62 @@ REST API
 
 You can also submit a document using the REST API, see :ref:`api-file_uploads` for details.
 
+.. _basic-searching:
+
+Searching
+#########
+
+Paperless offers an extensive searching mechanism that is designed to allow you to quickly
+find a document you're looking for (for example, that thing that just broke and you bought
+a couple months ago, that contract you signed 8 years ago).
+
+When you search paperless for a document, it tries to match this query against your documents.
+Paperless will look for matching documents by inspecting their content, title, correspondent,
+type and tags. Paperless returns a scored list of results, so that documents matching your query
+better will appear further up in the search results.
+
+By default, paperless returns only documents which contain all words typed in the search bar.
+However, paperless also offers advanced search syntax if you want to drill down the results
+further.
+
+Matching documents with logical expressions:
+
+.. code:: none
+
+  shopname AND (product1 OR product2)
+
+Matching specific tags, correspondents or types:
+
+.. code:: none
+
+  type:invoice tag:unpaid
+  correspondent:university certificate
+
+Matching dates:
+
+.. code:: none
+  
+  created:[2005 to 2009]
+  added:yesterday
+  modified:today
+
+Matching inexact words:
+
+.. code:: none
+
+  produ*name
+
+.. note::
+
+  Inexact terms are hard for search indexes. These queries might take a while to execute. That's why paperless offers
+  auto complete and query correction.
+
+All of these constructs can be combined as you see fit.
+If you want to learn more about the query language used by paperless, paperless uses Whoosh's default query language. 
+Head over to `Whoosh query language <https://whoosh.readthedocs.io/en/latest/querylang.html>`_.
+For details on what date parsing utilities are available, see
+`Date parsing <https://whoosh.readthedocs.io/en/latest/dates.html#parsing-date-queries>`_.
+ 
 
 .. _usage-recommended_workflow:
 

scripts/make-release.sh

@@ -42,6 +42,7 @@ fi
 mkdir "$PAPERLESS_DIST"
 mkdir "$PAPERLESS_DIST_APP"
 mkdir "$PAPERLESS_DIST_APP/docker"
+mkdir "$PAPERLESS_DIST_APP/scripts"
 mkdir "$PAPERLESS_DIST_DOCKERFILES"
 
 # setup dependencies.
@@ -104,6 +105,11 @@ cp "$PAPERLESS_ROOT/docker/gunicorn.conf.py" "$PAPERLESS_DIST_APP/docker/"
 cp "$PAPERLESS_ROOT/docker/imagemagick-policy.xml" "$PAPERLESS_DIST_APP/docker/"
 cp "$PAPERLESS_ROOT/docker/supervisord.conf" "$PAPERLESS_DIST_APP/docker/"
 
+# auxiliary files for bare metal installs
+cp "$PAPERLESS_ROOT/scripts/paperless-webserver.service" "$PAPERLESS_DIST_APP/scripts/"
+cp "$PAPERLESS_ROOT/scripts/paperless-consumer.service" "$PAPERLESS_DIST_APP/scripts/"
+cp "$PAPERLESS_ROOT/scripts/paperless-scheduler.service" "$PAPERLESS_DIST_APP/scripts/"
+
 # try to make the docker build.
 
 cd "$PAPERLESS_DIST_APP"

scripts/paperless-consumer.service

@@ -1,10 +1,12 @@
 [Unit]
 Description=Paperless consumer
+Requires=redis.service
 
 [Service]
 User=paperless
 Group=paperless
-ExecStart=/home/paperless/project/virtualenv/bin/python /home/paperless/project/src/manage.py document_consumer
+WorkingDirectory=/opt/paperless/src
+ExecStart=python3 manage.py document_consumer
 
 [Install]
 WantedBy=multi-user.target

scripts/paperless-scheduler.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Paperless consumer
+Requires=redis.service
+
+[Service]
+User=paperless
+Group=paperless
+WorkingDirectory=/opt/paperless/src
+ExecStart=python3 manage.py qcluster
+
+[Install]
+WantedBy=multi-user.target

scripts/paperless-webserver.service

@@ -2,11 +2,13 @@
 Description=Paperless webserver
 After=network.target
 Wants=network.target
+Requires=redis.service
 
 [Service]
 User=paperless
 Group=paperless
-ExecStart=/home/paperless/project/virtualenv/bin/gunicorn --pythonpath=/home/paperless/project/src paperless.wsgi -w 2
+WorkingDirectory=/opt/paperless/src
+ExecStart=/opt/paperless/.local/bin/gunicorn paperless.wsgi -w 2 -b 0.0.0.0:8000
 
 [Install]
 WantedBy=multi-user.target

src-ui/src/app/components/common/tag/tag.component.html

@@ -1,2 +1,2 @@
 <span *ngIf="!clickable" class="badge" [style.background]="getColour().value" [style.color]="getColour().textColor">{{tag.name}}</span>
-<a [routerLink]="" *ngIf="clickable" class="badge" [style.background]="getColour().value" [style.color]="getColour().textColor">{{tag.name}}</a>
+<a [routerLink]="" [title]="linkTitle" *ngIf="clickable" class="badge" [style.background]="getColour().value" [style.color]="getColour().textColor">{{tag.name}}</a>

src-ui/src/app/components/common/tag/tag.component.ts

@@ -14,10 +14,10 @@ export class TagComponent implements OnInit {
   tag: PaperlessTag
 
   @Input()
-  clickable: boolean = false
+  linkTitle: string = ""
 
-  @Output()
-  click = new EventEmitter()
+  @Input()
+  clickable: boolean = false
 
   ngOnInit(): void {
   }

src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.html

@@ -1,6 +1,9 @@
 <app-widget-frame [title]="savedView.title">
 
-  <table class="table table-sm table-hover table-borderless">
+  <a header-buttons [routerLink]="" (click)="showAll()">Show all</a>
+
+
+  <table content class="table table-sm table-hover table-borderless">
     <thead>
       <tr>
         <th>Created</th>

src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts

@@ -1,6 +1,8 @@
 import { Component, Input, OnInit } from '@angular/core';
+import { Router } from '@angular/router';
 import { PaperlessDocument } from 'src/app/data/paperless-document';
 import { SavedViewConfig } from 'src/app/data/saved-view-config';
+import { DocumentListViewService } from 'src/app/services/document-list-view.service';
 import { DocumentService } from 'src/app/services/rest/document.service';
 
 @Component({
@@ -10,7 +12,10 @@ import { DocumentService } from 'src/app/services/rest/document.service';
 })
 export class SavedViewWidgetComponent implements OnInit {
 
-  constructor(private documentService: DocumentService) { }
+  constructor(
+    private documentService: DocumentService,
+    private router: Router,
+    private list: DocumentListViewService) { }
   
   @Input()
   savedView: SavedViewConfig
@@ -23,4 +28,9 @@ export class SavedViewWidgetComponent implements OnInit {
     })
   }
 
+  showAll() {
+    this.list.load(this.savedView)
+    this.router.navigate(["documents"])
+  }
+
 }

src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html

@@ -1,4 +1,6 @@
 <app-widget-frame title="Statistics">
-  <p class="card-text">Documents in inbox: {{statistics.documents_inbox}}</p>
-  <p class="card-text">Total documents: {{statistics.documents_total}}</p>
+  <ng-container content>
+    <p class="card-text">Documents in inbox: {{statistics.documents_inbox}}</p>
+    <p class="card-text">Total documents: {{statistics.documents_total}}</p>
+  </ng-container>
 </app-widget-frame>

src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html

@@ -1,6 +1,6 @@
 <app-widget-frame title="Upload new documents">
 
-  <form>
+  <form content>
     <ngx-file-drop 
       dropZoneLabel="Drop documents here or" (onFileDrop)="dropped($event)"
       (onFileOver)="fileOver($event)" (onFileLeave)="fileLeave($event)"

src-ui/src/app/components/dashboard/widgets/widget-frame/widget-frame.component.html

@@ -1,8 +1,12 @@
 <div class="card mb-3 shadow">
   <div class="card-header">
-    <h5 class="card-title mb-0">{{title}}</h5>
+    <div class="d-flex justify-content-between align-items-center">
+      <h5 class="card-title mb-0">{{title}}</h5>
+      <ng-content select ="[header-buttons]"></ng-content>
+    </div>
+    
   </div>
   <div class="card-body text-dark">
-    <ng-content></ng-content>
+    <ng-content select ="[content]"></ng-content>
   </div>
 </div>

src-ui/src/app/components/document-detail/document-detail.component.ts

@@ -1,4 +1,3 @@
-import { DatePipe, formatDate } from '@angular/common';
 import { Component, OnInit } from '@angular/core';
 import { FormControl, FormGroup } from '@angular/forms';
 import { ActivatedRoute, Router } from '@angular/router';
@@ -6,17 +5,14 @@ import { NgbModal } from '@ng-bootstrap/ng-bootstrap';
 import { PaperlessCorrespondent } from 'src/app/data/paperless-correspondent';
 import { PaperlessDocument } from 'src/app/data/paperless-document';
 import { PaperlessDocumentType } from 'src/app/data/paperless-document-type';
-import { TAG_COLOURS, PaperlessTag } from 'src/app/data/paperless-tag';
 import { DocumentListViewService } from 'src/app/services/document-list-view.service';
 import { OpenDocumentsService } from 'src/app/services/open-documents.service';
 import { CorrespondentService } from 'src/app/services/rest/correspondent.service';
 import { DocumentTypeService } from 'src/app/services/rest/document-type.service';
 import { DocumentService } from 'src/app/services/rest/document.service';
-import { TagService } from 'src/app/services/rest/tag.service';
 import { DeleteDialogComponent } from '../common/delete-dialog/delete-dialog.component';
 import { CorrespondentEditDialogComponent } from '../manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component';
 import { DocumentTypeEditDialogComponent } from '../manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component';
-import { TagEditDialogComponent } from '../manage/tag-list/tag-edit-dialog/tag-edit-dialog.component';
 
 @Component({
   selector: 'app-document-detail',
@@ -133,8 +129,8 @@ export class DocumentDetailComponent implements OnInit {
 
   close() {
     this.openDocumentService.closeDocument(this.document)
-    if (this.documentListViewService.viewId) {
-      this.router.navigate(['view', this.documentListViewService.viewId])
+    if (this.documentListViewService.savedViewId) {
+      this.router.navigate(['view', this.documentListViewService.savedViewId])
     } else {
       this.router.navigate(['documents'])
     }

src-ui/src/app/components/document-list/document-card-large/document-card-large.component.html

@@ -7,7 +7,14 @@
       <div class="card-body">
 
         <div class="d-flex justify-content-between align-items-center">
-          <h5 class="card-title">{{document.correspondent ? document.correspondent.name + ': ' : ''}}{{document.title}}<app-tag [tag]="t" *ngFor="let t of document.tags" class="ml-1"></app-tag></h5>
+          <h5 class="card-title">    
+            <ng-container *ngIf="document.correspondent">
+              <a *ngIf="clickCorrespondent.observers.length ; else nolink" [routerLink]="" title="Filter by correspondent" (click)="clickCorrespondent.emit(document.correspondent)" class="font-weight-bold">{{document.correspondent.name}}</a>
+              <ng-template #nolink>{{document.correspondent.name}}</ng-template>:
+            </ng-container>
+            {{document.title}}
+            <app-tag [tag]="t" linkTitle="Filter by tag" *ngFor="let t of document.tags" class="ml-1" (click)="clickTag.emit(t)" [clickable]="clickTag.observers.length"></app-tag>
+          </h5>
           <h5 class="card-title" *ngIf="document.archive_serial_number">#{{document.archive_serial_number}}</h5>
         </div>
         <p class="card-text">
@@ -24,6 +31,13 @@
               </svg>
               Edit
             </a>
+            <a type="button" class="btn btn-sm btn-outline-secondary" [href]="getPreviewUrl()">
+              <svg width="1em" height="1em" viewBox="0 0 16 16" class="bi bi-search" fill="currentColor" xmlns="http://www.w3.org/2000/svg">
+                <path fill-rule="evenodd" d="M10.442 10.442a1 1 0 0 1 1.415 0l3.85 3.85a1 1 0 0 1-1.414 1.415l-3.85-3.85a1 1 0 0 1 0-1.415z"/>
+                <path fill-rule="evenodd" d="M6.5 12a5.5 5.5 0 1 0 0-11 5.5 5.5 0 0 0 0 11zM13 6.5a6.5 6.5 0 1 1-13 0 6.5 6.5 0 0 1 13 0z"/>
+              </svg>
+              View
+            </a>
             <a type="button" class="btn btn-sm btn-outline-secondary" [href]="getDownloadUrl()">
               <svg width="1em" height="1em" viewBox="0 0 16 16" class="bi bi-download" fill="currentColor" xmlns="http://www.w3.org/2000/svg">
                 <path fill-rule="evenodd" d="M.5 9.9a.5.5 0 0 1 .5.5v2.5a1 1 0 0 0 1 1h12a1 1 0 0 0 1-1v-2.5a.5.5 0 0 1 1 0v2.5a2 2 0 0 1-2 2H2a2 2 0 0 1-2-2v-2.5a.5.5 0 0 1 .5-.5z"/>

src-ui/src/app/components/document-list/document-card-large/document-card-large.component.ts

@@ -1,6 +1,7 @@
-import { Component, Input, OnInit } from '@angular/core';
+import { Component, EventEmitter, Input, OnInit, Output } from '@angular/core';
 import { DomSanitizer } from '@angular/platform-browser';
 import { PaperlessDocument } from 'src/app/data/paperless-document';
+import { PaperlessTag } from 'src/app/data/paperless-tag';
 import { DocumentService } from 'src/app/services/rest/document.service';
 
 @Component({
@@ -18,6 +19,12 @@ export class DocumentCardLargeComponent implements OnInit {
   @Input()
   details: any
 
+  @Output()
+  clickTag = new EventEmitter<PaperlessTag>()
+
+  @Output()
+  clickCorrespondent = new EventEmitter<PaperlessDocument>()
+
   ngOnInit(): void {
   }
 
@@ -41,4 +48,8 @@ export class DocumentCardLargeComponent implements OnInit {
   getDownloadUrl() {
     return this.documentService.getDownloadUrl(this.document.id)
   }
+
+  getPreviewUrl() {
+    return this.documentService.getPreviewUrl(this.document.id)
+  }
 }

src-ui/src/app/components/document-list/document-card-small/document-card-small.component.html

@@ -2,26 +2,34 @@
   <div class="card h-100 shadow-sm">
     <div class=" border-bottom doc-img pr-1" [ngStyle]="{'background-image': 'url(' + getThumbUrl() + ')'}">
       <div class="row" *ngFor="let t of document.tags">
-        <app-tag [tag]="t" class="col text-right"></app-tag>
+        <app-tag style="font-size: large;" [tag]="t" class="col text-right" (click)="clickTag.emit(t)" [clickable]="true" linkTitle="Filter by tag"></app-tag>
       </div>
     </div>
-
     
     <div class="card-body p-2">
       <p class="card-text">
-        <span class="font-weight-bold">{{document.correspondent? document.correspondent.name + ': ' : ''}}</span> {{document.title}}
+        <ng-container *ngIf="document.correspondent">
+          <a [routerLink]="" title="Filter by correspondent" (click)="clickCorrespondent.emit(document.correspondent)" class="font-weight-bold">{{document.correspondent.name}}</a>:
+        </ng-container>
+        {{document.title}}
       </p>
     </div>
     <div class="card-footer">
 
       <div class="d-flex justify-content-between align-items-center ml-n2">
         <div class="btn-group">
-          <a routerLink="/documents/{{document.id}}" class="btn btn-sm btn-outline-secondary">
+          <a routerLink="/documents/{{document.id}}" class="btn btn-sm btn-outline-secondary" title="Edit">
             <svg width="1em" height="1em" viewBox="0 0 16 16" class="bi bi-pencil" fill="currentColor" xmlns="http://www.w3.org/2000/svg">
               <path fill-rule="evenodd" d="M12.146.146a.5.5 0 0 1 .708 0l3 3a.5.5 0 0 1 0 .708l-10 10a.5.5 0 0 1-.168.11l-5 2a.5.5 0 0 1-.65-.65l2-5a.5.5 0 0 1 .11-.168l10-10zM11.207 2.5L13.5 4.793 14.793 3.5 12.5 1.207 11.207 2.5zm1.586 3L10.5 3.207 4 9.707V10h.5a.5.5 0 0 1 .5.5v.5h.5a.5.5 0 0 1 .5.5v.5h.293l6.5-6.5zm-9.761 5.175l-.106.106-1.528 3.821 3.821-1.528.106-.106A.5.5 0 0 1 5 12.5V12h-.5a.5.5 0 0 1-.5-.5V11h-.5a.5.5 0 0 1-.468-.325z"/>
             </svg>
           </a>
-          <a [href]="getDownloadUrl()" class="btn btn-sm btn-outline-secondary">
+          <a [href]="getPreviewUrl()" class="btn btn-sm btn-outline-secondary" title="View in browser">
+            <svg width="1em" height="1em" viewBox="0 0 16 16" class="bi bi-search" fill="currentColor" xmlns="http://www.w3.org/2000/svg">
+              <path fill-rule="evenodd" d="M10.442 10.442a1 1 0 0 1 1.415 0l3.85 3.85a1 1 0 0 1-1.414 1.415l-3.85-3.85a1 1 0 0 1 0-1.415z"/>
+              <path fill-rule="evenodd" d="M6.5 12a5.5 5.5 0 1 0 0-11 5.5 5.5 0 0 0 0 11zM13 6.5a6.5 6.5 0 1 1-13 0 6.5 6.5 0 0 1 13 0z"/>
+            </svg>
+          </a>
+          <a [href]="getDownloadUrl()" class="btn btn-sm btn-outline-secondary" title="Download">
             <svg width="1em" height="1em" viewBox="0 0 16 16" class="bi bi-download" fill="currentColor" xmlns="http://www.w3.org/2000/svg">
               <path fill-rule="evenodd" d="M.5 9.9a.5.5 0 0 1 .5.5v2.5a1 1 0 0 0 1 1h12a1 1 0 0 0 1-1v-2.5a.5.5 0 0 1 1 0v2.5a2 2 0 0 1-2 2H2a2 2 0 0 1-2-2v-2.5a.5.5 0 0 1 .5-.5z"/>
               <path fill-rule="evenodd" d="M7.646 11.854a.5.5 0 0 0 .708 0l3-3a.5.5 0 0 0-.708-.708L8.5 10.293V1.5a.5.5 0 0 0-1 0v8.793L5.354 8.146a.5.5 0 1 0-.708.708l3 3z"/>

src-ui/src/app/components/document-list/document-card-small/document-card-small.component.ts

@@ -1,5 +1,6 @@
-import { Component, Input, OnInit } from '@angular/core';
+import { Component, EventEmitter, Input, OnInit, Output } from '@angular/core';
 import { PaperlessDocument } from 'src/app/data/paperless-document';
+import { PaperlessTag } from 'src/app/data/paperless-tag';
 import { DocumentService } from 'src/app/services/rest/document.service';
 
 @Component({
@@ -14,6 +15,12 @@ export class DocumentCardSmallComponent implements OnInit {
   @Input()
   document: PaperlessDocument
 
+  @Output()
+  clickTag = new EventEmitter<PaperlessTag>()
+
+  @Output()
+  clickCorrespondent = new EventEmitter<PaperlessDocument>()
+
   ngOnInit(): void {
   }
 
@@ -24,4 +31,8 @@ export class DocumentCardSmallComponent implements OnInit {
   getDownloadUrl() {
     return this.documentService.getDownloadUrl(this.document.id)
   }
+
+  getPreviewUrl() {
+    return this.documentService.getPreviewUrl(this.document.id)
+  }
 }

src-ui/src/app/components/document-list/document-list.component.html

@@ -21,13 +21,12 @@
       </svg>
     </label>
   </div>
-  <div class="btn-group btn-group-toggle ml-2" ngbRadioGroup [(ngModel)]="docs.sortDirection"
-    *ngIf="!docs.viewId">
+  <div class="btn-group btn-group-toggle ml-2" ngbRadioGroup [(ngModel)]="list.sortDirection">
     <div ngbDropdown class="btn-group">
       <button class="btn btn-outline-primary btn-sm" id="dropdownBasic1" ngbDropdownToggle>Sort by</button>
       <div ngbDropdownMenu aria-labelledby="dropdownBasic1">
-        <button *ngFor="let f of getSortFields()" ngbDropdownItem (click)="setSort(f.field)"
-          [class.active]="docs.sortField == f.field">{{f.name}}</button>
+        <button *ngFor="let f of getSortFields()" ngbDropdownItem (click)="list.sortField = f.field"
+          [class.active]="list.sortField == f.field">{{f.name}}</button>
       </div>
     </div>
     <label ngbButtonLabel class="btn-outline-primary btn-sm">
@@ -43,7 +42,7 @@
       </svg>
     </label>
   </div>
-  <div class="btn-group ml-2" *ngIf="!docs.viewId">
+  <div class="btn-group ml-2">
 
     <button type="button" class="btn btn-sm btn-outline-primary" (click)="showFilter=!showFilter">
       <svg class="toolbaricon" fill="currentColor">
@@ -55,9 +54,13 @@
     <div class="btn-group" ngbDropdown role="group">
       <button class="btn btn-sm btn-outline-primary dropdown-toggle-split" ngbDropdownToggle></button>
       <div class="dropdown-menu" ngbDropdownMenu>
-        <button ngbDropdownItem *ngFor="let config of savedViewConfigService.getConfigs()" (click)="loadViewConfig(config)">{{config.title}}</button>
-        <div class="dropdown-divider" *ngIf="savedViewConfigService.getConfigs().length > 0"></div>
-        <button ngbDropdownItem (click)="saveViewConfig()">Save current view</button>
+        <ng-container *ngIf="!list.savedViewId" >
+          <button ngbDropdownItem *ngFor="let config of savedViewConfigService.getConfigs()" (click)="loadViewConfig(config)">{{config.title}}</button>
+          <div class="dropdown-divider" *ngIf="savedViewConfigService.getConfigs().length > 0"></div>
+        </ng-container>
+        
+        <button ngbDropdownItem (click)="saveViewConfig()" *ngIf="list.savedViewId">Save "{{list.savedViewTitle}}"</button>
+        <button ngbDropdownItem (click)="saveViewConfigAs()">Save as...</button>
       </div>
     </div>
 
@@ -72,16 +75,16 @@
 </div>
 
 <div class="row m-0 justify-content-end">
-  <ngb-pagination [pageSize]="docs.currentPageSize" [collectionSize]="docs.collectionSize" [(page)]="docs.currentPage" [maxSize]="5"
-  [rotate]="true" (pageChange)="reload()" aria-label="Default pagination"></ngb-pagination>
+  <ngb-pagination [pageSize]="list.currentPageSize" [collectionSize]="list.collectionSize" [(page)]="list.currentPage" [maxSize]="5"
+  [rotate]="true" (pageChange)="list.reload()" aria-label="Default pagination"></ngb-pagination>
 </div>
 
 <div *ngIf="displayMode == 'largeCards'">
-  <app-document-card-large *ngFor="let d of docs.documents" [document]="d" [details]="d.content">
+  <app-document-card-large *ngFor="let d of list.documents" [document]="d" [details]="d.content" (clickTag)="filterByTag($event)" (clickCorrespondent)="filterByCorrespondent($event)">
   </app-document-card-large>
 </div>
 
-<table class="table table-hover table-sm border shadow" *ngIf="displayMode == 'details'">
+<table class="table table-sm border shadow" *ngIf="displayMode == 'details'">
   <thead>
     <th class="d-none d-lg-table-cell">ASN</th>
     <th class="d-none d-md-table-cell">Correspondent</th>
@@ -91,20 +94,37 @@
     <th class="d-none d-xl-table-cell">Added</th>
   </thead>
   <tbody>
-    <tr *ngFor="let d of docs.documents" routerLink="/documents/{{d.id}}">
-      <td class="d-none d-lg-table-cell">{{d.archive_serial_number}}</td>
-      <td class="d-none d-md-table-cell">{{d.correspondent ? d.correspondent.name : ''}}</td>
-      <td>{{d.title}}<app-tag [tag]="t" *ngFor="let t of d.tags" class="ml-1"></app-tag></td>
-      <td class="d-none d-xl-table-cell">{{d.document_type ? d.document_type.name : ''}}</td>
-      <td>{{d.created | date}}</td>
-      <td class="d-none d-xl-table-cell">{{d.added | date}}</td>
+    <tr *ngFor="let d of list.documents">
+      <td class="d-none d-lg-table-cell">
+        {{d.archive_serial_number}}
+      </td>
+      <td class="d-none d-md-table-cell">
+        <ng-container *ngIf="d.correspondent">
+          <a [routerLink]="" (click)="filterByCorrespondent(d.correspondent)" title="Filter by correspondent">{{d.correspondent.name}}</a>
+        </ng-container>
+      </td>
+      <td>
+        <a routerLink="/documents/{{d.id}}" title="Edit document">{{d.title}}</a>
+        <app-tag [tag]="t" *ngFor="let t of d.tags" class="ml-1" clickable="true" linkTitle="Filter by tag" (click)="filterByTag(t)"></app-tag>
+      </td>
+      <td class="d-none d-xl-table-cell">
+        <ng-container *ngIf="d.document_type">
+          <a [routerLink]="" (click)="filterByDocumentType(d.document_type)" title="Filter by document type">{{d.document_type.name}}</a>
+        </ng-container>
+      </td>
+      <td>
+        {{d.created | date}}
+      </td>
+      <td class="d-none d-xl-table-cell">
+        {{d.added | date}}
+      </td>
     </tr>
   </tbody>
 </table>
 
 
 <div class=" m-n2 row" *ngIf="displayMode == 'smallCards'">
-  <app-document-card-small [document]="d" *ngFor="let d of docs.documents"></app-document-card-small>    
+  <app-document-card-small [document]="d" *ngFor="let d of list.documents" (clickTag)="filterByTag($event)" (clickCorrespondent)="filterByCorrespondent($event)"></app-document-card-small>    
 </div>
 
-<p *ngIf="docs.documents.length == 0" class="mx-auto">No results</p>
+<p *ngIf="list.documents.length == 0" class="mx-auto">No results</p>

src-ui/src/app/components/document-list/document-list.component.ts

@@ -1,11 +1,16 @@
 import { Component, OnInit } from '@angular/core';
-import { ActivatedRoute, Router } from '@angular/router';
+import { ActivatedRoute } from '@angular/router';
 import { NgbModal } from '@ng-bootstrap/ng-bootstrap';
 import { cloneFilterRules, FilterRule } from 'src/app/data/filter-rule';
+import { FILTER_CORRESPONDENT, FILTER_DOCUMENT_TYPE, FILTER_HAS_TAG, FILTER_RULE_TYPES } from 'src/app/data/filter-rule-type';
+import { PaperlessCorrespondent } from 'src/app/data/paperless-correspondent';
+import { PaperlessDocumentType } from 'src/app/data/paperless-document-type';
+import { PaperlessTag } from 'src/app/data/paperless-tag';
 import { SavedViewConfig } from 'src/app/data/saved-view-config';
 import { DocumentListViewService } from 'src/app/services/document-list-view.service';
 import { DOCUMENT_SORT_FIELDS } from 'src/app/services/rest/document.service';
 import { SavedViewConfigService } from 'src/app/services/saved-view-config.service';
+import { Toast, ToastService } from 'src/app/services/toast.service';
 import { SaveViewConfigDialogComponent } from './save-view-config-dialog/save-view-config-dialog.component';
 
 @Component({
@@ -16,9 +21,10 @@ import { SaveViewConfigDialogComponent } from './save-view-config-dialog/save-vi
 export class DocumentListComponent implements OnInit {
 
   constructor(
-    public docs: DocumentListViewService,
+    public list: DocumentListViewService,
     public savedViewConfigService: SavedViewConfigService,
     public route: ActivatedRoute,
+    private toastService: ToastService,
     public modalService: NgbModal) { }
 
   displayMode = 'smallCards' // largeCards, smallCards, details
@@ -27,17 +33,13 @@ export class DocumentListComponent implements OnInit {
   showFilter = false
 
   getTitle() {
-    return this.docs.viewConfigOverride ? this.docs.viewConfigOverride.title : "Documents"
+    return this.list.savedViewTitle || "Documents"
   }
 
   getSortFields() {
     return DOCUMENT_SORT_FIELDS
   }
 
-  setSort(field: string) {
-    this.docs.sortField = field
-  }
-
   saveDisplayMode() {
     localStorage.setItem('document-list:displayMode', this.displayMode)
   }
@@ -48,41 +50,74 @@ export class DocumentListComponent implements OnInit {
     }
     this.route.paramMap.subscribe(params => {
       if (params.has('id')) {
-        this.docs.viewConfigOverride = this.savedViewConfigService.getConfig(params.get('id'))
+        this.list.savedView = this.savedViewConfigService.getConfig(params.get('id'))
       } else {
-        this.filterRules = this.docs.filterRules
-        this.showFilter = this.filterRules.length > 0
-        this.docs.viewConfigOverride = null
+        this.list.savedView = null
       }
-      this.reload()
+      this.filterRules = this.list.filterRules
+      //this.showFilter = this.filterRules.length > 0
+      // prevents temporarily visible results from previous views
+      this.list.documents = []
+      this.list.reload()
     })
   }
 
-  reload() {
-    this.docs.reload()
-  }
-
   applyFilterRules() {
-    this.docs.filterRules = this.filterRules
+    this.list.filterRules = this.filterRules
   }
 
   loadViewConfig(config: SavedViewConfig) {
     this.filterRules = cloneFilterRules(config.filterRules)
-    this.docs.loadViewConfig(config)
+    this.list.load(config)
   }
 
   saveViewConfig() {
+    this.savedViewConfigService.updateConfig(this.list.savedView)
+    this.toastService.showToast(Toast.make("Information", `View "${this.list.savedView.title}" saved successfully.`))
+  }
+
+  saveViewConfigAs() {
     let modal = this.modalService.open(SaveViewConfigDialogComponent, {backdrop: 'static'})
     modal.componentInstance.saveClicked.subscribe(formValue => {
-      this.savedViewConfigService.saveConfig({
+      this.savedViewConfigService.newConfig({
         title: formValue.title,
         showInDashboard: formValue.showInDashboard,
         showInSideBar: formValue.showInSideBar,
-        filterRules: this.docs.filterRules,
-        sortDirection: this.docs.sortDirection,
-        sortField: this.docs.sortField
+        filterRules: this.list.filterRules,
+        sortDirection: this.list.sortDirection,
+        sortField: this.list.sortField
       })
       modal.close()
     })
   }
+
+  filterByTag(t: PaperlessTag) {
+    if (this.filterRules.find(rule => rule.type.id == FILTER_HAS_TAG && rule.value == t.id)) {
+      return
+    }
+
+    this.filterRules.push({type: FILTER_RULE_TYPES.find(t => t.id == FILTER_HAS_TAG), value: t.id})
+    this.applyFilterRules()
+  }
+
+  filterByCorrespondent(c: PaperlessCorrespondent) {
+    let existing_rule = this.filterRules.find(rule => rule.type.id == FILTER_CORRESPONDENT)
+    if (existing_rule) {
+      existing_rule.value = c.id
+    } else {
+      this.filterRules.push({type: FILTER_RULE_TYPES.find(t => t.id == FILTER_CORRESPONDENT), value: c.id})
+    }
+    this.applyFilterRules()
+  }
+
+  filterByDocumentType(dt: PaperlessDocumentType) {
+    let existing_rule = this.filterRules.find(rule => rule.type.id == FILTER_DOCUMENT_TYPE)
+    if (existing_rule) {
+      existing_rule.value = dt.id
+    } else {
+      this.filterRules.push({type: FILTER_RULE_TYPES.find(t => t.id == FILTER_DOCUMENT_TYPE), value: dt.id})
+    }
+    this.applyFilterRules()
+  }
+
 }

src-ui/src/app/components/search/search.component.html

@@ -1,13 +1,21 @@
 <app-page-header title="Search results">
 </app-page-header>
 
-<p>Search string: <i>{{query}}</i></p>
+<div *ngIf="errorMessage" class="alert alert-danger">Invalid search query: {{errorMessage}}</div>
 
-<div [class.result-content-searching]="searching" infiniteScroll (scrolled)="onScroll()">
+<p>
+    Search string: <i>{{query}}</i>
+    <ng-container *ngIf="correctedQuery">
+        - Did you mean "<a [routerLink]="" (click)="searchCorrectedQuery()">{{correctedQuery}}</a>"?
+    </ng-container>
+
+</p>
+
+<div *ngIf="!errorMessage" [class.result-content-searching]="searching" infiniteScroll (scrolled)="onScroll()">
     <p>{{resultCount}} result(s)</p>
     <app-document-card-large *ngFor="let result of results"
         [document]="result.document"
         [details]="result.highlights">
 
 </app-document-card-large>
-</div>
+</div>

src-ui/src/app/components/search/search.component.scss

@@ -11,5 +11,5 @@
 }
 
 .result-content-searching {
-    opacity: 0.2;
+    opacity: 0.3;
 }

src-ui/src/app/components/search/search.component.ts

@@ -1,5 +1,5 @@
 import { Component, OnInit } from '@angular/core';
-import { ActivatedRoute } from '@angular/router';
+import { ActivatedRoute, Router } from '@angular/router';
 import { SearchHit } from 'src/app/data/search-result';
 import { SearchService } from 'src/app/services/rest/search.service';
 
@@ -9,7 +9,7 @@ import { SearchService } from 'src/app/services/rest/search.service';
   styleUrls: ['./search.component.scss']
 })
 export class SearchComponent implements OnInit {
-  
+
   results: SearchHit[] = []
 
   query: string = ""
@@ -22,7 +22,11 @@ export class SearchComponent implements OnInit {
 
   resultCount
 
-  constructor(private searchService: SearchService, private route: ActivatedRoute) { }
+  correctedQuery: string = null
+
+  errorMessage: string
+
+  constructor(private searchService: SearchService, private route: ActivatedRoute, private router: Router) { }
 
   ngOnInit(): void {
     this.route.queryParamMap.subscribe(paramMap => {
@@ -31,10 +35,16 @@ export class SearchComponent implements OnInit {
       this.currentPage = 1
       this.loadPage()
     })
-    
+
+  }
+
+  searchCorrectedQuery() {
+    this.router.navigate(["search"], {queryParams: {query: this.correctedQuery}})
   }
 
   loadPage(append: boolean = false) {
+    this.errorMessage = null
+    this.correctedQuery = null
     this.searchService.search(this.query, this.currentPage).subscribe(result => {
       if (append) {
         this.results.push(...result.results)
@@ -44,12 +54,17 @@ export class SearchComponent implements OnInit {
       this.pageCount = result.page_count
       this.searching = false
       this.resultCount = result.count
+      this.correctedQuery = result.corrected_query
+    }, error => {
+      this.searching = false
+      this.resultCount = 1
+      this.pageCount = 1
+      this.results = []
+      this.errorMessage = error.error
     })
   }
 
   onScroll() {
-    console.log(this.currentPage)
-    console.log(this.pageCount)
     if (this.currentPage < this.pageCount) {
       this.currentPage += 1
       this.loadPage(true)

src-ui/src/app/data/filter-rule-type.ts

@@ -1,31 +1,51 @@
+export const FILTER_TITLE = 0
+export const FILTER_CONTENT = 1
+export const FILTER_ASN = 2
+export const FILTER_CORRESPONDENT = 3
+export const FILTER_DOCUMENT_TYPE = 4
+export const FILTER_IS_IN_INBOX = 5
+export const FILTER_HAS_TAG = 6
+export const FILTER_HAS_ANY_TAG = 7
+export const FILTER_CREATED_BEFORE = 8
+export const FILTER_CREATED_AFTER = 9
+export const FILTER_CREATED_YEAR = 10
+export const FILTER_CREATED_MONTH = 11
+export const FILTER_CREATED_DAY = 12
+export const FILTER_ADDED_BEFORE = 13
+export const FILTER_ADDED_AFTER = 14
+export const FILTER_MODIFIED_BEFORE = 15
+export const FILTER_MODIFIED_AFTER = 16
+
 export const FILTER_RULE_TYPES: FilterRuleType[] = [
-  {name: "Title contains", filtervar: "title__icontains", datatype: "string", multi: false},
-  {name: "Content contains", filtervar: "content__icontains", datatype: "string", multi: false},
+
+  {id: FILTER_TITLE, name: "Title contains", filtervar: "title__icontains", datatype: "string", multi: false},
+  {id: FILTER_CONTENT, name: "Content contains", filtervar: "content__icontains", datatype: "string", multi: false},
   
-  {name: "ASN is", filtervar: "archive_serial_number", datatype: "number", multi: false},
+  {id: FILTER_ASN, name: "ASN is", filtervar: "archive_serial_number", datatype: "number", multi: false},
   
-  {name: "Correspondent is", filtervar: "correspondent__id", datatype: "correspondent", multi: false},
-  {name: "Document type is", filtervar: "document_type__id", datatype: "document_type", multi: false},
+  {id: FILTER_CORRESPONDENT, name: "Correspondent is", filtervar: "correspondent__id", datatype: "correspondent", multi: false},
+  {id: FILTER_DOCUMENT_TYPE, name: "Document type is", filtervar: "document_type__id", datatype: "document_type", multi: false},
 
-  {name: "Is in Inbox", filtervar: "is_in_inbox", datatype: "boolean", multi: false},  
-  {name: "Has tag", filtervar: "tags__id__all", datatype: "tag", multi: true},  
-  {name: "Has any tag", filtervar: "is_tagged", datatype: "boolean", multi: false},
+  {id: FILTER_IS_IN_INBOX, name: "Is in Inbox", filtervar: "is_in_inbox", datatype: "boolean", multi: false},  
+  {id: FILTER_HAS_TAG, name: "Has tag", filtervar: "tags__id__all", datatype: "tag", multi: true},  
+  {id: FILTER_HAS_ANY_TAG, name: "Has any tag", filtervar: "is_tagged", datatype: "boolean", multi: false},
 
-  {name: "Created before", filtervar: "created__date__lt", datatype: "date", multi: false},
-  {name: "Created after", filtervar: "created__date__gt", datatype: "date", multi: false},
+  {id: FILTER_CREATED_BEFORE, name: "Created before", filtervar: "created__date__lt", datatype: "date", multi: false},
+  {id: FILTER_CREATED_AFTER, name: "Created after", filtervar: "created__date__gt", datatype: "date", multi: false},
 
-  {name: "Year created is", filtervar: "created__year", datatype: "number", multi: false},
-  {name: "Month created is", filtervar: "created__month", datatype: "number", multi: false},
-  {name: "Day created is", filtervar: "created__day", datatype: "number", multi: false},
+  {id: FILTER_CREATED_YEAR, name: "Year created is", filtervar: "created__year", datatype: "number", multi: false},
+  {id: FILTER_CREATED_MONTH, name: "Month created is", filtervar: "created__month", datatype: "number", multi: false},
+  {id: FILTER_CREATED_DAY, name: "Day created is", filtervar: "created__day", datatype: "number", multi: false},
 
-  {name: "Added before", filtervar: "added__date__lt", datatype: "date", multi: false},
-  {name: "Added after", filtervar: "added__date__gt", datatype: "date", multi: false},
+  {id: FILTER_ADDED_BEFORE, name: "Added before", filtervar: "added__date__lt", datatype: "date", multi: false},
+  {id: FILTER_ADDED_AFTER, name: "Added after", filtervar: "added__date__gt", datatype: "date", multi: false},
   
-  {name: "Modified before", filtervar: "modified__date__lt", datatype: "date", multi: false},
-  {name: "Modified after", filtervar: "modified__date__gt", datatype: "date", multi: false},
+  {id: FILTER_MODIFIED_BEFORE, name: "Modified before", filtervar: "modified__date__lt", datatype: "date", multi: false},
+  {id: FILTER_MODIFIED_AFTER, name: "Modified after", filtervar: "modified__date__gt", datatype: "date", multi: false},
 ]
 
 export interface FilterRuleType {
+  id: number
   name: string
   filtervar: string
   datatype: string //number, string, boolean, date

src-ui/src/app/data/search-result.ts

@@ -21,7 +21,9 @@ export interface SearchResult {
   page?: number
   page_count?: number
 
+  corrected_query?: string
+
   results?: SearchHit[]
 
 
-}
+}

src-ui/src/app/services/document-list-view.service.ts

@@ -7,6 +7,12 @@ import { DOCUMENT_LIST_SERVICE, GENERAL_SETTINGS } from '../data/storage-keys';
 import { DocumentService } from './rest/document.service';
 
 
+/**
+ * This service manages the document list which is displayed using the document list view.
+ * 
+ * This service also serves saved views by transparently switching between the document list
+ * and saved views on request. See below.
+ */
 @Injectable({
   providedIn: 'root'
 })
@@ -14,80 +20,127 @@ export class DocumentListViewService {
 
   static DEFAULT_SORT_FIELD = 'created'
 
+  isReloading: boolean = false
   documents: PaperlessDocument[] = []
   currentPage = 1
   currentPageSize: number = +localStorage.getItem(GENERAL_SETTINGS.DOCUMENT_LIST_SIZE) || GENERAL_SETTINGS.DOCUMENT_LIST_SIZE_DEFAULT
   collectionSize: number
   
-  private currentViewConfig: SavedViewConfig
-  //TODO: make private
-  viewConfigOverride: SavedViewConfig
+  /**
+   * This is the current config for the document list. The service will always remember the last settings used for the document list.
+   */
+  private _documentListViewConfig: SavedViewConfig
+  /**
+   * Optionally, this is the currently selected saved view, which might be null.
+   */
+  private _savedViewConfig: SavedViewConfig
 
-  get viewId() {
-    return this.viewConfigOverride?.id
+  get savedView() {
+    return this._savedViewConfig
+  }
+
+  set savedView(value) {
+    if (value) {
+      //this is here so that we don't modify value, which might be the actual instance of the saved view.
+      this._savedViewConfig = Object.assign({}, value)
+    } else {
+      this._savedViewConfig = null
+    }
+  }
+
+  get savedViewId() {
+    return this.savedView?.id
+  }
+
+  get savedViewTitle() {
+    return this.savedView?.title
+  }
+
+  get documentListView() {
+    return this._documentListViewConfig
+  }
+
+  set documentListView(value) {
+    if (value) {
+      this._documentListViewConfig = Object.assign({}, value)
+      this.saveDocumentListView()
+    }
+  }
+
+  /**
+   * This is what switches between the saved views and the document list view. Everything on the document list uses
+   * this property to determine the settings for the currently displayed document list.
+   */
+  get view() {
+    return this.savedView || this.documentListView
+  }
+
+  load(config: SavedViewConfig) {
+    this.view.filterRules = cloneFilterRules(config.filterRules)
+    this.view.sortDirection = config.sortDirection
+    this.view.sortField = config.sortField
+    this.reload()
   }
 
   reload(onFinish?) {
-    let viewConfig = this.viewConfigOverride || this.currentViewConfig
-
+    this.isReloading = true
     this.documentService.list(
       this.currentPage,
       this.currentPageSize,
-      viewConfig.sortField,
-      viewConfig.sortDirection,
-      viewConfig.filterRules).subscribe(
+      this.view.sortField,
+      this.view.sortDirection,
+      this.view.filterRules).subscribe(
         result => {
           this.collectionSize = result.count
           this.documents = result.results
           if (onFinish) {
             onFinish()
           }
+          this.isReloading = false
         },
         error => {
           if (error.error['detail'] == 'Invalid page.') {
             this.currentPage = 1
             this.reload()
           }
+          this.isReloading = false
         })
   }
 
   set filterRules(filterRules: FilterRule[]) {
-    this.currentViewConfig.filterRules = cloneFilterRules(filterRules)
-    this.saveCurrentViewConfig()
+    //we're going to clone the filterRules object, since we don't
+    //want changes in the filter editor to propagate into here right away.
+    this.view.filterRules = cloneFilterRules(filterRules)
     this.reload()
+    this.saveDocumentListView()
   }
 
   get filterRules(): FilterRule[] {
-    return cloneFilterRules(this.currentViewConfig.filterRules)
+    return cloneFilterRules(this.view.filterRules)
   }
 
   set sortField(field: string) {
-    this.currentViewConfig.sortField = field
-    this.saveCurrentViewConfig()
+    this.view.sortField = field
+    this.saveDocumentListView()
     this.reload()
   }
 
   get sortField(): string {
-    return this.currentViewConfig.sortField
+    return this.view.sortField
   }
 
   set sortDirection(direction: string) {
-    this.currentViewConfig.sortDirection = direction
-    this.saveCurrentViewConfig()
+    this.view.sortDirection = direction
+    this.saveDocumentListView()
     this.reload()
   }
 
   get sortDirection(): string {
-    return this.currentViewConfig.sortDirection
+    return this.view.sortDirection
   }
 
-  loadViewConfig(config: SavedViewConfig) {
-    Object.assign(this.currentViewConfig, config)
-    this.reload()
-  }
-
-  private saveCurrentViewConfig() {
-    sessionStorage.setItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG, JSON.stringify(this.currentViewConfig))
+  private saveDocumentListView() {
+    sessionStorage.setItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG, JSON.stringify(this.documentListView))
   }
 
   getLastPage(): number {
@@ -134,21 +187,21 @@ export class DocumentListViewService {
   }
 
   constructor(private documentService: DocumentService) { 
-    let currentViewConfigJson = sessionStorage.getItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG)
-    if (currentViewConfigJson) {
+    let documentListViewConfigJson = sessionStorage.getItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG)
+    if (documentListViewConfigJson) {
       try {
-        this.currentViewConfig = JSON.parse(currentViewConfigJson)
+        this.documentListView = JSON.parse(documentListViewConfigJson)
       } catch (e) {
         sessionStorage.removeItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG)
-        this.currentViewConfig = null
+        this.documentListView = null
       }
     }
-    if (!this.currentViewConfig) {
-      this.currentViewConfig = {
+    if (!this.documentListView) {
+      this.documentListView = {
         filterRules: [],
         sortDirection: 'des',
         sortField: 'created'
       }
-      }
+    }
   }
 }

src-ui/src/app/services/saved-view-config.service.ts

@@ -36,13 +36,21 @@ export class SavedViewConfigService {
     return this.configs.find(sf => sf.id == id)
   }
 
-  saveConfig(config: SavedViewConfig) {
+  newConfig(config: SavedViewConfig) {
     config.id = uuidv4()
     this.configs.push(config)
 
     this.save()
   }
 
+  updateConfig(config: SavedViewConfig) {
+    let savedConfig = this.configs.find(c => c.id == config.id)
+    if (savedConfig) {
+      Object.assign(savedConfig, config)
+      this.save()
+    }
+  }
+
   private save() {
     localStorage.setItem('saved-view-config-service:savedConfigs', JSON.stringify(this.configs))
   }

src/documents/consumer.py

@@ -9,10 +9,11 @@ from django.db import transaction
 from django.utils import timezone
 
 from .classifier import DocumentClassifier, IncompatibleClassifierVersionError
-from .file_handling import generate_filename, create_source_path_directory
+from .file_handling import create_source_path_directory
 from .loggers import LoggingMixin
 from .models import Document, FileInfo, Correspondent, DocumentType, Tag
-from .parsers import ParseError, get_parser_class_for_mime_type
+from .parsers import ParseError, get_parser_class_for_mime_type, \
+    get_supported_file_extensions
 from .signals import (
     document_consumption_finished,
     document_consumption_started
@@ -39,6 +40,21 @@ class Consumer(LoggingMixin):
             raise ConsumerError("Cannot consume {}: It is not a file".format(
                 self.path))
 
+    def pre_check_file_extension(self):
+        extensions = get_supported_file_extensions()
+        _, ext = os.path.splitext(self.filename)
+
+        if not ext:
+            raise ConsumerError(
+                f"Not consuming {self.filename}: File type unknown."
+            )
+
+        if ext not in extensions:
+            raise ConsumerError(
+                f"Not consuming {self.filename}: File extension {ext} does "
+                f"not map to any known file type ({str(extensions)})"
+            )
+
     def pre_check_duplicate(self):
         with open(self.path, "rb") as f:
             checksum = hashlib.md5(f.read()).hexdigest()
@@ -80,6 +96,7 @@ class Consumer(LoggingMixin):
         # Make sure that preconditions for consuming the file are met.
 
         self.pre_check_file_exists()
+        self.pre_check_file_extension()
         self.pre_check_directories()
         self.pre_check_duplicate()
 
@@ -167,6 +184,13 @@ class Consumer(LoggingMixin):
                 self._write(document, self.path, document.source_path)
                 self._write(document, thumbnail, document.thumbnail_path)
 
+                # Afte performing all database operations and moving files
+                # into place, tell paperless where the file is.
+                document.filename = os.path.basename(document.source_path)
+                # Saving the document now will trigger the filename handling
+                # logic.
+                document.save()
+
                 # Delete the file only if it was successfully consumed
                 self.log("debug", "Deleting file {}".format(self.path))
                 os.unlink(self.path)
@@ -217,12 +241,6 @@ class Consumer(LoggingMixin):
 
         self.apply_overrides(document)
 
-        document.filename = generate_filename(document)
-
-        # We need to save the document twice, since we need the PK of the
-        # document in order to create its filename above.
-        document.save()
-
         return document
 
     def apply_overrides(self, document):

src/documents/index.py

@@ -4,10 +4,11 @@ from contextlib import contextmanager
 
 from django.conf import settings
 from whoosh import highlight
-from whoosh.fields import Schema, TEXT, NUMERIC
+from whoosh.fields import Schema, TEXT, NUMERIC, KEYWORD, DATETIME
 from whoosh.highlight import Formatter, get_text
 from whoosh.index import create_in, exists_in, open_dir
 from whoosh.qparser import MultifieldParser
+from whoosh.qparser.dateparse import DateParserPlugin
 from whoosh.writing import AsyncWriter
 
 
@@ -59,14 +60,19 @@ def get_schema():
         id=NUMERIC(stored=True, unique=True, numtype=int),
         title=TEXT(stored=True),
         content=TEXT(),
-        correspondent=TEXT(stored=True)
+        correspondent=TEXT(stored=True),
+        tag=KEYWORD(stored=True, commas=True, scorable=True, lowercase=True),
+        type=TEXT(stored=True),
+        created=DATETIME(stored=True, sortable=True),
+        modified=DATETIME(stored=True, sortable=True),
+        added=DATETIME(stored=True, sortable=True),
     )
 
 
 def open_index(recreate=False):
     try:
         if exists_in(settings.INDEX_DIR) and not recreate:
-            return open_dir(settings.INDEX_DIR)
+            return open_dir(settings.INDEX_DIR, schema=get_schema())
     except Exception as e:
         logger.error(f"Error while opening the index: {e}, recreating.")
 
@@ -77,11 +83,17 @@ def open_index(recreate=False):
 
 def update_document(writer, doc):
     logger.debug("Indexing {}...".format(doc))
+    tags = ",".join([t.name for t in doc.tags.all()])
     writer.update_document(
         id=doc.pk,
         title=doc.title,
         content=doc.content,
-        correspondent=doc.correspondent.name if doc.correspondent else None
+        correspondent=doc.correspondent.name if doc.correspondent else None,
+        tag=tags if tags else None,
+        type=doc.document_type.name if doc.document_type else None,
+        created=doc.created,
+        added=doc.added,
+        modified=doc.modified,
     )
 
 
@@ -103,16 +115,27 @@ def remove_document_from_index(document):
 
 
 @contextmanager
-def query_page(ix, query, page):
+def query_page(ix, querystring, page):
     searcher = ix.searcher()
     try:
-        query_parser = MultifieldParser(["content", "title", "correspondent"],
-                                        ix.schema).parse(query)
-        result_page = searcher.search_page(query_parser, page)
+        qp = MultifieldParser(
+            ["content", "title", "correspondent", "tag", "type"],
+            ix.schema)
+        qp.add_plugin(DateParserPlugin())
+
+        q = qp.parse(querystring)
+        result_page = searcher.search_page(q, page)
         result_page.results.fragmenter = highlight.ContextFragmenter(
             surround=50)
         result_page.results.formatter = JsonFormatter()
-        yield result_page
+
+        corrected = searcher.correct_query(q, querystring)
+        if corrected.query != q:
+            corrected_query = corrected.string
+        else:
+            corrected_query = None
+
+        yield result_page, corrected_query
     finally:
         searcher.close()
 

src/documents/models.py

@@ -1,7 +1,6 @@
 # coding=utf-8
 
 import logging
-import mimetypes
 import os
 import re
 from collections import OrderedDict
@@ -12,6 +11,8 @@ from django.db import models
 from django.utils import timezone
 from django.utils.text import slugify
 
+from documents.parsers import get_default_file_extension
+
 
 class MatchingModel(models.Model):
 
@@ -198,7 +199,7 @@ class Document(models.Model):
         ordering = ("correspondent", "title")
 
     def __str__(self):
-        created = self.created.strftime("%Y%m%d%H%M%S")
+        created = self.created.strftime("%Y%m%d")
         if self.correspondent and self.title:
             return "{}: {} - {}".format(
                 created, self.correspondent, self.title)
@@ -230,8 +231,7 @@ class Document(models.Model):
 
     @property
     def file_type(self):
-        # TODO: this is not stable across python versions
-        return mimetypes.guess_extension(str(self.mime_type))
+        return get_default_file_extension(self.mime_type)
 
     @property
     def thumbnail_path(self):

src/documents/parsers.py

@@ -1,4 +1,5 @@
 import logging
+import mimetypes
 import os
 import re
 import shutil
@@ -42,6 +43,29 @@ def is_mime_type_supported(mime_type):
     return get_parser_class_for_mime_type(mime_type) is not None
 
 
+def get_default_file_extension(mime_type):
+    for response in document_consumer_declaration.send(None):
+        parser_declaration = response[1]
+        supported_mime_types = parser_declaration["mime_types"]
+
+        if mime_type in supported_mime_types:
+            return supported_mime_types[mime_type]
+
+    return None
+
+
+def get_supported_file_extensions():
+    extensions = set()
+    for response in document_consumer_declaration.send(None):
+        parser_declaration = response[1]
+        supported_mime_types = parser_declaration["mime_types"]
+
+        for mime_type in supported_mime_types:
+            extensions.update(mimetypes.guess_all_extensions(mime_type))
+
+    return extensions
+
+
 def get_parser_class_for_mime_type(mime_type):
 
     options = []

src/documents/signals/handlers.py

@@ -9,6 +9,7 @@ from django.contrib.contenttypes.models import ContentType
 from django.db import models, DatabaseError
 from django.dispatch import receiver
 from django.utils import timezone
+from rest_framework.reverse import reverse
 
 from .. import index, matching
 from ..file_handling import delete_empty_directories, generate_filename, \
@@ -157,10 +158,10 @@ def run_post_consume_script(sender, document, **kwargs):
         settings.POST_CONSUME_SCRIPT,
         str(document.pk),
         document.file_name,
-        document.source_path,
-        document.thumbnail_path,
-        None,
-        None,
+        os.path.normpath(document.source_path),
+        os.path.normpath(document.thumbnail_path),
+        reverse("document-download", kwargs={"pk": document.pk}),
+        reverse("document-thumb", kwargs={"pk": document.pk}),
         str(document.correspondent),
         str(",".join(document.tags.all().values_list("slug", flat=True)))
     )).wait()
@@ -217,7 +218,11 @@ def update_filename_and_move_files(sender, instance, **kwargs):
     try:
         os.rename(old_path, new_path)
         instance.filename = new_filename
-        instance.save()
+        # Don't save here to prevent infinite recursion.
+        Document.objects.filter(pk=instance.pk).update(filename=new_filename)
+
+        logging.getLogger(__name__).debug(
+            f"Moved file {old_path} to {new_path}.")
 
     except OSError as e:
         instance.filename = old_filename

src/documents/tests/test_api.py

@@ -5,16 +5,17 @@ from unittest import mock
 from django.contrib.auth.models import User
 from pathvalidate import ValidationError
 from rest_framework.test import APITestCase
+from whoosh.writing import AsyncWriter
 
 from documents import index
 from documents.models import Document, Correspondent, DocumentType, Tag
 from documents.tests.utils import DirectoriesMixin
 
 
-class DocumentApiTest(DirectoriesMixin, APITestCase):
+class TestDocumentApi(DirectoriesMixin, APITestCase):
 
     def setUp(self):
-        super(DocumentApiTest, self).setUp()
+        super(TestDocumentApi, self).setUp()
 
         user = User.objects.create_superuser(username="temp_admin")
         self.client.force_login(user=user)
@@ -173,7 +174,7 @@ class DocumentApiTest(DirectoriesMixin, APITestCase):
         d1=Document.objects.create(title="invoice", content="the thing i bought at a shop and paid with bank account", checksum="A", pk=1)
         d2=Document.objects.create(title="bank statement 1", content="things i paid for in august", pk=2, checksum="B")
         d3=Document.objects.create(title="bank statement 3", content="things i paid for in september", pk=3, checksum="C")
-        with index.open_index(False).writer() as writer:
+        with AsyncWriter(index.open_index()) as writer:
             # Note to future self: there is a reason we dont use a model signal handler to update the index: some operations edit many documents at once
             # (retagger, renamer) and we don't want to open a writer for each of these, but rather perform the entire operation with one writer.
             # That's why we cant open the writer in a model on_save handler or something.
@@ -209,7 +210,7 @@ class DocumentApiTest(DirectoriesMixin, APITestCase):
         self.assertEqual(len(results), 0)
 
     def test_search_multi_page(self):
-        with index.open_index(False).writer() as writer:
+        with AsyncWriter(index.open_index()) as writer:
             for i in range(55):
                 doc = Document.objects.create(checksum=str(i), pk=i+1, title=f"Document {i+1}", content="content")
                 index.update_document(writer, doc)
@@ -248,7 +249,7 @@ class DocumentApiTest(DirectoriesMixin, APITestCase):
         self.assertEqual(len(results), 5)
 
     def test_search_invalid_page(self):
-        with index.open_index(False).writer() as writer:
+        with AsyncWriter(index.open_index()) as writer:
             for i in range(15):
                 doc = Document.objects.create(checksum=str(i), pk=i+1, title=f"Document {i+1}", content="content")
                 index.update_document(writer, doc)
@@ -288,6 +289,22 @@ class DocumentApiTest(DirectoriesMixin, APITestCase):
         self.assertEqual(response.status_code, 200)
         self.assertEqual(len(response.data), 10)
 
+    def test_search_spelling_correction(self):
+        with AsyncWriter(index.open_index()) as writer:
+            for i in range(55):
+                doc = Document.objects.create(checksum=str(i), pk=i+1, title=f"Document {i+1}", content=f"Things document {i+1}")
+                index.update_document(writer, doc)
+
+        response = self.client.get("/api/search/?query=thing")
+        correction = response.data['corrected_query']
+
+        self.assertEqual(correction, "things")
+
+        response = self.client.get("/api/search/?query=things")
+        correction = response.data['corrected_query']
+
+        self.assertEqual(correction, None)
+
     def test_statistics(self):
 
         doc1 = Document.objects.create(title="none1", checksum="A")

src/documents/tests/test_consumer.py

@@ -423,7 +423,7 @@ class TestConsumer(DirectoriesMixin, TestCase):
         m = patcher.start()
         m.return_value = [(None, {
             "parser": self.make_dummy_parser,
-            "mime_types": ["application/pdf"],
+            "mime_types": {"application/pdf": ".pdf"},
             "weight": 0
         })]
 
@@ -435,6 +435,7 @@ class TestConsumer(DirectoriesMixin, TestCase):
         fd, f = tempfile.mkstemp(suffix=".pdf", dir=self.dirs.scratch_dir)
         return f
 
+    @override_settings(PAPERLESS_FILENAME_FORMAT=None)
     def testNormalOperation(self):
 
         filename = self.get_test_file()
@@ -519,7 +520,7 @@ class TestConsumer(DirectoriesMixin, TestCase):
         try:
             self.consumer.try_consume_file(self.get_test_file())
         except ConsumerError as e:
-            self.assertTrue(str(e).startswith("No parsers abvailable"))
+            self.assertTrue("File extension .pdf does not map to any" in str(e))
             return
 
         self.fail("Should throw exception")
@@ -528,7 +529,7 @@ class TestConsumer(DirectoriesMixin, TestCase):
     def testFaultyParser(self, m):
         m.return_value = [(None, {
             "parser": self.make_faulty_parser,
-            "mime_types": ["application/pdf"],
+            "mime_types": {"application/pdf": ".pdf"},
             "weight": 0
         })]
 
@@ -563,13 +564,34 @@ class TestConsumer(DirectoriesMixin, TestCase):
 
         document = self.consumer.try_consume_file(filename, override_filename="Bank - Test.pdf", override_title="new docs")
 
-        print(document.source_path)
-        print("===")
-
         self.assertEqual(document.title, "new docs")
         self.assertEqual(document.correspondent.name, "Bank")
         self.assertEqual(document.filename, "bank/new-docs-0000001.pdf")
 
+    @override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}")
+    @mock.patch("documents.signals.handlers.generate_filename")
+    def testFilenameHandlingUnstableFormat(self, m):
+
+        filenames = ["this", "that", "now this", "i cant decide"]
+
+        def get_filename():
+            f = filenames.pop()
+            filenames.insert(0, f)
+            return f
+
+        m.side_effect = lambda f: get_filename()
+
+        filename = self.get_test_file()
+
+        Tag.objects.create(name="test", is_inbox_tag=True)
+
+        document = self.consumer.try_consume_file(filename, override_filename="Bank - Test.pdf", override_title="new docs")
+
+        self.assertEqual(document.title, "new docs")
+        self.assertEqual(document.correspondent.name, "Bank")
+        self.assertIsNotNone(os.path.isfile(document.title))
+        self.assertTrue(os.path.isfile(document.source_path))
+
     @mock.patch("documents.consumer.DocumentClassifier")
     def testClassifyDocument(self, m):
         correspondent = Correspondent.objects.create(name="test")

src/documents/tests/test_file_handling.py

@@ -1,14 +1,15 @@
 import os
 import shutil
 from pathlib import Path
+from unittest import mock
 from uuid import uuid4
 
 from django.conf import settings
+from django.db import DatabaseError
 from django.test import TestCase, override_settings
 
 from ..file_handling import generate_filename, create_source_path_directory, delete_empty_directories
 from ..models import Document, Correspondent
-from ..signals.handlers import update_filename_and_move_files
 
 
 class TestDate(TestCase):
@@ -133,18 +134,14 @@ class TestDate(TestCase):
         document.correspondent = Correspondent.objects.get_or_create(
             name="test")[0]
 
-        # This will cause save() to fail.
-        document.checksum = document1.checksum
+        with mock.patch("documents.signals.handlers.Document.objects.filter") as m:
+            m.side_effect = DatabaseError()
+            document.save()
 
-        # Assume saving the document initially works, this gets called.
-        # After renaming, an error occurs, and filename is not saved:
-        # document should still be available at document.filename.
-        update_filename_and_move_files(None, document)
-
-        # Check proper handling of files
-        self.assertTrue(os.path.isfile(document.source_path))
-        self.assertEqual(os.path.isfile(settings.MEDIA_ROOT + "/documents/originals/none/none-{:07d}.pdf".format(document.pk)), True)
-        self.assertEqual(document.filename, "none/none-{:07d}.pdf".format(document.pk))
+            # Check proper handling of files
+            self.assertTrue(os.path.isfile(document.source_path))
+            self.assertEqual(os.path.isfile(settings.MEDIA_ROOT + "/documents/originals/none/none-{:07d}.pdf".format(document.pk)), True)
+            self.assertEqual(document.filename, "none/none-{:07d}.pdf".format(document.pk))
 
     @override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{correspondent}")
     def test_document_delete(self):

src/documents/tests/test_management_decrypt.py

@@ -17,7 +17,8 @@ class TestDecryptDocuments(TestCase):
     @override_settings(
         ORIGINALS_DIR=os.path.join(os.path.dirname(__file__), "samples", "originals"),
         THUMBNAIL_DIR=os.path.join(os.path.dirname(__file__), "samples", "thumb"),
-        PASSPHRASE="test"
+        PASSPHRASE="test",
+        PAPERLESS_FILENAME_FORMAT=None
     )
     @mock.patch("documents.management.commands.decrypt_documents.input")
     def test_decrypt(self, m):
@@ -34,8 +35,8 @@ class TestDecryptDocuments(TestCase):
             PASSPHRASE="test"
         ).enable()
 
-        shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "originals", "0000002.pdf.gpg"), os.path.join(originals_dir, "0000002.pdf.gpg"))
-        shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "thumb", "0000002.png.gpg"), os.path.join(thumb_dir, "0000002.png.gpg"))
+        shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000002.pdf.gpg"), os.path.join(originals_dir, "0000002.pdf.gpg"))
+        shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "documents", "thumbnails", "0000002.png.gpg"), os.path.join(thumb_dir, "0000002.png.gpg"))
 
         Document.objects.create(checksum="9c9691e51741c1f4f41a20896af31770", title="wow", filename="0000002.pdf.gpg", id=2, mime_type="application/pdf", storage_type=Document.STORAGE_TYPE_GPG)
 

src/documents/tests/test_management_exporter.py

@@ -1,6 +1,7 @@
 import hashlib
 import json
 import os
+import shutil
 import tempfile
 
 from django.core.management import call_command
@@ -8,17 +9,19 @@ from django.test import TestCase, override_settings
 
 from documents.management.commands import document_exporter
 from documents.models import Document, Tag, DocumentType, Correspondent
+from documents.tests.utils import DirectoriesMixin
 
 
-class TestExporter(TestCase):
+class TestExporter(DirectoriesMixin, TestCase):
 
     @override_settings(
-        ORIGINALS_DIR=os.path.join(os.path.dirname(__file__), "samples", "originals"),
-        THUMBNAIL_DIR=os.path.join(os.path.dirname(__file__), "samples", "thumb"),
         PASSPHRASE="test"
     )
     def test_exporter(self):
-        file = os.path.join(os.path.dirname(__file__), "samples", "originals", "0000001.pdf")
+        shutil.rmtree(os.path.join(self.dirs.media_dir, "documents"))
+        shutil.copytree(os.path.join(os.path.dirname(__file__), "samples", "documents"), os.path.join(self.dirs.media_dir, "documents"))
+
+        file = os.path.join(self.dirs.originals_dir, "0000001.pdf")
 
         with open(file, "rb") as f:
             checksum = hashlib.md5(f.read()).hexdigest()

src/documents/tests/test_parsers.py

@@ -4,7 +4,10 @@ from unittest import mock
 
 from django.test import TestCase
 
-from documents.parsers import get_parser_class
+from documents.parsers import get_parser_class, get_supported_file_extensions, get_default_file_extension, \
+    get_parser_class_for_mime_type
+from paperless_tesseract.parsers import RasterisedDocumentParser
+from paperless_text.parsers import TextDocumentParser
 
 
 def fake_magic_from_file(file, mime=False):
@@ -27,7 +30,7 @@ class TestParserDiscovery(TestCase):
             pass
 
         m.return_value = (
-            (None, {"weight": 0, "parser": DummyParser, "mime_types": ["application/pdf"]}),
+            (None, {"weight": 0, "parser": DummyParser, "mime_types": {"application/pdf": ".pdf"}}),
         )
 
         self.assertEqual(
@@ -45,8 +48,8 @@ class TestParserDiscovery(TestCase):
             pass
 
         m.return_value = (
-            (None, {"weight": 0, "parser": DummyParser1, "mime_types": ["application/pdf"]}),
-            (None, {"weight": 1, "parser": DummyParser2, "mime_types": ["application/pdf"]}),
+            (None, {"weight": 0, "parser": DummyParser1, "mime_types": {"application/pdf": ".pdf"}}),
+            (None, {"weight": 1, "parser": DummyParser2, "mime_types": {"application/pdf": ".pdf"}}),
         )
 
         self.assertEqual(
@@ -61,3 +64,21 @@ class TestParserDiscovery(TestCase):
             self.assertIsNone(
                 get_parser_class("doc.pdf")
             )
+
+
+class TestParserAvailability(TestCase):
+
+    def test_file_extensions(self):
+
+        for ext in [".pdf", ".jpe", ".jpg", ".jpeg", ".txt", ".csv"]:
+            self.assertIn(ext, get_supported_file_extensions())
+        self.assertEqual(get_default_file_extension('application/pdf'), ".pdf")
+        self.assertEqual(get_default_file_extension('image/png'), ".png")
+        self.assertEqual(get_default_file_extension('image/jpeg'), ".jpg")
+        self.assertEqual(get_default_file_extension('text/plain'), ".txt")
+        self.assertEqual(get_default_file_extension('text/csv'), ".csv")
+        self.assertEqual(get_default_file_extension('aasdasd/dgfgf'), None)
+
+        self.assertEqual(get_parser_class_for_mime_type('application/pdf'), RasterisedDocumentParser)
+        self.assertEqual(get_parser_class_for_mime_type('text/plain'), TextDocumentParser)
+        self.assertEqual(get_parser_class_for_mime_type('text/sdgsdf'), None)

src/documents/tests/test_post_consume_handlers.py

@@ -0,0 +1,57 @@
+from unittest import mock
+
+from django.test import TestCase, override_settings
+
+from documents.models import Document, Tag, Correspondent
+from documents.signals.handlers import run_post_consume_script
+
+
+class PostConsumeTestCase(TestCase):
+
+    @mock.patch("documents.signals.handlers.Popen")
+    @override_settings(POST_CONSUME_SCRIPT=None)
+    def test_no_post_consume_script(self, m):
+        doc = Document.objects.create(title="Test", mime_type="application/pdf")
+        tag1 = Tag.objects.create(name="a")
+        tag2 = Tag.objects.create(name="b")
+        doc.tags.add(tag1)
+        doc.tags.add(tag2)
+
+        run_post_consume_script(None, doc)
+
+        m.assert_not_called()
+
+    @mock.patch("documents.signals.handlers.Popen")
+    @override_settings(POST_CONSUME_SCRIPT="script")
+    def test_post_consume_script_simple(self, m):
+        doc = Document.objects.create(title="Test", mime_type="application/pdf")
+
+        run_post_consume_script(None, doc)
+
+        m.assert_called_once()
+
+    @mock.patch("documents.signals.handlers.Popen")
+    @override_settings(POST_CONSUME_SCRIPT="script")
+    def test_post_consume_script_simple(self, m):
+        c = Correspondent.objects.create(name="my_bank")
+        doc = Document.objects.create(title="Test", mime_type="application/pdf", correspondent=c)
+        tag1 = Tag.objects.create(name="a")
+        tag2 = Tag.objects.create(name="b")
+        doc.tags.add(tag1)
+        doc.tags.add(tag2)
+
+        run_post_consume_script(None, doc)
+
+        m.assert_called_once()
+
+        args, kwargs = m.call_args
+
+        command = args[0]
+
+        self.assertEqual(command[0], "script")
+        self.assertEqual(command[1], str(doc.pk))
+        self.assertEqual(command[5], f"/api/documents/{doc.pk}/download/")
+        self.assertEqual(command[6], f"/api/documents/{doc.pk}/thumb/")
+        self.assertEqual(command[7], "my_bank")
+        # TODO: tags are unordered by default.
+        self.assertEqual(command[8], "a,b")

src/documents/tests/test_tasks.py

@@ -0,0 +1,23 @@
+from datetime import datetime
+
+from django.test import TestCase
+
+from documents import tasks
+from documents.models import Document
+from documents.tests.utils import DirectoriesMixin
+
+
+class TestTasks(DirectoriesMixin, TestCase):
+
+    def test_index_reindex(self):
+        Document.objects.create(title="test", content="my document", checksum="wow", added=datetime.now(), created=datetime.now(), modified=datetime.now())
+
+        tasks.index_reindex()
+
+    def test_index_optimize(self):
+        Document.objects.create(title="test", content="my document", checksum="wow", added=datetime.now(), created=datetime.now(), modified=datetime.now())
+
+        tasks.index_optimize()
+
+    def test_train_classifier(self):
+        tasks.train_classifier()

src/documents/views.py

@@ -217,30 +217,34 @@ class SearchView(APIView):
                 }
 
     def get(self, request, format=None):
-        if 'query' in request.query_params:
-            query = request.query_params['query']
-            try:
-                page = int(request.query_params.get('page', 1))
-            except (ValueError, TypeError):
-                page = 1
-
-            if page < 1:
-                page = 1
-
-            with index.query_page(self.ix, query, page) as result_page:
-                return Response(
-                    {'count': len(result_page),
-                     'page': result_page.pagenum,
-                     'page_count': result_page.pagecount,
-                     'results': list(map(self.add_infos_to_hit, result_page))})
-
-        else:
+        if 'query' not in request.query_params:
             return Response({
                 'count': 0,
                 'page': 0,
                 'page_count': 0,
                 'results': []})
 
+        query = request.query_params['query']
+        try:
+            page = int(request.query_params.get('page', 1))
+        except (ValueError, TypeError):
+            page = 1
+
+        if page < 1:
+            page = 1
+
+        try:
+            with index.query_page(self.ix, query, page) as (result_page,
+                                                            corrected_query):
+                return Response(
+                    {'count': len(result_page),
+                     'page': result_page.pagenum,
+                     'page_count': result_page.pagecount,
+                     'corrected_query': corrected_query,
+                     'results': list(map(self.add_infos_to_hit, result_page))})
+        except Exception as e:
+            return HttpResponseBadRequest(str(e))
+
 
 class SearchAutoCompleteView(APIView):
 

src/paperless/db.py

@@ -17,16 +17,3 @@ class GnuPG:
             passphrase = settings.PASSPHRASE
 
         return cls.gpg.decrypt_file(file_handle, passphrase=passphrase).data
-
-    @classmethod
-    def encrypted(cls, file_handle, passphrase=None):
-
-        if not passphrase:
-            passphrase = settings.PASSPHRASE
-
-        return cls.gpg.encrypt_file(
-            file_handle,
-            recipients=None,
-            passphrase=passphrase,
-            symmetric=True
-        ).data

src/paperless/version.py

@@ -1 +1 @@
-__version__ = (0, 9, 3)
+__version__ = (0, 9, 4)

src/paperless_tesseract/signals.py

@@ -5,9 +5,9 @@ def tesseract_consumer_declaration(sender, **kwargs):
     return {
         "parser": RasterisedDocumentParser,
         "weight": 0,
-        "mime_types": [
-            "application/pdf",
-            "image/jpeg",
-            "image/png"
-        ]
+        "mime_types": {
+            "application/pdf": ".pdf",
+            "image/jpeg": ".jpg",
+            "image/png": ".png"
+        }
     }

src/paperless_text/signals.py

@@ -5,8 +5,8 @@ def text_consumer_declaration(sender, **kwargs):
     return {
         "parser": TextDocumentParser,
         "weight": 10,
-        "mime_types": [
-            "text/plain",
-            "text/comma-separated-values"
-        ]
+        "mime_types": {
+            "text/plain": ".txt",
+            "text/csv": ".csv",
+        }
     }