[BUG] 404 For All Books When Downloading Files #21

Chirishman · 2023-08-22T16:42:08Z

So the maintainers of the LinuxServer Docker container for COPS have switched to this branch which is broadly good but I've been unable to get downloads of books working ever since linuxserver/docker-cops/issues/44

checkconfig.php looks fine across the board and I've now updated all of the config files excepting config_local.php which points the app towards my specific libraries.

Browsing the collection all works just as expected on both index.php and feed.php but all book download attempts 404

Is there something I need to know/change in that config versus the old version/branch?

Note: I've tried it with cops_use_url_rewriting set to both 1 and 0 without any appreciable difference

Here's my config_local.php:

<?php
    if (!isset($config))
        $config = array();

    /*
     * The directory containing calibre's metadata.db file, with sub-directories
     * containing all the formats.
     * BEWARE : it has to end with a /
	 * $config['calibre_directory'] = '/books/ebooks/';
     */
	 $config['calibre_directory'] = array ("eBooks" => "/books/ebooks/", "Comics" => "/books/comics/", "Manga" => "/books/manga/", "Fanfiction" => "/books/fanfiction/", "Webfiction" => "/books/webfiction/", "Light Novels" => "/books/lightnovels/");

    // $config['calibre_internal_directory'] = '/books/ebooks/';

    if (!empty ($_GET) && isset($_GET['db']) && $_GET['db'] != '') {
        if ( $_GET['db'] == 0 )
            $config['calibre_internal_directory'] = "/books/ebooks/files/";
        elseif ( $_GET['db'] == 1 )
               $config['calibre_internal_directory'] = "/books/comics/files/";
        elseif ( $_GET['db'] == 2 )
               $config['calibre_internal_directory'] = "/books/manga/files/";
        elseif ( $_GET['db'] == 3 )
               $config['calibre_internal_directory'] = "/books/fanfiction/files/";
        elseif ( $_GET['db'] == 4 )
               $config['calibre_internal_directory'] = "/books/webfiction/files/";
        elseif ( $_GET['db'] == 5 )
               $config['calibre_internal_directory'] = "/books/lightnovels/files/";
    }

    /*
     * Catalog's title
     */
    $config['cops_title_default'] = "COPS";

    /*
     * use URL rewriting for downloading of ebook in HTML catalog
     * See README for more information
     *  1 : enable
     *  0 : disable
     */
    $config['cops_use_url_rewriting'] = "0";
    $config['cops_ignored_categories'] = array("language");
    $config['cops_calibre_custom_column'] = array("new","characters","fictype","pairings","pairingtypes","parody","relationships","status");
    $config['cops_calibre_custom_column_list'] = array ("new","characters","fictype","updated","lastchecked","chapters","wordcount","pairings","pairingtypes","parody","relationships","status","warnings","contentrated");
    $config['cops_calibre_custom_column_preview'] = array ("new","characters","fictype","updated","lastchecked","chapters","wordcount","pairings","pairingtypes","parody","relationships","status","warnings","contentrated");

	/*
     * Which header to use when downloading books outside the web directory
     * Possible values are :
     *   X-Accel-Redirect   : For Nginx
     *   X-Sendfile         : For Lightttpd or Apache (with mod_xsendfile)
     *   No value (default) : Let PHP handle the download
     */

	 $config['cops_x_accel_redirect'] = "X-Accel-Redirect";
    /* Enable cache folder
     * especially useful for lower power hosts
     */
         $config['cops_thumbnail_handling'] = "";
         $config['cops_thumbnail_cache_directory'] = "/config/cache/";
    /*
     * Enable and configure Send To Kindle (or Email) feature.
     *
     * Don't forget to authorize the sender email you configured in your Kindle's  Approved Personal Document E-mail List.
     *
     * If you want to use a simple smtp server (provided by your ISP for example), you can configure it like that :
     * $config['cops_mail_configuration'] = array( "smtp.host"     => "smtp.free.fr",
     *                                           "smtp.username" => "",
     *                                           "smtp.password" => "",
     *                                           "smtp.secure"   => "",
     *                                           "address.from"  => "[email protected]"
     *                                           );
     *
     * For Gmail (ssl is mandatory) :
     * $config['cops_mail_configuration'] = array( "smtp.host"     => "smtp.gmail.com",
     *                                           "smtp.username" => "YOUR GMAIL ADRESS",
     *                                           "smtp.password" => "YOUR GMAIL PASSWORD",
     *                                           "smtp.secure"   => "ssl",
     *                                           "address.from"  => "[email protected]"
     *                                           );
     *
     * You'll also need to enable Allow Less Secure Apps in you Gmail account.
     */

	$config['cops_mail_configuration'] = array( "smtp.host"     => "",
						    "smtp.username" => "",
						    "smtp.password" => "",
						    "smtp.secure"   => "ssl",
						    "address.from"  => "[email protected]"
);

Edit: I've also tried not setting cops_x_accel_redirect but that results in a 502 error instead of the prior 404

The text was updated successfully, but these errors were encountered:

mikespub · 2023-08-22T19:04:03Z

Sorry to hear about that. There's no reason in general why a configuration like this shouldn't work with the current release, but as usual the issue might be in the details - especially the part where COPS and nginx have to work together to serve the right file.

First of all, I saw from linuxserver/docker-cops#44 that on the linuxserver side you had to upgrade some nginx configuration files - makes sense, since there have been quite some changes there since 2018, and it's independent of COPS or any other PHP application deployed behind it :-)

Since the rest of COPS works fine for you, I'm guessing these configuration changes worked OK for the most part - but you might share some of the details there if you have the time, so the next person knows where to start...

Secondly, looking at your config_local file three things jump out which might cause some problems:
a. the multiple database setup itself
b. the fact that you catch the 'db' param to adapt calibre_internal_directory, which you probably need because:
c. you're trying to use the old x-accel-redirect feature from nginx to send back the file itself

For b. and c. you'll notice at the bottom of fetch.php that COPS basically either tells nginx which file to send, or it sends the file itself using the standard PHP readfile(...) function, depending on whether you have $config['cops_x_accel_redirect'] in your config (= your case) or not (= default).

And it only does that for the books themselves - so it could very well be that images, thumbnails etc. work fine even with the multiple database setup but downloading the file doesn't. Is that what happens here?

One other thing - the internal directory thingie isn't actually used in code anymore since the 1.1.3 release, but it used to do the necessary (re-)mapping of where PHP finds things on the filesystem, compared to where nginx locates things via internal redirects.

Which means that now for your configuration, your nginx files should have something like that for each of your databases:

// Edit: just noticed you need an extra /files in there
    location /books/ebooks {
        root /books/ebooks/files;
        internal;
    }
    location /books/comics {
        root /books/comics/files;
        internal;
    }
   // ... and so on ...

and that if you browse directly to one of these files yourself in your browser (not via COPS), you should get back the right file too.

If you get a 404 error there, you know where the problem is - the mapping of the redirect location to the actual directory in the nginx configuration file. It's worth checking out :-)

It would also help if you could access the nginx error log file to see what it's trying to send, or where COPS told it to redirect to....

mikespub · 2023-08-22T19:49:45Z

About the other things you noticed:

a. about the 502 error when you remove cops_x_accel_redirect

That would be my default recommendation too - that way you wouldn't need any extra nginx configuration.

Do you have any specifics about that 502 error, and does it apply only for books or COPS in general? Because this sounds more like you have a typo in config_local - the option is not set in there by default anyway, so it shouldn't generate any errors...

b. the use_url_rewriting setting should still work in in the sense that it should re-write the fetch.php?data={id}&... URLs for books (and only them) to something like download/{id}/The%20Title.epub in COPS.

Now whether nginx can then actually serve those download/... files will depend on the nginx config files again...

You'd need to have some location /download/ with a rewrite to fetch.php in there to handle it - see https://github.com/seblucas/cops/wiki/Full-example-with-Nginx

But this rewrite is mainly needed for older Kobo devices and in your case you're already letting ngninx send back the book, so there's little point in adding yet another layer of redirection in there...

mikespub · 2023-08-22T22:22:01Z

Corrected the example above - you have an extra /files in your internal directories

Chirishman · 2023-08-22T22:32:22Z

Secondly, looking at your config_local file three things jump out which might cause some problems: a. the multiple database setup itself b. the fact that you catch the 'db' param to adapt calibre_internal_directory, which you probably need because: c. you're trying to use the old x-accel-redirect feature from nginx to send back the file itself

Is there any working example on your fork of multi-database instances?

And it only does that for the books themselves - so it could very well be that images, thumbnails etc. work fine even with the multiple database setup but downloading the file doesn't. Is that what happens here?

Book covers are not working on the prior version, but serving of files is which is all I've really cared about.

One other thing - the internal directory thingie isn't actually used in code anymore since the 1.1.3 release, but it used to do the necessary (re-)mapping of where PHP finds things on the filesystem, compared to where nginx locates things via internal redirects.

So are you saying that I should remove those clauses from config_local.php?

I found default.conf to contain:

    location /books {
        root /;
        internal;
    }

I have tried both configurations which you suggested both replacing root /; with root /books; and individual calls for each of the libraries' paths without making any apparent difference, both are returning a 502 error when $config['cops_x_accel_redirect'] = ""; and a 404 when $config['cops_x_accel_redirect'] = "X-Accel-Redirect";

and that if you browse directly to one of these files yourself in your browser (not via COPS), you should get back the right file too.

What do you mean by browsing directly to this? Going to something like http://localhost:8080/books/ etc? Because that doesn't work on either the old version or the new.

I think I may have found the problem though!

Something either in the prior fork of COPS itself or at least in the code that was making the multi-database code I stole from somewhere work required the layout of the mapped folder to be:

\books
    \ebooks
        \files
            [Actual Calibre Library Folder Contents]
        \metadata.db

Essentially with the metadata.db symlinked one folder up from where the actual calibre library is in a way that Calibre doesn't actually create on it's own.

So the path getting generated is
/books/fanfiction/Half_Baked_Cat/Tanya Degurechaff and the Philosoph (9154)/Tanya Degurechaff and the Philo - Half_Baked_Cat.epub

instead of
/books/fanfiction/files/Half_Baked_Cat/Tanya Degurechaff and the Philosoph (9154)/Tanya Degurechaff and the Philo - Half_Baked_Cat.epub

Edit: Looks like you noticed the /files thing too while I was writing my response

Chirishman · 2023-08-22T23:32:06Z

Ok, moving the junction up and removing the use of the /files folder appears to have solved things.

For reference @mikespub I did try your revised suggestion but it was producing results like this:

2023/08/22 17:45:47 [error] 245#245: *6 open() "/books/fanfiction/files/books/fanfiction/Seras/Ghost in the City (9125)/Ghost in the City - Seras.epub" failed (2: No such file or directory), client: 172.17.0.1, server: _, request:

with rewriting on and this without:

2023/08/22 17:48:28 [error] 245#245: *2 FastCGI sent in stderr: "PHP message: PHP Warning: filesize(): stat failed for /books/fanfiction/Seras/Ghost in the City (9125)/Ghost in the City - Seras.epub in /app/www/public/fetch.php on line 144; PHP message: PHP Warning: readfile(/books/fanfiction/Seras/Ghost in the City (9125)/Ghost in the City - Seras.epub): Failed to open stream: No such file or directory in /app/www/public/fetch.php on line 145" while reading response header from upstream, client: 172.17.0.1, server: _, request: "GET /fetch.php?data=53576&type=epub&id=9125&db=3 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "leliel.chiriserv.com:8080", referrer: "http://leliel.chiriserv.com:8080/index.php?page=10&db=3"

so I figured I'd restructure the target folders as the extra complication no longer seemed necessary and that appears to have worked.

For anyone folowing after the final working configs are as such:

With the individual libraries actually mounted into /books/comics instead of /books/comics/files as I previously had it which I believe had something to do with this snippet seblucas/issues/421 and possibly some other examples I was experimenting with at the time that I first set this up in 2020 which I no longer fully remember.

For config_local.php

<?php
    if (!isset($config))
        $config = array();

    /*
     * The directory containing calibre's metadata.db file, with sub-directories
     * containing all the formats.
     * BEWARE : it has to end with a /
	 * $config['calibre_directory'] = '/books/ebooks/';
     */
    $config['calibre_directory'] = array ("eBooks" => "/books/ebooks/", "Comics" => "/books/comics/", "Manga" => "/books/manga/", "Fanfiction" => "/books/fanfiction/", "Webfiction" => "/books/webfiction/", "Light Novels" => "/books/lightnovels/");

    if (!empty ($_GET) && isset($_GET['db']) && $_GET['db'] != '') {
        if ( $_GET['db'] == 0 )
            $config['calibre_internal_directory'] = "/books/ebooks/";
        elseif ( $_GET['db'] == 1 )
               $config['calibre_internal_directory'] = "/books/comics/";
        elseif ( $_GET['db'] == 2 )
               $config['calibre_internal_directory'] = "/books/manga/";
        elseif ( $_GET['db'] == 3 )
               $config['calibre_internal_directory'] = "/books/fanfiction/";
        elseif ( $_GET['db'] == 4 )
               $config['calibre_internal_directory'] = "/books/webfiction/";
        elseif ( $_GET['db'] == 5 )
               $config['calibre_internal_directory'] = "/books/lightnovels/";
    }

    /*
     * Catalog's title
     */
    $config['cops_title_default'] = "COPS";

    /*
     * use URL rewriting for downloading of ebook in HTML catalog
     * See README for more information
     *  1 : enable
     *  0 : disable
     */
    $config['cops_use_url_rewriting'] = "0";
    $config['cops_ignored_categories'] = array("language");
    $config['cops_ignored_formats'] = ['ORIGINAL_EPUB', 'ORIGINAL_AZW3'];
    $config['cops_calibre_custom_column'] = array("new","characters","fictype","pairings","pairingtypes","parody","relationships","status");
    $config['cops_calibre_custom_column_list'] = array ("new","characters","fictype","updated","lastchecked","chapters","wordcount","pairings","pairingtypes","parody","relationships","status","warnings","contentrated");
    $config['cops_calibre_custom_column_preview'] = array ("new","characters","fictype","updated","lastchecked","chapters","wordcount","pairings","pairingtypes","parody","relationships","status","warnings","contentrated");

     $config['cops_x_accel_redirect'] = "";
    /* Enable cache folder
     * especially useful for lower power hosts
     */
    $config['cops_thumbnail_handling'] = "";
    $config['cops_thumbnail_cache_directory'] = "/config/cache/";

For \config\nginx\site-confs\default.conf

## Version 2023/04/13 - Changelog: https://github.com/linuxserver/docker-cops/commits/master/root/defaults/nginx/site-confs/default.conf.sample

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;

    server_name _;

    include /config/nginx/ssl.conf;

    root /app/www/public;
    index index.html index.htm index.php;

    #Useful only for Kobo reader
    location /cops/ {
        rewrite ^/download/(\d+)/(\d+)/.*\.(.*)$ /fetch.php?data=$1&db=$2&type=$3 last;
        rewrite ^/download/(\d+)/.*\.(.*)$ /fetch.php?data=$1&type=$2 last;
        break;
    }

    #Can break loading the images - if you don't see anything, comment
    location ~ ^/images.*\.(gif|png|ico|jpg)$ {
        expires 31d;
    }
    #Can also break loading the images, comment if it happens
    location ~ .(js|css|eot|svg|woff|ttf)$ {
        expires 31d;
    }

    location /books {
        root /books;
        internal;
    }

    location ~ ^(.+\.php)(.*)$ {
        fastcgi_split_path_info ^(.+\.php)(.*)$;
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_index index.php;
        include /etc/nginx/fastcgi_params;
    }

    # deny access to .htaccess/.htpasswd files
    location ~ /\.ht {
        deny all;
    }
}

Oh and as a bonus it fixed all of the cover images which hadn't been working too.

mikespub · 2023-08-22T23:57:20Z

Excellent, thanks :-)

As far as other multiple database setups go, I don't use them myself in practice, but they're part of the normal COPS test suite that I run on nearly every commit. From your initial description, it didn't sound like point a. above would be the actual problem, but I thought I'd start with it because it does add a potential issue if I missed a db=... parameter somewhere.

One thing the tests don't cover is the possible configurations with nginx unfortunately, so using a standard file structure now does help indeed. And yes, you even get your covers back :-)

tobybryans · 2023-12-27T14:36:39Z

In case anyone else is reading this who is using lighttpd, I have found you need to add the following to your url.rewrite-once config entry for COPS since the URLs changed:

"^/ebooks/download/(\d+)/.*\.(.*)$" => "/ebooks/fetch.php?data=$1&type=$2"

Chirishman mentioned this issue Aug 22, 2023

[Feature Request] Ability to Filter on Non-Tag Columns seblucas/cops#500

Open

mikespub added the linuxserver/cops About using linuxserver/cops docker image label Aug 22, 2023

mikespub closed this as completed Aug 24, 2023

mikespub mentioned this issue Oct 11, 2023

[BUG] 404 For All Books When Downloading Files On Latest Image linuxserver/docker-cops#44

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] 404 For All Books When Downloading Files #21

[BUG] 404 For All Books When Downloading Files #21

Chirishman commented Aug 22, 2023 •

edited

Loading

mikespub commented Aug 22, 2023 •

edited

Loading

mikespub commented Aug 22, 2023

mikespub commented Aug 22, 2023

Chirishman commented Aug 22, 2023 •

edited

Loading

Chirishman commented Aug 22, 2023 •

edited

Loading

mikespub commented Aug 22, 2023

tobybryans commented Dec 27, 2023

[BUG] 404 For All Books When Downloading Files #21

[BUG] 404 For All Books When Downloading Files #21

Comments

Chirishman commented Aug 22, 2023 • edited Loading

mikespub commented Aug 22, 2023 • edited Loading

mikespub commented Aug 22, 2023

mikespub commented Aug 22, 2023

Chirishman commented Aug 22, 2023 • edited Loading

Chirishman commented Aug 22, 2023 • edited Loading

mikespub commented Aug 22, 2023

tobybryans commented Dec 27, 2023

Chirishman commented Aug 22, 2023 •

edited

Loading

mikespub commented Aug 22, 2023 •

edited

Loading

Chirishman commented Aug 22, 2023 •

edited

Loading

Chirishman commented Aug 22, 2023 •

edited

Loading