Feature request: new API endpoint to get the torrent file from the infohash or magnet link

[josecelano]

I can't find a reliable online service to get the torrent file (metainfo) from the infohash or magnet link.

I would like to have an endpoint in the rqbit API to do that. Something like:

• GET /dht/metainfo/info-hash/{info-hash}
• GET /dht/metainfo/magnet-link/{magnet-link}

Or:

• GET /dht/metainfo?info-hash?{info-hash}

• GET /dht/metainfo?magnet-link?{magnet-link}

• Magnet link: magnet:?xt=urn:btih:3f4f9cd8e4a44540002e7386151eb759f76dec51

• URL encoded magnet link: magnet%3A%3Fxt%3Durn%3Abtih%3A3f4f9cd8e4a44540002e7386151eb759f76dec51

I've changed the Ubuntu example to implement a new function in the session type, which was easy. I haven't opened a PR because I don't know the project/conventions, and if you want to add this feature.

In the long term, I would like to have a client linked (via API) to the Index to which I'm contributing. This feature would allow users to add new torrents to the index without having the torrent file.

In general, I miss a Rust BitTorrent client whose API is not focused on interacting with the client GUI but allowing other apps to interact with the swarm via this client/service.

[ikatson]

@josecelano there is a feature like this already kind of

you add the torrent with "list_only=true" parameter, and it'll spit out some details. Try it out, let us know if it has enough information

[josecelano]

Hi @ikatson, I would like to have the .torrent file. I think the returned info does not even have all the needed data to rebuild the torrent.

Request:

curl -d 'magnet:?xt=urn:btih:2b66980093bc11806fab50cb3cb41835b95a0362' http://127.0.0.1:3030/torrents?list_only=true

Output:

{
  "id": null,
  "details": {
    "info_hash": "2b66980093bc11806fab50cb3cb41835b95a0362",
    "name": "debian-12.5.0-amd64-netinst.iso",
    "files": [
      {
        "name": "debian-12.5.0-amd64-netinst.iso",
        "components": [
          "debian-12.5.0-amd64-netinst.iso"
        ],
        "length": 659554304,
        "included": true
      }
    ]
  },
  "output_folder": "/home/josecelano/Downloads/TorrentSeeds/",
  "seen_peers": [
    "193.235.207.24:58695",
    "192.42.116.242:50655",
    "109.126.59.221:1280",
    "192.42.116.243:47583",
    "192.42.116.242:58653",
    "192.42.116.242:44241",
    "192.42.116.244:50681",
    "192.42.116.244:34772",
    "84.49.124.100:6881",
    "198.167.203.177:45357",
    "95.168.168.29:51054",
    "192.42.116.242:42007",
    "188.127.168.212:51413",
    "192.42.116.243:36759",
    "91.189.216.137:61008",
    "192.42.116.244:50766",
    "192.42.116.241:39976",
    "192.42.116.241:57562",
    "192.42.116.242:51988",
    "192.42.116.243:52627",
    "70.20.228.136:26270",
    "192.42.116.244:6881",
    "192.42.116.244:40188",
    "192.42.116.241:59746",
    "192.42.116.242:44258",
    "192.42.116.241:42338",
    "162.55.80.203:6881",
    "178.51.181.136:32814",
    "192.42.116.241:34890",
    "109.110.128.174:54728",
    "108.16.116.63:6881",
    "107.137.195.215:6881",
    "192.42.116.241:42496",
    "172.111.38.128:25999",
    "192.42.116.241:33960",
    "192.42.116.241:38827",
    "183.250.93.143:51413",
    "193.32.127.233:58998",
    "192.42.116.243:49928",
    "212.7.204.116:57980",
    "192.42.116.243:48181",
    "192.42.116.244:1",
    "192.42.116.241:48082",
    "192.42.116.242:1",
    "94.125.242.232:17704",
    "83.80.178.57:37854",
    "213.152.187.210:56202",
    "108.56.74.247:61904",
    "75.226.40.76:57737",
    "192.42.116.243:55788",
    "192.42.116.243:52715",
    "149.233.191.38:27791",
    "192.42.116.241:50449",
    "192.42.116.241:53787",
    "88.153.139.35:6881",
    "71.25.95.181:27112",
    "78.44.144.6:24468",
    "192.42.116.241:44796",
    "192.42.116.243:47471",
    "192.42.116.241:47055",
    "5.59.106.61:45377",
    "94.21.148.170:22633",
    "45.129.140.191:1580",
    "31.16.243.190:1",
    "192.42.116.242:47936",
    "192.42.116.244:57666",
    "84.49.124.100:57626",
    "192.42.116.243:57626",
    "192.42.116.243:56339",
    "149.88.17.169:33906",
    "213.152.187.210:6882",
    "192.42.116.242:35261",
    "50.39.238.164:61759",
    "192.42.116.241:41094",
    "45.10.155.201:17061",
    "192.42.116.241:50433",
    "47.144.175.241:51413",
    "192.42.116.244:57626",
    "192.42.116.243:44591",
    "134.19.179.211:64273",
    "192.42.116.243:59908",
    "192.42.116.242:33714",
    "192.42.116.243:42929",
    "109.250.191.112:51442",
    "183.253.140.165:21088",
    "192.42.116.241:40501",
    "192.42.116.244:40097",
    "183.253.140.165:21386",
    "192.42.116.242:40085",
    "192.42.116.242:46000",
    "192.42.116.244:33005",
    "192.42.116.242:57882",
    "82.65.143.61:58047",
    "192.42.116.241:38427",
    "172.88.226.12:19779",
    "80.78.21.128:23078",
    "183.253.140.165:1",
    "136.38.209.195:56124",
    "84.151.125.203:4269",
    "85.145.236.95:6881",
    "192.42.116.243:46328",
    "192.42.116.241:47012",
    "192.42.116.243:37628",
    "88.152.186.90:21152",
    "88.152.186.90:1",
    "192.42.116.242:39667",
    "37.153.37.47:4414",
    "192.42.116.242:59451",
    "71.173.149.203:19039",
    "206.82.17.213:51413",
    "192.95.122.95:63146",
    "192.42.116.242:34674",
    "192.42.116.243:35926",
    "198.54.130.37:60472",
    "192.42.116.242:60149",
    "192.42.116.242:45184",
    "146.71.73.51:51230",
    "192.42.116.243:51334",
    "192.42.116.243:49123",
    "173.244.49.75:44889",
    "192.42.116.243:51055",
    "183.253.140.165:11696",
    "192.42.116.244:35331",
    "192.42.116.241:36457",
    "192.42.116.242:6881",
    "192.42.116.242:43873",
    "45.133.5.182:27241",
    "192.42.116.242:35628",
    "192.42.116.244:32824",
    "192.42.116.242:37204",
    "198.44.136.82:3667",
    "83.44.166.215:51152",
    "192.42.116.242:46188",
    "192.42.116.241:55746",
    "192.42.116.243:44860",
    "192.42.116.243:38280",
    "192.42.116.241:37974",
    "192.42.116.243:57882",
    "192.42.116.241:39490",
    "159.196.170.8:37289",
    "31.16.243.190:6881",
    "183.253.140.165:30001",
    "172.58.240.71:62168",
    "32.215.238.55:34804",
    "95.65.215.117:28406",
    "37.187.9.88:1",
    "46.246.3.196:63357",
    "5.189.140.45:9999",
    "82.44.250.71:51745",
    "45.133.5.182:10019",
    "192.42.116.242:60356",
    "192.42.116.244:39084",
    "95.28.20.184:64001",
    "192.42.116.242:56958",
    "192.42.116.243:42123",
    "95.111.230.250:9999",
    "192.42.116.243:55779",
    "189.7.87.167:57626",
    "192.42.116.244:44351",
    "109.126.59.221:16205",
    "192.42.116.241:43289",
    "45.139.48.253:45556",
    "188.165.209.227:13639",
    "41.216.53.41:29706",
    "82.10.89.215:3232",
    "62.249.129.30:11940",
    "85.175.5.190:15704",
    "138.199.7.137:57122",
    "188.213.34.25:16620",
    "77.238.131.190:48162",
    "185.89.39.16:50608",
    "192.42.116.241:51401",
    "189.7.87.167:6881",
    "192.42.116.244:45191",
    "72.13.211.98:54728",
    "192.42.116.243:6881",
    "192.42.116.241:1",
    "183.253.140.165:21415",
    "77.238.131.190:47712",
    "192.42.116.242:52633",
    "192.42.116.242:35526",
    "212.241.123.73:6881",
    "192.42.116.243:56716",
    "217.160.89.173:25919",
    "192.42.116.241:57626",
    "31.16.243.190:57626",
    "192.42.116.241:57882",
    "192.42.116.242:36091",
    "192.42.116.243:37543",
    "192.42.116.241:39315",
    "192.42.116.241:35094",
    "192.42.116.243:1",
    "45.130.87.8:52203",
    "192.42.116.241:6882",
    "192.42.116.241:57000",
    "192.42.116.243:38896",
    "80.210.71.237:1028",
    "185.172.53.105:34255",
    "77.72.18.101:50281",
    "45.133.5.182:1",
    "173.174.75.242:57520",
    "192.42.116.243:44735",
    "192.42.116.244:60761",
    "192.42.116.242:57626",
    "195.154.181.225:54992",
    "192.42.116.242:38665",
    "37.19.200.166:53691",
    "192.42.116.241:55207",
    "192.42.116.241:56289",
    "192.42.116.242:36702",
    "37.214.74.222:36929",
    "192.42.116.241:50376",
    "192.42.116.244:46823",
    "37.214.74.222:37686",
    "82.65.194.119:51765",
    "188.213.34.25:59538",
    "192.42.116.244:54900",
    "192.42.116.241:6881",
    "202.187.151.185:9407",
    "185.65.134.207:53731",
    "45.84.139.104:24752",
    "24.60.193.255:37202",
    "192.42.116.241:40311"
  ]
}

Besides, it seems there is no way to download the torrent file from the UI:

[josecelano]

To add more context, I'm actually thinking about using only the lib and building my own API wrapper with just one endpoint. In that case, I would only need a function like the following in the Session struct:

pub fn get_torrent_from_infohash<'a>(
        self: &'a Arc<Self>,
        add: AddTorrent<'a>,
        opts: Option<AddTorrentOptions>,
    ) -> BoxFuture<'a, anyhow::Result<TorrentMetaV1Info<ByteBufOwned>>> {
        async move {
            // Magnet links are different in that we first need to discover the metadata.
            let span = error_span!("add_torrent");
            let _ = span.enter();

            let opts = opts.unwrap_or_default();

            let paused = opts.list_only || opts.paused;

            let announce_port = if paused { None } else { self.tcp_listen_port };

            // The main difference between magnet link and torrent file, is that we need to resolve the magnet link
            // into a torrent file by connecting to peers that support extended handshakes.
            // So we must discover at least one peer and connect to it to be able to proceed further.

            let (info_hash, info, trackers, peer_rx, initial_peers) = match add {
                AddTorrent::Url(magnet) if magnet.starts_with("magnet:") => {
                    let magnet = Magnet::parse(&magnet)
                        .context("provided path is not a valid magnet URL")?;
                    let info_hash = magnet
                        .as_id20()
                        .context("magnet link didn't contain a BTv1 infohash")?;

                    let peer_rx = self.make_peer_rx(
                        info_hash,
                        if opts.disable_trackers {
                            Default::default()
                        } else {
                            magnet.trackers.clone()
                        },
                        announce_port,
                        opts.force_tracker_interval,
                    )?;
                    let peer_rx = match peer_rx {
                        Some(peer_rx) => peer_rx,
                        None => bail!("can't find peers: DHT disabled and no trackers in magnet"),
                    };

                    debug!(?info_hash, "querying DHT");
                    let (info, peer_rx, initial_peers) = match read_metainfo_from_peer_receiver(
                        self.peer_id,
                        info_hash,
                        opts.initial_peers.clone().unwrap_or_default(),
                        peer_rx,
                        Some(self.merge_peer_opts(opts.peer_opts)),
                    )
                    .await
                    {
                        ReadMetainfoResult::Found { info, rx, seen } => (info, rx, seen),
                        ReadMetainfoResult::ChannelClosed { .. } => {
                            bail!("DHT died, no way to discover torrent metainfo")
                        }
                    };
                    debug!(?info, "received result from DHT");
                    (
                        info_hash,
                        info,
                        magnet.trackers.into_iter().unique().collect(),
                        Some(peer_rx),
                        initial_peers,
                    )
                }
                other => {
                    let torrent = match other {
                        AddTorrent::Url(url)
                            if url.starts_with("http://") || url.starts_with("https://") =>
                        {
                            torrent_from_url(&url).await?
                        }
                        AddTorrent::Url(url) => {
                            bail!(
                                "unsupported URL {:?}. Supporting magnet:, http:, and https",
                                url
                            )
                        }
                        AddTorrent::TorrentFileBytes(bytes) => {
                            torrent_from_bytes(&bytes).context("error decoding torrent")?
                        }
                        AddTorrent::TorrentInfo(t) => *t,
                    };

                    let trackers = torrent
                        .iter_announce()
                        .unique()
                        .filter_map(|tracker| match std::str::from_utf8(tracker.as_ref()) {
                            Ok(url) => Some(url.to_owned()),
                            Err(_) => {
                                warn!("cannot parse tracker url as utf-8, ignoring");
                                None
                            }
                        })
                        .collect::<Vec<_>>();

                    let peer_rx = if paused {
                        None
                    } else {
                        self.make_peer_rx(
                            torrent.info_hash,
                            if opts.disable_trackers {
                                Default::default()
                            } else {
                                trackers.clone()
                            },
                            announce_port,
                            opts.force_tracker_interval,
                        )?
                    };

                    (
                        torrent.info_hash,
                        torrent.info,
                        trackers,
                        peer_rx,
                        opts.initial_peers
                            .clone()
                            .unwrap_or_default()
                            .into_iter()
                            .collect(),
                    )
                }
            };

            println!("Info: {:#?}", info);

            Ok(info)
        }
        .boxed()
    }

It's just the add_torrent partially modified to get the metainfo but not to add the torrent.

[josecelano]

Maybe I don't even need the .torrent file in binary format, but at least I need all the info required to rebuild the .torrent file. I suppose the TorrentMetaV1Info<BufType> type contains all the info.

Info: TorrentMetaV1Info {
    name: Some(
        "ubuntu-21.04-live-server-amd64.iso",
    ),
    pieces: <89600 bytes>,
    piece_length: 262144,
    length: Some(
        1174243328,
    ),
    md5sum: None,
    files: None,
}

[ikatson]

If you want to make a PR, here's how I would structure it:

1. Keep the .torrent binary content somewhere after it was resolved
2. Ensure that session.add_torrent returns this together with whatever it returns in list_only mode
3. Create a new HTTP endpoint to return the torrent binary.

Otherwise I might do it next time I work on rqbit

[ikatson]

Check out this: #181

[josecelano]

Check out this: #181

Hi @ikatson I really appreciate it!, I've commented on the PR.

One more question, is there a way to add authentication for both the GUI and the API? I don't need it now, but I would be a nice feature to have if I decided to link this client to the Index. I would like to have an Index connected to both a tracker (already implemented) and a client with a good API (hopefully this one :-)). I guess I can deploy the client behind the proxy and just open the BitTorrent port. However with authentication I would be able to access the GUI and deploy the Client/API as an independent server (independent network).

[josecelano]

Implemented