Cloudy

This is a media indexing and streaming server. For now, the only type of supported media is video.

Built in are

HTTP file server, that is able to serve the media over the web.
HTTP/JSON RPC admin API, that allows to add local files to the internal index.
Internal persistent storage, used to store the media index, organized using a tree like hierarchical structure and the media files, in particular the transcoded video files.
Video transcoder, based on multiple profiles. Any video file has several transcoded versions.
A very low level, simple admin page. See "dashboard" below.

What this lacks

A proper front-end. This is intended to be integrated into a bigger system.
User management
A full HTTP 1.1 server. Instead here is a stripped down, custom implementation.

Expected enhancements

ffmpeg libav wrapper needs more fixes.
Extended support of other media types such as image, pdf, audio.
Static front-end generator, maybe.

Dependencies

boost, crypto++ and ffmpeg are external dependencies.
mesh.pp, belt.pp and a simple cmake utility are git submodules.

How to build cloudy?

user@pc:~$ mkdir projects
user@pc:~$ cd projects
user@pc:~/projects$ git clone https://github.com/tigrangh/cloudy
user@pc:~/projects$ cd cloudy
user@pc:~/projects/cloudy$ git submodule update --init --recursive
user@pc:~/projects/cloudy$ cd ..
user@pc:~/projects$ mkdir cloudy.build
user@pc:~/projects$ cd cloudy.build
user@pc:~/projects/cloudy.build$ cmake -DCMAKE_INSTALL_PREFIX=./install -DCMAKE_BUILD_TYPE=Release ../cloudy
user@pc:~/projects/cloudy.build$ cmake --build . --target install

Running the server

user@pc:~/projects/cloudy.build/install$ ./bin/cloudy/cloudyd -a 127.0.0.1:4444 -s 0.0.0.0:4445 -d ~/cloudy.datadir

-s is the storage endpoint, intended to be exposed to the web
-a is the admin endpoint, intended for internal use
-d path to data directory Use also -k option to define a cryptographic private key. just run once without it, and later reuse the same key that was automatically generated.

Minimal usage example

Dashboard

In below sections, there are the details about the API. This video is a quick demonstration of the features of Cloudy.
This simple JS application is included in the repository as "dashboard.html".
This application is also embedded inside the server and can be accessed with the url 127.0.0.1:4444/dashboard.

Add a file to media library

Consider /path/to/media/file.mp4 to be a local video file

user@pc:~$ curl -X PUT --data '[{"rtt":24, "container_extension":"mp4", "audio":{"rtt":25, "transcode":{"rtt":26, "codec":"aac"}}, "video":{"rtt":25, "transcode":{"rtt":26, "codec":"libx264", "parameters":{"preset":"fast"}, "filter":{"rtt":27, "adjust":false, "height":1080, "width":1920, "fps":29, "rotate":0}}}}, {"rtt":24, "container_extension":"mp4", "audio":{"rtt":25, "transcode":{"rtt":26, "codec":"aac"}}, "video":{"rtt":25, "transcode":{"rtt":26, "codec":"libx264", "parameters":{"preset":"fast"}, "filter":{"rtt":27, "adjust":false, "height":720, "width":1280, "fps":29, "rotate":0}}}}, {"rtt":24, "container_extension":"mp4", "audio":{"rtt":25, "transcode":{"rtt":26, "codec":"aac"}}, "video":{"rtt":25, "transcode":{"rtt":26, "codec":"libx264", "parameters":{"preset":"fast"}, "filter":{"rtt":27, "adjust":true, "height":360, "width":640, "fps":29, "rotate":0}}}}]' "127.0.0.1:4444/library/path/to/media/file.mp4"
{"rtt":7,"lib_files":[],"lib_directories":[],"fs_files":["file.mp4"],"fs_directories":[]}

The response shows already existing files and folders in the library and in the fs (in the current directory), in this case nothing yet in the library. This is an asyncronous request.

Check to know when the video is processed

user@pc:~$ curl "127.0.0.1:4444/log"
{"rtt":12,"log":[{"rtt":13,"path":["path","to","media","file.mp4"]}]}

The array "log" will be empty unless the waiting for the video transcoding is over. When it's done we have the log entry as in the example above. There are different codes to indicate an error, warning or success. The above example shows success.

So, what is done actually?

user@pc:~$ curl "127.0.0.1:4444/library/path/to/media"
{"rtt":7,"lib_files":[{"rtt":8,"name":"file.mp4","checksum":"GvN8WbnpBtXe6GzJPbQtmanD6gxg7Bt8XHibwU7x546m"}],"lib_directories":[],"fs_files":[],"fs_directories":[]}

With this we get the checksum of the file - sha256 hash
And then

user@pc:~$ curl "127.0.0.1:4444/index/GvN8WbnpBtXe6GzJPbQtmanD6gxg7Bt8XHibwU7x546m"
{"rtt":22,"paths":[["path","to","media","file.mp4"]],"type_definitions":[{"rtt":21,"type_description":{"rtt":24,"audio":{"rtt":25,"transcode":{"rtt":26,"codec":"aac"}},"video":{"rtt":25,"transcode":{"rtt":26,"codec":"libx264","parameters":{"preset":"fast"},"filter":{"rtt":27,"adjust":true,"height":360,"width":640,"fps":29,"rotate":0.000000}}},"container_extension":"mp4"},"sequence":{"rtt":19,"done":true,"frames":[{"rtt":20,"count":54213,"uri":"ASCvRY6YCMsLAD2iPyMHPnnb9Lqjg1Zhq15o8JnxYSfM"}]}},{"rtt":21,"type_description":{"rtt":24,"audio":{"rtt":25,"transcode":{"rtt":26,"codec":"aac"}},"video":{"rtt":25,"transcode":{"rtt":26,"codec":"libx264","parameters":{"preset":"fast"},"filter":{"rtt":27,"adjust":false,"height":1080,"width":1920,"fps":29,"rotate":0.000000}}},"container_extension":"mp4"},"sequence":{"rtt":19,"done":true,"frames":[{"rtt":20,"count":54213,"uri":"2abSXktkdeFWBTpJGFXvxT6x5nuPn1MAX9xKD2GPQqv9"}]}},{"rtt":21,"type_description":{"rtt":24,"audio":{"rtt":25,"transcode":{"rtt":26,"codec":"aac"}},"video":{"rtt":25,"transcode":{"rtt":26,"codec":"libx264","parameters":{"preset":"fast"},"filter":{"rtt":27,"adjust":false,"height":720,"width":1280,"fps":29,"rotate":0.000000}}},"container_extension":"mp4"},"sequence":{"rtt":19,"done":true,"frames":[{"rtt":20,"count":54213,"uri":"C56jZnpinpaeS5KDGxtuBRRy3YxcbXx46eFpkgBC1XW4"}]}}]}

This shows that there are three transcoded versions of the original video file, details about the transcoding options and the "uri" of each transcoded file, which can be used to request the file from storage server. By the way, "count" shows the duration of the video in milliseconds. But just the uri is made to be not enough to get the file, we need to ask the admin interface to sign it, and get an authorization.

Get the authorization

user@pc:~$ curl "127.0.0.1:4444/authorization?file=C56jZnpinpaeS5KDGxtuBRRy3YxcbXx46eFpkgBC1XW4&seconds=3600"
eyJydHQiOjE3LCJ0b2tlbiI6eyJydHQiOjE2LCJmaWxlX3VyaSI6IkM1NmpabnBpbnBhZVM1S0RHeHR1QlJSeTNZeGNiWHg0NmVGcGtnQkMxWFc0Iiwic2Vzc2lvbl9pZCI6IiIsInNlY29uZHMiOjM2MDAsInRpbWVfcG9pbnQiOiIyMDIwLTA0LTIzIDA5OjAwOjAwIn0sImF1dGhvcml6YXRpb24iOnsicnR0IjoxOCwiYWRkcmVzcyI6IkNsb3VkeS01d21EYlJxVEt3c1ZZSEVjb0Y2blhmSzZGVFZoU2FUZEs1VVMxOURXMmpROW1hNmtOZiIsInNpZ25hdHVyZSI6IjM4MXlYWW5uOUxIYVBwZWVNRmRVczNuU2dSVnFUTVo0bzZUOVpXd2hZWXJzb01TZXBuenJnRHE1Sld2elJCWkxTVnhQVE0xeGhQeVlFbXZveDNxSnllQ3ZVdzdFdW0zaSJ9fQ==

This long string is a base64 encoded json structure that includes cryptographic signature and link validity information. We'll use it to get the actual transcoded video file from storage api. Be sure to url encode it properly

Get the media library file

user@pc:~$ wget "0.0.0.0:4445/storage?authorization=eyJydHQiOjE3LCJ0b2tlbiI6eyJydHQiOjE2LCJmaWxlX3VyaSI6IkM1NmpabnBpbnBhZVM1S0RHeHR1QlJSeTNZeGNiWHg0NmVGcGtnQkMxWFc0Iiwic2Vzc2lvbl9pZCI6IiIsInNlY29uZHMiOjM2MDAsInRpbWVfcG9pbnQiOiIyMDIwLTA0LTIzIDA5OjAwOjAwIn0sImF1dGhvcml6YXRpb24iOnsicnR0IjoxOCwiYWRkcmVzcyI6IkNsb3VkeS01d21EYlJxVEt3c1ZZSEVjb0Y2blhmSzZGVFZoU2FUZEs1VVMxOURXMmpROW1hNmtOZiIsInNpZ25hdHVyZSI6IjM4MXlYWW5uOUxIYVBwZWVNRmRVczNuU2dSVnFUTVo0bzZUOVpXd2hZWXJzb01TZXBuenJnRHE1Sld2elJCWkxTVnhQVE0xeGhQeVlFbXZveDNxSnllQ3ZVdzdFdW0zaSJ9fQ%3D%3D"

Create a simple static html page

We can "upload" any file to cloudy. For example let's have /path/to/index.html file with the following content.

<video width="800" height="600" controls>
  <source src="http://example.com:4445/storage?authorization=eyJydHQiOjE3LCJ0b2tlbiI6eyJydHQiOjE2LCJmaWxlX3VyaSI6IkM1NmpabnBpbnBhZVM1S0RHeHR1QlJSeTNZeGNiWHg0NmVGcGtnQkMxWFc0Iiwic2Vzc2lvbl9pZCI6IiIsInNlY29uZHMiOjM2MDAsInRpbWVfcG9pbnQiOiIyMDIwLTA0LTIzIDA5OjAwOjAwIn0sImF1dGhvcml6YXRpb24iOnsicnR0IjoxOCwiYWRkcmVzcyI6IkNsb3VkeS01d21EYlJxVEt3c1ZZSEVjb0Y2blhmSzZGVFZoU2FUZEs1VVMxOURXMmpROW1hNmtOZiIsInNpZ25hdHVyZSI6IjM4MXlYWW5uOUxIYVBwZWVNRmRVczNuU2dSVnFUTVo0bzZUOVpXd2hZWXJzb01TZXBuenJnRHE1Sld2elJCWkxTVnhQVE0xeGhQeVlFbXZveDNxSnllQ3ZVdzdFdW0zaSJ9fQ%3D%3D" type="video/mp4">
  Your browser does not support the video tag.
</video>

Then

user@pc:~$ curl -X PUT --data '[{"rtt":29, "mime_type":"text/html"}]' "127.0.0.1:4444/library/path/to/index.html"
{"rtt":7,"lib_files":[],"lib_directories":[],"fs_files":["index.html"],"fs_directories":[]}

With this we do a similar thing as above when adding a video file to library, but instead of telling cloudy to transcode a video file, we simply ask it to copy this file to internal structure, and remember its mime-type as "text/html". Following the examples from above steps, we can get the url of this simple html page, and share it with other people or services.

JSON protocol

The following is not a real JSON schema, but it gives enough information how to tweak the JSON parameters. This is defined in a built-in small language, which helps to autogenerate whole lot of c++ code during the build process, which is used to actually implement the protocol.

There are also tools to autogenerate php and TypeScript libraries.

user@pc:~$ curl 127.0.0.1:4444/protocol
{

    "IndexListGet": {
        "type": "object",
        "rtt": 0,
        "properties": {}
    },

    "IndexListResponse": {
        "type": "object",
        "rtt": 1,
        "properties": {
            "list_index": { "type": "Hash String LibraryIndex"}
        }
    },

    "IndexGet": {
        "type": "object",
        "rtt": 2,
        "properties": {
            "sha256sum": { "type": "String"}
        }
    },

    "IndexDelete": {
        "type": "object",
        "rtt": 3,
        "properties": {
            "sha256sum": { "type": "String"}
        }
    },

    "LibraryGet": {
        "type": "object",
        "rtt": 4,
        "properties": {
            "path": { "type": "Array String"}
        }
    },

    "LibraryPut": {
        "type": "object",
        "rtt": 5,
        "properties": {
            "path": { "type": "Array String"},
            "type_descriptions": { "type": "Array Variant"}
        }
    },

    "LibraryDelete": {
        "type": "object",
        "rtt": 6,
        "properties": {
            "path": { "type": "Array String"}
        }
    },

    "LibraryResponse": {
        "type": "object",
        "rtt": 7,
        "properties": {
            "lib_files": { "type": "Array FileItem"},
            "lib_directories": { "type": "Array DirectoryItem"},
            "fs_files": { "type": "Array FileItem"},
            "fs_directories": { "type": "Array DirectoryItem"}
        }
    },

    "FileItem": {
        "type": "object",
        "rtt": 8,
        "properties": {
            "name": { "type": "String"},
            "checksum": { "type": "Optional String"}
        }
    },



    "DirectoryItem": {
        "type": "object",
        "rtt": 9,
        "properties": {
            "name": { "type": "String"}
        }
    },

    "LogGet": {
        "type": "object",
        "rtt": 10,
        "properties": {}
    },

    "LogDelete": {
        "type": "object",
        "rtt": 11,
        "properties": {
            "count": { "type": "UInt64"}
        }
    },

    "Log": {
        "type": "object",
        "rtt": 12,
        "properties": {
            "log": { "type": "Array Variant"}
        }
    },

    "CheckMediaResult": {
        "type": "object",
        "rtt": 13,
        "properties": {
            "path": { "type": "Array String"}
        }
    },

    "CheckMediaError": {
        "type": "object",
        "rtt": 14,
        "properties": {
            "path": { "type": "Array String"},
            "reason": { "type": "String"}
        }
    },

    "CheckMediaWarning": {
        "type": "object",
        "rtt": 15,
        "properties": {
            "path": { "type": "Array String"},
            "reason": { "type": "String"}
        }
    },

    "StorageAuthorization": {
        "type": "object",
        "rtt": 16,
        "properties": {
            "file_uri": { "type": "String"},
            "session_id": { "type": "String"},
            "seconds": { "type": "UInt64"},
            "time_point": { "type": "TimePoint"}
        }
    },

    "SignedStorageAuthorization": {
        "type": "object",
        "rtt": 17,
        "properties": {
            "token": { "type": "StorageAuthorization"},
            "authorization": { "type": "Authority"}
        }
    },

    "Authority": {
        "type": "object",
        "rtt": 18,
        "properties": {
            "address": { "type": "String"},
            "signature": { "type": "String"}
        }
    },



    "MediaSequence": {
        "type": "object",
        "rtt": 19,
        "properties": {
            "done": { "type": "Bool"},
            "frames": { "type": "Array MediaFrame"}
        }
    },

    "MediaFrame": {
        "type": "object",
        "rtt": 20,
        "properties": {
            "count": { "type": "UInt64"},
            "uri": { "type": "String"}
        }
    },

    "MediaTypeDefinition": {
        "type": "object",
        "rtt": 21,
        "properties": {
            "type_description": { "type": "Variant"},
            "sequence": { "type": "MediaSequence"}
        }
    },

    "LibraryIndex": {
        "type": "object",
        "rtt": 22,
        "properties": {
            "paths": { "type": "Array Array"},
            "type_definitions": { "type": "Array MediaTypeDefinition"}
        }
    },

    "RemoteError": {
        "type": "object",
        "rtt": 23,
        "properties": {
            "message": { "type": "String"}
        }
    },

    "MediaTypeDescriptionAVContainer": {
        "type": "object",
        "rtt": 24,
        "properties": {
            "audio": { "type": "Optional MediaTypeDescriptionAVStream"},
            "video": { "type": "Optional MediaTypeDescriptionAVStream"},
            "muxer_parameters": { "type": "Optional Hash"},
            "container_extension": { "type": "String"}
        }
    },

    "MediaTypeDescriptionAVStream": {
        "type": "object",
        "rtt": 25,
        "properties": {
            "transcode": { "type": "Optional MediaTypeDescriptionAVStreamTranscode"}
        }
    },

    "MediaTypeDescriptionAVStreamTranscode": {
        "type": "object",
        "rtt": 26,
        "properties": {
            "codec": { "type": "String"},
            "parameters": { "type": "Optional Hash"},
            "filter": { "type": "Optional Variant"}
        }
    },

    "MediaTypeDescriptionVideoFilter": {
        "type": "object",
        "rtt": 27,
        "properties": {
            "adjust": { "type": "Bool"},
            "height": { "type": "UInt64"},
            "width": { "type": "UInt64"},
            "fps": { "type": "UInt64"},
            "rotate": { "type": "Float64"},
            "background_color": { "type": "Optional String"},
            "stabilize": { "type": "Optional Bool"}
        }
    },

    "MediaTypeDescriptionAudioFilter": {
        "type": "object",
        "rtt": 28,
        "properties": {
            "volume": { "type": "Float64"}
        }
    },

    "MediaTypeDescriptionRaw": {
        "type": "object",
        "rtt": 29,
        "properties": {
            "mime_type": { "type": "String"}
        }
    }

}

Name		Name	Last commit message	Last commit date
Latest commit History 43 Commits
src		src
.gitignore		.gitignore
.gitmodules		.gitmodules
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
README.md		README.md
dashboard.html		dashboard.html

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Cloudy

Built in are

What this lacks

Expected enhancements

Dependencies

How to build cloudy?

Running the server

Minimal usage example

Dashboard

Add a file to media library

Check to know when the video is processed

So, what is done actually?

Get the authorization

Get the media library file

Create a simple static html page

JSON protocol

About

Releases

Packages

Languages

License

tigrangh/cloudy

Folders and files

Latest commit

History

Repository files navigation

Cloudy

Built in are

What this lacks

Expected enhancements

Dependencies

How to build cloudy?

Running the server

Minimal usage example

Dashboard

Add a file to media library

Check to know when the video is processed

So, what is done actually?

Get the authorization

Get the media library file

Create a simple static html page

JSON protocol

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages